Skip to content

Commit 2f117e4

Browse files
committed
Merge branch 'develop' of github.com:vortex-data/vortex into reverse-order-scans
2 parents 0e64d5e + 44a6367 commit 2f117e4

557 files changed

Lines changed: 24885 additions & 27851 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/ISSUE_TEMPLATE/epic.md

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
name: Epic
3+
about: Umbrella issue for a major initiative tracked via sub-issues.
4+
title: "Epic: "
5+
labels: epic
6+
---
7+
8+
<!--
9+
An Epic is the umbrella for work that ships as several tracking issues. Attach
10+
them with GitHub's "Create sub-issue" or "Add existing sub-issue" button. The
11+
sub-issue panel tracks progress, so this issue does not need a checklist.
12+
13+
Keep this issue readable. Route discussion elsewhere:
14+
15+
- Open-ended scope or product questions: open a GitHub Discussion and link it
16+
here.
17+
- Design debate scoped to one piece of work: open a tracking issue as a
18+
sub-issue and discuss it there.
19+
- Bug reports: file a bug issue and reference this Epic.
20+
21+
Short clarifying comments are fine. Long threads should be redirected.
22+
-->
23+
24+
This Epic is for ...
25+
26+
<!--
27+
One or two sentences explaining this Epic.
28+
29+
Please also link any roadmap, design, or external context that helps readers
30+
understand the scope.
31+
-->
32+
33+
## Status
34+
35+
<!--
36+
Proposed / Active / Blocked / Complete. Briefly explain the current state.
37+
-->
38+
39+
**Proposed.**
40+
41+
## Goal
42+
43+
<!--
44+
Cover both:
45+
46+
- What "done" looks like for a user or downstream consumer. Name the concrete
47+
API, format, encoding, or benchmark.
48+
- What people might reasonably expect this Epic to cover but it will not.
49+
Non-goals stop scope creep.
50+
-->
51+
52+
## Motivation
53+
54+
<!--
55+
Why this, why now? Cover:
56+
57+
- The concrete problem and who/what hits it. Examples: a downstream integration
58+
(DataFusion, DuckDB, Python, Java), a benchmark gap (TPC-H, ClickBench,
59+
vortex-bench), or a file format or encoding limitation.
60+
- What is currently blocked, slower, or larger than it should be. Link the
61+
benchmark numbers, profile output, or related issues.
62+
63+
If the answer is "it would be nice to have", the Epic is not ready.
64+
-->
65+
66+
## Unresolved questions
67+
68+
<!--
69+
Scope-level questions that must be answered before this Epic closes.
70+
Implementation questions belong in the relevant tracking issue.
71+
72+
Each entry should be specific enough that a future reader can tell whether it
73+
has been answered. If a question grows large, please create a new tracking
74+
issue or Discussion and link it.
75+
-->
76+
77+
- [ ] None yet.
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
---
2+
name: Tracking Issue
3+
about: A tracking issue for a feature or initiative in Vortex.
4+
title: "Tracking Issue: "
5+
labels: tracking-issue
6+
---
7+
8+
<!--
9+
Thank you for opening a tracking issue. Tracking issues record a feature's
10+
progress from design to landing and connect related bugs, PRs, and design
11+
questions. They should stay readable a year from now.
12+
13+
Route discussion elsewhere:
14+
15+
- Bug reports surfaced while implementing: file a bug issue and reference this
16+
tracking issue.
17+
- A design thread big enough to need its own page: spawn a tracking issue (as
18+
a sub-issue of this one if it is narrow) and discuss it there.
19+
20+
Short clarifying comments are fine. Long threads should be redirected.
21+
-->
22+
23+
This is a tracking issue for ...
24+
25+
<!--
26+
State what the feature is and where it lives in the codebase (crate, module,
27+
file format, encoding). Link the parent Epic, design doc, prototype PR, or
28+
external reference if any.
29+
-->
30+
31+
## Motivation
32+
33+
<!--
34+
Optional. Skip if this is a sub-issue of an Epic that already covers the why.
35+
36+
For standalone tracking issues, cover:
37+
38+
- The concrete problem and who hits it. Examples: a downstream integration
39+
(DataFusion, DuckDB, Python, Java), a benchmark gap (TPC-H, ClickBench,
40+
vortex-bench), a regression against Parquet or Arrow, or a file format or
41+
encoding limitation.
42+
- What is currently blocked, slower, or larger than it should be. Link the
43+
benchmark numbers, profile output, or related issues.
44+
-->
45+
46+
## Design
47+
48+
<!--
49+
Optional but recommended. Size to fit the work:
50+
51+
- API additions: a stripped-down signature block. Rust traits or functions,
52+
file-format struct, or wire-format message. Drop doc comments and bodies.
53+
- Behavioral changes: a short before / after description, ideally with a small
54+
example.
55+
- Architectural changes: a paragraph explaining the new shape, or a link to a
56+
design doc.
57+
58+
Skip only if the change is fully obvious from the title. The Design section is
59+
what lets a reviewer judge whether the Steps below are the right ones.
60+
-->
61+
62+
## Steps
63+
64+
<!--
65+
Major checkpoints required to call this feature done. A handful of milestones,
66+
not every PR. Tick steps as they land; do not delete them.
67+
68+
Steps are milestones, not PRs (initial implementation, documentation,
69+
stabilization). If a step is itself a separable shippable unit, promote it to
70+
a sub-issue.
71+
-->
72+
73+
- [ ] Initial implementation
74+
- [ ] Documentation
75+
- [ ] Public API stabilization
76+
77+
## Unresolved questions
78+
79+
<!--
80+
Open design or implementation questions blocking progress. Link discussions
81+
and conclusions so a future reader can see how each question was settled.
82+
83+
If a question is large enough to need its own thread, spawn a tracking issue
84+
or Discussion and link it.
85+
-->
86+
87+
- [ ] None yet.
88+
89+
## Implementation history
90+
91+
<!--
92+
A running log of every PR that touched this feature: initial implementation,
93+
follow-ups, fixes, reverts. Grows continuously and is never ticked off. The
94+
archaeology trail for someone reading this issue a year later.
95+
96+
Add PRs as they merge, in chronological order. A one-line description per PR
97+
is helpful but not required.
98+
-->

.github/actions/alert-incident-io/action.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ inputs:
2727
runs:
2828
using: "composite"
2929
steps:
30-
- uses: incident-io/github-action@v1
30+
- uses: incident-io/github-action@7aa5f85e67679cd8fdf2a19aed3d5450335a004b # v1
3131
with:
3232
api-key: ${{ inputs.api-key }}
3333
alert-source-id: 01KH4EYTH3HA4PDZPRAPEV1Q10

.github/actions/setup-rust/action.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ runs:
3636

3737
- name: Install Mold
3838
if: runner.os == 'Linux'
39-
uses: rui314/setup-mold@v1
39+
uses: rui314/setup-mold@9c9c13bf4c3f1adef0cc596abc155580bcb04444 # v1
4040

4141
- name: Check for rustup
4242
id: check-rustup
@@ -45,7 +45,7 @@ runs:
4545

4646
- name: Rust Toolchain
4747
id: rust-toolchain
48-
uses: dtolnay/rust-toolchain@stable
48+
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
4949
if: steps.check-rustup.outputs.exists != 'true'
5050
with:
5151
toolchain: "${{ steps.toolchain-config.outputs.toolchain }}"
@@ -73,7 +73,7 @@ runs:
7373
7474
- name: Rust Compile Cache
7575
if: inputs.enable-sccache == 'true'
76-
uses: mozilla-actions/sccache-action@v0.0.9
76+
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
7777

7878
- name: Pre-start sccache server
7979
if: inputs.enable-sccache == 'true'

.github/runs-on.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,6 @@
11
_extends: .github-private
2+
images:
3+
ubuntu24-full-arm64-pre:
4+
platform: "linux"
5+
arch: "arm64"
6+
ami: "ami-08f97c6362847dc5d"

.github/workflows/approvals.yml

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,21 +23,20 @@ jobs:
2323
pull_number: pr.number,
2424
});
2525
26-
// Count unique human approvals (latest review per user)
26+
// Determine if PR author is a bot/GitHub Actions
27+
const authorType = pr.user.type; // 'Bot' vs 'User'
28+
const authorLogin = pr.user.login; // e.g. 'github-actions[bot]'
29+
const isBot = authorType === 'Bot' || authorLogin.endsWith('[bot]');
30+
const oneApprovalBotAuthors = new Set(['renovate[bot]']);
31+
32+
// Count unique approvals, including bot approvals.
2733
const latestByUser = {};
2834
for (const review of reviews.data) {
29-
if (review.user.type === 'Bot') continue;
3035
latestByUser[review.user.login] = review.state;
3136
}
3237
const approvalCount = Object.values(latestByUser)
3338
.filter(state => state === 'APPROVED').length;
3439
35-
// Determine if PR author is a bot/GitHub Actions
36-
const authorType = pr.user.type; // 'Bot' vs 'User'
37-
const authorLogin = pr.user.login; // e.g. 'github-actions[bot]'
38-
const isBot = authorType === 'Bot' || authorLogin.endsWith('[bot]');
39-
const oneApprovalBotAuthors = new Set(['renovate[bot]']);
40-
4140
const required = isBot && !oneApprovalBotAuthors.has(authorLogin) ? 2 : 1;
4241
4342
console.log(`PR author: ${authorLogin} (${authorType}), isBot: ${isBot}`);
@@ -46,7 +45,7 @@ jobs:
4645
4746
if (isBot && (approvalCount < required)) {
4847
core.setFailed(
49-
`This PR needs ${required} human approval(s) but has ${approvalCount}. ` +
48+
`This PR needs ${required} approval(s) but has ${approvalCount}. ` +
5049
`(Author is ${isBot ? 'a bot' : 'human'})`
5150
);
5251
}

.github/workflows/bench-dispatch.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121
timeout-minutes: 10
2222
if: github.event.label.name == 'action/benchmark'
2323
steps:
24-
- uses: actions-ecosystem/action-remove-labels@v1
24+
- uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1
2525
if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex'
2626
with:
2727
labels: action/benchmark
@@ -37,7 +37,7 @@ jobs:
3737
timeout-minutes: 10
3838
if: github.event.label.name == 'action/benchmark-sql'
3939
steps:
40-
- uses: actions-ecosystem/action-remove-labels@v1
40+
- uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1
4141
if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex'
4242
with:
4343
labels: action/benchmark-sql

.github/workflows/bench-pr.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
if: github.event.pull_request.head.repo.fork == false
3939
with:
4040
sccache: s3
41-
- uses: actions/checkout@v6
41+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
4242
with:
4343
ref: ${{ github.event.pull_request.head.sha }}
4444
- name: Setup benchmark environment
@@ -64,7 +64,7 @@ jobs:
6464
6565
- name: Setup Polar Signals
6666
if: github.event.pull_request.head.repo.fork == false
67-
uses: polarsignals/gh-actions-ps-profiling@v0.8.1
67+
uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1
6868
with:
6969
polarsignals_cloud_token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
7070
labels: "branch=${{ github.ref_name }};gh_run_id=${{ github.run_id }};benchmark=${{ matrix.benchmark.id }}"
@@ -83,13 +83,13 @@ jobs:
8383
8484
- name: Setup AWS CLI
8585
if: github.event.pull_request.head.repo.fork == false
86-
uses: aws-actions/configure-aws-credentials@v6
86+
uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6
8787
with:
8888
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
8989
aws-region: us-east-1
9090

9191
- name: Install uv
92-
uses: spiraldb/actions/.github/actions/setup-uv@0.18.5
92+
uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6
9393
with:
9494
sync: false
9595

@@ -117,14 +117,14 @@ jobs:
117117
118118
- name: Comment PR
119119
if: github.event.pull_request.head.repo.fork == false
120-
uses: thollander/actions-comment-pull-request@v3
120+
uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3
121121
with:
122122
file-path: comment.md
123123
comment-tag: bench-pr-comment-${{ matrix.benchmark.id }}
124124

125125
- name: Comment PR on failure
126126
if: failure() && github.event.pull_request.head.repo.fork == false
127-
uses: thollander/actions-comment-pull-request@v3
127+
uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3
128128
with:
129129
message: |
130130
# BENCHMARK FAILED

.github/workflows/bench.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,9 @@ jobs:
1616
runs-on: ubuntu-latest
1717
timeout-minutes: 10
1818
steps:
19-
- uses: actions/checkout@v6
19+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
2020
- name: Setup AWS CLI
21-
uses: aws-actions/configure-aws-credentials@v6
21+
uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6
2222
with:
2323
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
2424
aws-region: us-east-1
@@ -54,7 +54,7 @@ jobs:
5454
if: github.repository == 'vortex-data/vortex'
5555
with:
5656
sccache: s3
57-
- uses: actions/checkout@v6
57+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
5858
- name: Setup benchmark environment
5959
run: sudo bash scripts/setup-benchmark.sh
6060
- uses: ./.github/actions/setup-rust
@@ -77,7 +77,7 @@ jobs:
7777
cargo build --bin ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} --features unstable_encodings
7878
7979
- name: Setup Polar Signals
80-
uses: polarsignals/gh-actions-ps-profiling@v0.8.1
80+
uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1
8181
with:
8282
polarsignals_cloud_token: ${{ secrets.POLAR_SIGNALS_API_KEY }}
8383
labels: "branch=${{ github.ref_name }};gh_run_id=${{ github.run_id }};benchmark=${{ matrix.benchmark.id }}"
@@ -95,7 +95,7 @@ jobs:
9595
bash scripts/bench-taskset.sh target/release_debug/${{ matrix.benchmark.id }} --formats ${{ matrix.benchmark.formats }} -d gh-json -o results.json
9696
9797
- name: Setup AWS CLI
98-
uses: aws-actions/configure-aws-credentials@v6
98+
uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6
9999
with:
100100
role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole
101101
aws-region: us-east-1

0 commit comments

Comments
 (0)