Reverse order scans are an optimization for queries like ORDER BY timestamp DESC LIMIT n where the data is ordered by timestamp ASC. Such read patterns appear constantly in time-series workloads where callers want the most recent rows. With the current implementation users would follow naive approaches: fully scan a Vortex file, buffer all rows and then reverse the output or sort all rows of the file. This is unnecessarily expensive.
If files are already written in sorted order, a scan in opposite direction can be answered by iterating chunks from last to first and reversing the rows within each chunk. Avoiding sorting and buffering.
I opened a PR with an implementation proposal: #7777. This PR implements this by reversing ranges in the scan layer and reversing the Vortex array representation. Let's discuss if there are more appropriate ways of implementing this in Vortex (I'm new to the codebase).
Let me know if you think this should be split into several sub-tasks.
Related Work
Similar ideas have been discussed in Datafusion: apache/datafusion#17172
A former colleague of mine worked on reversing parquet streams in Arrow: apache/arrow-rs#3922
Reverse order scans are an optimization for queries like
ORDER BY timestamp DESC LIMIT nwhere the data is ordered bytimestamp ASC. Such read patterns appear constantly in time-series workloads where callers want the most recent rows. With the current implementation users would follow naive approaches: fully scan a Vortex file, buffer all rows and then reverse the output or sort all rows of the file. This is unnecessarily expensive.If files are already written in sorted order, a scan in opposite direction can be answered by iterating chunks from last to first and reversing the rows within each chunk. Avoiding sorting and buffering.
I opened a PR with an implementation proposal: #7777. This PR implements this by reversing ranges in the scan layer and reversing the Vortex array representation. Let's discuss if there are more appropriate ways of implementing this in Vortex (I'm new to the codebase).
Let me know if you think this should be split into several sub-tasks.
Related Work
Similar ideas have been discussed in Datafusion: apache/datafusion#17172
A former colleague of mine worked on reversing parquet streams in Arrow: apache/arrow-rs#3922