You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Squashed PR snapshot of the long-lived `final` work, rebased on top of
current develop (2a864c5). Combines the original "TsFile C++ batch
read/write optimization" (5f12115) snapshot with subsequent build /
platform fixes and a follow-up read-path optimization commit
(c902b2b).
═════════════════════════════════════════════════════════════════════
Read path
═════════════════════════════════════════════════════════════════════
- Decoder base gains batch APIs (read_batch_int32 / int64 / float /
double, skip_*); PLAIN, TS2DIFF, Gorilla decoders implement them.
TS2DIFF has block-level peeking so time filters can skip blocks
without decoding. Gorilla adds a raw-pointer GorillaBitReader that
bypasses ByteStream overhead.
- ChunkReader / AlignedChunkReader add *_DECODE_TV_BATCH methods that
decode time + value into a TsBlock in one pass, applying batch time
filters before append.
- AlignedChunkReader supports a multi-value mode: one time chunk + N
value chunks decoded in a single pass, sharing the decoded timestamps
and filter mask. SingleDeviceTsBlockReader auto-detects same-device
measurements via VectorMeasurementColumnContext.
- Optional page-level parallel decompression via a DecodeThreadPool when
ENABLE_THREADS is set. Page-plan classification
(SKIP / FULL_PASS / BOUNDARY) lets a scatter-free memcpy fast path
fire when every row passes and no column has nulls.
Additional optimizations (from c902b2b, ported from `final`):
- Aligned fast path: enable_dense_aligned_fast_path defaults true and
compute_dense_row_count falls back to the TimeseriesIndex top-level
statistic for single-chunk timeseries (chunk-level stat is omitted
during serialization for those). Re-enables the bulk-copy SSI -->
caller path that was defensively disabled.
- Chunk-level parallel decode: per-column tasks own all that column's
pages and write into a per-(col,page) PageDecodedState slot; one
wait_all per chunk amortizes thread-pool overhead. Hybrid dispatch
in get_next_page_multi — chunk-level for narrow chunks (<= 6 value
columns), 4/6 thesis path otherwise to avoid cache thrash.
- Per-worker time decoder/compressor pool (via ThreadPool::
current_worker_id) parallelizes the previously-serial time-page
decode loop.
- Pre-decode int64/float/double values in the parallel worker into
ValueColumnState::pending_decoded_values; multi_DECODE_TV_BATCH then
memcpys the per-batch slice instead of calling the decoder inline.
- Partial-page bulk scatter: bulk-memcpy path now copies
min(budget, remaining_in_page) rows from page_time_cursor_ so the
tail page of every SSI tsblock takes the memcpy fast path instead of
bleeding into the row-by-row scatter loop.
- tsblock_max_memory_ 64KB -> 2MB so a 10K-row page fits in one SSI
tsblock and bulk_copy_into doesn't fragment into many tiny batches.
═════════════════════════════════════════════════════════════════════
Write path
═════════════════════════════════════════════════════════════════════
- ValuePageWriter gains write_batch / write_string_batch that take
timestamp + value + nullness arrays directly, removing the per-value
append loop. Tablet exposes set_timestamps / set_column_values /
set_column_string_repeated / reset for bulk reuse and switches
StringColumn to an Arrow-compatible offset+buffer layout.
- TS2DIFFEncoder::flush packs all deltas with a single pack_bits_msb +
write_buf instead of per-value write_bits, falling back to the scalar
path for the rare bit_width > 56 case.
- Int64Statistic::update_batch (NEON-accelerated min/max/sum).
═════════════════════════════════════════════════════════════════════
Encoding / SIMD
═════════════════════════════════════════════════════════════════════
- TS2DIFF batch decode adds AVX2 helpers via SIMDe (already on develop)
for both i32 and i64; scalar fallback unchanged.
- PLAIN byte-swap path uses ARM NEON (vrev64q_u8 / vrev32q_u8) when
available, falling back to __builtin_bswap.
- CMakeLists adds ENABLE_SIMD; Release builds turn on -O3 -march=native
-flto (off when ASan is on or on Windows/MinGW).
═════════════════════════════════════════════════════════════════════
Allocator / ByteStream / ThreadPool
═════════════════════════════════════════════════════════════════════
- ByteStream caches page_mask_ (= page_size - 1) so the hot path uses a
bitmask instead of modulo; wrap_from rounds buffer sizes up to a
power of two for correctness.
- common::ThreadPool gets a thread_local current_worker_id() accessor
(set by worker_loop) and a num_threads() getter, letting callers
attach per-worker state without contention.
═════════════════════════════════════════════════════════════════════
Build / platform
═════════════════════════════════════════════════════════════════════
- Linux Release: -march=native + -flto by default, automatically
dropped under ASan to keep leak detection accurate.
- MSVC / MinGW: replace GCC-only intrinsics, restore lost includes,
disable LTO + -march=native there.
- Restore tag_filter_create/between, metadata test, and segment
behavior; restore cwrapper metadata + tag_filter/batch_size args on
table query C APIs that the batch-opt snapshot had dropped.
- Disable QueryByRowPerformanceTest and the flaky
QueryByRowFasterThanManualNext test.
═════════════════════════════════════════════════════════════════════
Python binding
═════════════════════════════════════════════════════════════════════
- read_series_by_row: pull TsBlocks via Arrow IPC instead of the
row-by-row Python loop. Aligns reader query plumbing with develop
so the binding sees the same parameter set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0 commit comments