CASSANDRA-21134: Direct I/O for background SSTable writes

Summary

Opt-in O_DIRECT write path for background SSTable producers, bypassing the OS page cache for write-once read-never data. Memtable flushes remain buffered (hot data benefits from the cache).

background_write_disk_access_mode: direct    # default: standard
direct_write_buffer_size: 1MiB                # aligned up to FS block size; auto-grows to chunk_length

Gated by (1) config, (2) table compression enabled, (3) an OperationType allowlist (DataComponent#DIRECT_WRITE_SUPPORT). Selection is central in DataComponent.buildWriter; producers are unchanged.

Performance

Benchmark results are attached to the JIRA. Significant p99 read latency improvements under throttled compaction.

Operations covered (DIO eligible)

The allowlist is exhaustive: any new OperationType with writesData == true that is not classified fails static initialization (AssertionError).

Operations NOT covered

Removing an UNSUPPORTED_CORRECTNESS entry requires code changes; UNSUPPORTED_POLICY is a policy decision.

Key code

• io/DirectIoSupport.java — eligibility enum (SUPPORTED / UNSUPPORTED_CORRECTNESS /
  UNSUPPORTED_POLICY / NOT_APPLICABLE).
• io/sstable/format/DataComponent.java — selection, allowlist, exhaustiveness check.
• io/compress/DirectCompressedSequentialWriter.java — new writer; aligned buffers,
  mark()/resetAndTruncate() unsupported; durable-offset tracking (chunk-boundary ring
  buffer + post-flush listener) so preemptive early open binds only to whole-block durable
  offsets.
• io/compress/CompressedSequentialWriter.java — refactored so the DIO subclass can
  override the write-chunk path; writeChunk contract documented and asserted.
• config/Config.java, config/DatabaseDescriptor.java — new knobs, validation, startup
  wiring; buffer size aligned to FS block size, auto-grown to chunk length.
• service/StartupChecks.java — fails fast if direct is requested on a platform/FS
  that does not support O_DIRECT.

Tests introduced

• Property-based DIO-writer sweep — write/read integrity and on-disk byte-identity
  vs. the buffered writer over compressors × chunk lengths × random payload sizes;
  seed-logged for repro (DirectCompressedSequentialWriterTest).
• Durable-offset + fault-injection tests — property-test that the post-flush listener
  reports exactly the on-disk whole-block offsets (including under preemptive early open);
  fault-injecting FileChannel verifies write/truncate/position failures surface as
  FSWriteError/FSReadError with clean abort (DirectCompressedSequentialWriterTest).
• Parameterised buffer-size tests — three regimes pinning the distinct branches of
  flushCompleteBlocks (DirectCompressedSequentialWriterTest).
• Selection-matrix tests — per-OperationType eligibility, allowlist exhaustiveness,
  compression gate, config-mode gate (DataComponentDirectWriteSelectionTest).
• End-to-end coverage per allowlist arm — WRITE (CQLSSTableWriterDaemonTest) and
  the compaction family + ANTICOMPACTION (extended CompactionsTest,
  AntiCompactionTest).
• Regression guards — constructor channel-leak protection, non-power-of-two
  block-size rejection, once-per-JVM undersized-buffer warn, SCRUB-gating canaries
  (DirectCompressedSequentialWriterTest).
• Resource-leak detection — BufferPoolMXBean check that the off-heap aligned
  buffer is returned on close (DirectCompressedSequentialWriterTest).
• Config validation — new YAML knobs (mode parsing, buffer-size bounds, defaults)
  in DatabaseDescriptorTest.

Not in scope

• Uncompressed SSTable writers.
• ZCS streaming.

Reviewer notes

Findings from the Cassandra bug-hunting skills (Opus 4.7 xhigh & kimi-k2.6:cloud) were addressed prior to
review.