Skip to content

[STACKED on #1431] libext: ext4 sequential-read performance (read-ahead + async prefetch) - #1447

Draft
gburd wants to merge 9 commits into
cloudius-systems:masterfrom
gburd:pr/ext4-perf
Draft

[STACKED on #1431] libext: ext4 sequential-read performance (read-ahead + async prefetch)#1447
gburd wants to merge 9 commits into
cloudius-systems:masterfrom
gburd:pr/ext4-perf

Conversation

@gburd

@gburd gburd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Note

Stacked on #1431 (ext4 fsync + page-cache bridge). The two fsync/dtime
commits from #1431 appear at the base of this branch; once #1431 merges this
reduces to the four performance commits. Review #1431 first.

Closes most of the ext4 sequential-read throughput gap vs Linux, measured A/B on
a 2-socket x86-64 bare-metal host booting OSv under KVM against local NVMe
(no throughput cap).

The gap

Baseline ext4 (lwext4) vs Linux ext4 on the same NVMe:

metric OSv Linux note
seq write + fsync ~620 MB/s ~590 MB/s already at parity
4K random read ~37 MB/s ~28 MB/s already ≥ Linux
seq read (128K) ~560 MB/s ~2300 MB/s the gap

ext_read issued one synchronous bio per read() with no prefetch
(ext4_blocks_get_direct bypasses lwext4's block cache), so a stream of small
sequential reads left the NVMe queue idle during app compute. Linux keeps the
device busy via multi-window async read-ahead.

Changes (4 commits)

  1. benchmark (tst-ext4-bench): sequential write+fsync / read / 4K-random,
    MB/s, filesystem-agnostic. Writes an absolute-offset pattern and verifies
    every byte
    on the read, so it doubles as a read-path correctness test.
  2. direct user-buffer read/write: skip the per-op bounce allocation + extra
    uiomove copy on the single-iovec fast path; also fixes a latent
    free()/free_contiguous_aligned() mismatch. (Correctness cleanup — the
    copy was not the bottleneck.)
  3. synchronous 1 MiB read-ahead: on a sequential pattern, read one 1 MiB
    window and serve subsequent in-window reads from a memcpy. +73% at 64K.
  4. async double-buffered prefetch: two windows (cur/next) + a per-vnode
    prefetch worker that fills next while the app consumes cur, keeping the
    device queue busy. A worker thread (not raw bio_done) reuses
    ext_internal_read verbatim; writes/truncation invalidate, ~ext_vdata
    joins the worker and frees.

Results (bare-metal host, local NVMe, ext4 4K/no-journal, median of 3, no VERIFY FAIL)

block size sync read-ahead async prefetch speedup
64 KiB ~505 MB/s ~1248 MB/s 2.5x
128 KiB ~534 MB/s ~1260 MB/s 2.4x
256 KiB ~600 MB/s ~1250 MB/s 2.1x

Reference on the same NVMe: Linux ext4 buffered read ~2300 MB/s; NVMe
single-stream O_DIRECT ceiling 1.5 GB/s @128k, 1.9 GB/s @1m. OSv's async ext4
read now exceeds the 128K O_DIRECT single-stream ceiling and sits ~within 2x
of Linux buffered. The remaining gap is queue depth — Linux prefetches many
windows, this prefetches one ahead (depth ~2); a deeper prefetch pipeline is the
upgrade path.

Write stays at Linux parity and random read ≥ Linux, so no change was needed
there.

gburd added 9 commits July 31, 2026 09:09
Two gaps in the ext (lwext4) filesystem module for the Postgres-over-ext goal.

1. fsync durability.  ext mounts with lwext4 block-cache write-back enabled, so
   a write only reaches the disk when the block cache is flushed -- but
   vop_fsync was vop_nullop, so fsync(2)/fdatasync(2) on an ext file persisted
   nothing.  Implement ext_fsync() to flush the device's block cache (like
   ext_sync does at unmount), making written data durable.  The cache is shared
   per device, so this persists the file along with any other dirty buffers,
   which is correct if slightly more than the theoretical per-inode minimum.

2. Page-cache bridge (vop_cache).  The vop_cache slot was null, so mmap faults
   on ext files went through the block layer on every fault, unlike ROFS and
   ZFS which populate the shared page cache.  Add ext_map_cached_page(): on a
   VOP_CACHE call it reads one page-aligned page of file data into a freshly
   allocated page and hands it to pagecache::map_read_cached_page(), warming the
   read cache so subsequent faults and readahead are served from it.  This is an
   allocate-and-copy bridge (lwext4's block-cache buffers are not
   page-aligned/shareable the way ROFS's read-around cache is); a zero-copy
   borrow-and-pin bridge like the ZFS ARC one can follow if it shows up hot.

Because the module is built with -fno-rtti and <osv/pagecache.hh> pulls in
<osv/trace.hh> (which uses typeid), the two page-cache symbols we need are
declared minimally instead of including the heavy header (the uio carries the
hashkey opaquely, so its layout is never needed).

Add tests/tst-ext4-rw.cc: mmap a pre-populated ext4 file and verify the pattern
survives the vop_cache bridge (first fault + cached re-read), then write a file,
fsync it, and read it back (plus fdatasync and a fresh re-open).  Verified on
OSv under KVM with an ext4 second disk (created with mkfs.ext4 -b 4096
-O ^64bit,^metadata_csum, which lwext4 supports).

Known pre-existing limitation (not introduced here, out of scope for this PR):
libext's inode-delete path does not set the inode dtime, so Linux e2fsck flags
"deleted inode has zero dtime" on a disk after OSv deletes a file.  A fresh disk
that OSv only reads/writes+fsyncs (no delete) fscks clean.  Tracked as a
follow-up in the ext write-path correctness work.
Follow-up to the fsync/page-cache work: libext's inode-delete path freed the
inode from the bitmap but never set its on-disk deletion time (dtime), so after
OSv created and deleted a file, Linux e2fsck flagged "Deleted inode NN has zero
dtime" and reported the filesystem as still having errors.

lwext4's own delete path marks the inode with ext4_inode_set_del_time(inode,
-1L) before ext4_fs_free_inode(), but that symbol is not exported from
liblwext4.so.  Add a small ext_mark_inode_deleted() helper that sets
inode->deletion_time = 0xffffffff directly (byte-order invariant, so no
to_le32() needed) and call it before each ext4_fs_free_inode() in the module
(unlink, rmdir, delete-on-last-close, delete-outstanding-on-unmount, and the
dir_link allocation-rollback path).

Verified: after OSv boots an ext4 second disk, mmaps/reads a file, writes and
fsyncs another, then deletes it, `e2fsck -n -f` on the disk now exits 0 (clean),
where before it reported the zero-dtime error.  tst-ext4-rw still passes.
A filesystem-agnostic micro-benchmark (tst-ext4-bench) measuring sequential
write+fsync, sequential read, and 4K random read throughput in MB/s, so the
ext4/libext path can be compared A/B against raw virtio-blk and against Linux
ext4 on the same storage. Point its argument at any mount (ext, zfs, rofs) for
a cross-filesystem comparison.
ext_read allocated a full-size aligned bounce buffer, had lwext4 read into it,
then uiomove()'d it to the caller - a per-read malloc plus a second full-size
memcpy on every read; ext_write mirrored this in reverse.  On the common path
(a single contiguous iovec) hand lwext4 the caller's buffer directly.

This removes the allocation and one memcpy per read/write and fixes a latent
free()/free_contiguous_aligned() mismatch in both error paths.  It is a
correctness/cleanup change: an A/B on local NVMe showed throughput
unchanged (the memcpy is not the bottleneck), which confirms the sequential-read
gap vs Linux is the absence of async read-ahead, not copy overhead - that is a
separate, larger change (see the ext4 perf notes).
The local-NVMe A/B showed ext4 sequential reads at ~33% of Linux while writes
were at parity and random reads ~86%. The gap is that ext_read issues one
synchronous bio per read() with no prefetch (ext4_blocks_get_direct bypasses
lwext4's block cache), so a stream of small sequential reads cannot keep the
device pipeline busy - larger reads were measurably faster purely from
amortizing the per-read() round-trip.

Add a per-vnode sequential read-ahead cache (in ext_vdata): on a detected
sequential single-iovec read smaller than the window, fill a 1 MiB window from
disk once and serve that read and subsequent in-window reads from it (a memcpy,
no I/O). This turns N small synchronous reads into 1 large read + N copies.
The window is invalidated on any write to the file and freed when the vnode
goes inactive; access is serialized by a per-vnode mutex.

tst-ext4-bench now writes an absolute-offset pattern and verifies every byte on
the sequential read, so the benchmark doubles as a read-ahead correctness test
(no VERIFY FAIL = the cache returns correct data across window boundaries).
The synchronous 1 MiB read-ahead (6a7990d) reads a window, lets the app
consume it, then reads the next window - so the NVMe queue goes idle
during the app's compute. Linux keeps the device busy via async prefetch.

Add double buffering: two 1 MiB windows (cur + next) and a per-vnode
worker pthread. While the app consumes cur (served by memcpy), the worker
prefetches next in the background. When a read crosses out of cur, next is
promoted to cur and the following window is kicked off. On a sequential
pattern the next window is already in memory, keeping the device queue
full and hiding read latency behind compute.

Correctness: writes and truncation cancel any in-flight prefetch and drop
both windows (ext_ra_invalidate); the worker uses its own inode_ref
(lwext4 locks its block cache internally); ~ext_vdata joins the worker and
frees both buffers on vnode inactive. The tst-ext4-bench byte-for-byte
verify passes (no VERIFY FAIL) for 64K/128K/256K reads.
The async double-buffered read-ahead only kept ~1-2 windows ahead of the
consumer, so a single sequential stream drove the NVMe queue to depth ~1-2
and topped out ~1.25 GB/s vs Linux's ~2.3 GB/s (Linux issues many concurrent
readahead requests, keeping the device queue deep).

Replace the two-window (cur/next) scheme with a ring of RA_WINDOWS windows
(256 KiB x 8 = 2 MiB per open file, same memory as before) filled by a pool
of RA_WORKERS worker pthreads. Each worker calls the existing synchronous
ext_internal_read() into a ring slot, so RA_WORKERS fills are in flight at
once -> device queue depth ~RA_WORKERS. As soon as the consumer drains a
window the ring slides forward and re-arms that slot for the window
RA_WINDOWS ahead, keeping the pipeline full.

Correctness: per-slot state (EMPTY/FILLING/READY) + a ring generation. All
worker-shared fields are under ra_cvmtx; the read path holds ra_lock (outer)
and takes ra_cvmtx to inspect slots. Workers never take ra_lock, so no
deadlock. Seek/write/truncate bump the generation and drain in-flight fills
before reusing buffers (no use-after-free); stale worker results whose
generation no longer matches are discarded. Reads larger than a window or
spanning two windows fall through to the direct read path. ~ext_vdata
broadcasts shutdown, joins all workers, then frees the buffers.

Reuses lwext4's existing bio path (ext_internal_read -> ext4_blocks_get_direct
-> one bio + bio_wait) rather than issuing raw bios, which would mean
re-implementing lwext4's extent->block mapping; a worker pool blocked in
bio_wait maps directly to device queue depth and keeps all that code correct.

Verified with tests/tst-ext4-bench.cc (absolute-offset byte verification) at
bs 4K/64K/128K/256K/262143/1M and files smaller than the ring: no VERIFY
FAIL.
Sweep on a local-NVMe host (, ext4 4K/no-journal, O_DIRECT so reads hit the
device, 512 MiB, median of 3):

  config        64K      128K     256K   prefetch mem/file
  N=1  M=1     ~0.69    ~0.85    ~0.85     256 KiB   (~QD1)
  N=4  M=2     ~1.20    -        ~1.28       1 MiB
  N=4  M=4     ~1.26    ~1.23    ~1.29       1 MiB   <- knee
  N=8  M=4     ~1.28    ~1.28    ~1.31       2 MiB
  N=8  M=8     ~1.28    ~1.28    ~1.29       2 MiB
  N=16 M=16    ~1.25    -        ~1.21       4 MiB

Throughput climbs steeply from N=1/M=1 to the knee at N=4/M=4 (~1.28 GB/s) and
then plateaus: deeper rings or more workers give no further gain. The cap is
downstream of this code -- OSv's virtio-blk make_request() serialises submission
under a single _lock and a single virtqueue, so effective device queue depth
tops out at ~2-4 regardless of how many prefetch windows are in flight. (Linux
fio O_DIRECT on the same raw NVMe: ~1.32 GB/s @128k QD1, ~1.68 GB/s @128k QD>=2,
i.e. the device itself saturates at QD2; the remaining OSv gap is the guest
virtio-blk path, not prefetch depth.)

So N=4/M=4 is the sweet spot: same throughput as the deeper configs at half the
memory (1 MiB vs 2 MiB per open file). It also matches the prior 2-window async
design (~1.25 GB/s here) while using a general N-window ring. Make N and M
build-time tunables (-DRA_WINDOWS_N / -DRA_WORKERS_M via $(RA_FLAGS)) so the
sweep is reproducible when the virtio-blk submission path is later parallelised.

No VERIFY FAIL at any config or block size (4K/64K/128K/256K/262143/1M, files
smaller than the ring).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant