perf(ext4): batch nojournal range allocation (#2135)

* perf(ext4): batch nojournal range allocation

CubeSandbox tmp writes on nojournal ext4 allocated and published every missing 4 KiB block independently. Each block synchronously rewrote allocation metadata, zeroed data, and updated the inode, keeping virtio-blk at queue depth one and amplifying fixed request latency.

Add a bounded DirectRangeStage for depth-zero contiguous appends. Plan allocation metadata in transaction-private images, bulk-zero the complete range, enforce a reliable pre-publication flush, and publish bitmap, group descriptor, superblock, and inode homes in semantic order. Runtime failures restore allocation homes when safe and poison the direct backend when inode state or rollback durability is uncertain.

Add fallible DMA/BIO construction, exact block I/O completion handling, bounded contiguous ext4 block transfers, and compile-time-disabled block/ext4 diagnostic counters. Preserve journaled, fragmented, sparse, small-write, unsupported-flush, and resource-pressure behavior through explicit legacy fallbacks.

Cover extent validation, unwritten tails, allocation bounds, zero and flush failures, every allocation-home failure, rollback failures, inode uncertainty, and transaction snapshot ownership. The nojournal crash contract remains unchanged: abnormal power loss still requires fsck.

Validation: make fmt; cargo test --manifest-path kernel/crates/another_ext4/Cargo.toml --lib; RUST_MIN_STACK=33554432 make kernel; local nojournal ext4 hash and e2fsck; CubeSandbox fresh-volume A1/B/A2 performance and hash checks.
Signed-off-by: longjin <longjin@dragonos.org>

* fix(ext4): address direct range review findings

Make nojournal direct-range publication flush allocation metadata before writing the inode home block. If the durability barrier fails, restore and flush every allocation home; poison the backend when rollback durability cannot be established. Add fault-injection coverage for the new ordering and failure states.

Keep coherent DMA memory in the normal write-back direct mapping instead of changing RAM cache attributes without architecture cache maintenance. Reject unsupported non-coherent cache policies explicitly, removing partial multi-page remaps and TLB shootdowns from allocation and interrupt-context release paths.

Prune expired ext4 statistics registry entries during mount registration so repeated mount/unmount cycles do not retain dead Weak control blocks.

Signed-off-by: longjin <longjin@dragonos.org>

* fix(io): address direct range review findings

Classify nojournal direct-range candidates under the compatible metadata gate before requesting an exclusive transaction snapshot. Unsupported small, overwrite, sparse, and oversized writes now continue directly through the legacy allocator, while real candidates are re-read and revalidated after exclusive acquisition to avoid stale plans.

Restore zero-length virtio-blk synchronous I/O semantics by returning Ok(0) before buffer validation or BIO submission. Keep zero-length read/write BIOs invalid as an internal submission invariant.

Add direct-range eligibility coverage for the optimized and legacy request shapes.

Signed-off-by: longjin <longjin@dragonos.org>

---------

Signed-off-by: longjin <longjin@dragonos.org>

Author: fslongjin

kernel/crates/another_ext4/src/ext4/alloc.rs

@@ -4,6 +4,14 @@ use crate::ext4_defs::*;
 use crate::format_error;
 use crate::prelude::*;
 use crate::return_error;
+use core::cmp::min;
+
+pub(super) struct DirectRangeAllocation {
+    pub first: PBlockId,
+    pub allocation_homes: [PBlockId; 3],
+}
+
+const DIRECT_RANGE_MAX_GROUP_PROBES: u32 = 4;
 
 fn extent_tail_batch_limit(
     first_data_block: PBlockId,
@@ -71,6 +79,135 @@ impl Ext4 {
         core::cmp::min(sb.blocks_per_group() as u64, total - first) as usize
     }
 
+    /// Stage one group-local contiguous data allocation without publishing
+    /// any cache or disk metadata.  The caller owns the filesystem-wide
+    /// transactional metadata gate, so no direct allocator can race the
+    /// transaction-private bitmap snapshot while zero I/O is in progress.
+    pub(super) fn transaction_alloc_direct_range(
+        &self,
+        transaction: &mut super::journal_transaction::Transaction<'_>,
+        inode_id: InodeId,
+        preferred_first: Option<PBlockId>,
+        require_preferred: bool,
+        count: u32,
+    ) -> Result<DirectRangeAllocation> {
+        if count == 0 {
+            return_error!(ErrCode::EINVAL, "Cannot allocate an empty block range");
+        }
+        let mut sb = self.transaction_read_super_block(transaction)?;
+        self.prepare_stats.record_superblock_io();
+        let bg_count = sb.block_group_count();
+        let preferred_inode_group = ((inode_id - 1) / sb.inodes_per_group()) as BlockGroupId;
+        let preferred_group = preferred_first
+            .filter(|block| *block >= sb.first_data_block() as PBlockId)
+            .map(|block| {
+                ((block - sb.first_data_block() as PBlockId) / sb.blocks_per_group() as PBlockId)
+                    as BlockGroupId
+            })
+            .filter(|group| *group < bg_count)
+            .unwrap_or(preferred_inode_group);
+        let count_usize = count as usize;
+
+        for offset in 0..min(bg_count, DIRECT_RANGE_MAX_GROUP_PROBES) {
+            let bgid = ((preferred_group as u64 + offset as u64) % bg_count as u64) as BlockGroupId;
+            let blocks_in_group = Self::block_group_block_count(&sb, bgid);
+            if blocks_in_group < count_usize {
+                continue;
+            }
+            let mut bg = self.transaction_read_block_group(transaction, bgid)?;
+            self.prepare_stats.record_gdt_io();
+            if bg.desc.get_free_blocks_count() < count as u64 {
+                continue;
+            }
+            let bitmap_home = bg.desc.block_bitmap_block();
+            let bitmap_block = transaction.read(self.block_device.as_ref(), bitmap_home)?;
+            self.prepare_stats.record_bitmap_io();
+            let checksum_bytes = (sb.clusters_per_group() as usize) / 8;
+            if sb.has_read_only_compatible_feature(SuperBlock::FEATURE_RO_COMPAT_METADATA_CSUM) {
+                if !bg.verify_checksum(sb.metadata_checksum_seed()) {
+                    return_error!(ErrCode::EIO, "Corrupt block-group descriptor checksum");
+                }
+                if !bg.desc.verify_block_bitmap_csum(
+                    sb.metadata_checksum_seed(),
+                    &*bitmap_block,
+                    checksum_bytes,
+                ) {
+                    return_error!(ErrCode::EIO, "Corrupt block bitmap checksum");
+                }
+            }
+
+            let group_first = Self::block_group_first_block(&sb, bgid);
+            let exact_hint = preferred_first
+                .filter(|_| offset == 0)
+                .and_then(|block| block.checked_sub(group_first))
+                .and_then(|bit| usize::try_from(bit).ok())
+                .filter(|bit| {
+                    bit.checked_add(count_usize)
+                        .is_some_and(|end| end <= blocks_in_group)
+                        && (*bit..*bit + count_usize)
+                            .all(|index| bitmap_block[index / 8] & (1 << (index % 8)) == 0)
+                });
+            let bit = exact_hint.or_else(|| {
+                (!require_preferred)
+                    .then(|| {
+                        Bitmap::first_clear_run_in(
+                            &*bitmap_block,
+                            blocks_in_group,
+                            0,
+                            blocks_in_group,
+                            count_usize,
+                        )
+                    })
+                    .flatten()
+            });
+            let Some(bit) = bit else { continue };
+            let first = group_first
+                .checked_add(bit as PBlockId)
+                .ok_or_else(|| Ext4Error::new(ErrCode::EFBIG))?;
+            self.validate_data_blocks(first, count as u64)?;
+            let new_bg_free = bg
+                .desc
+                .get_free_blocks_count()
+                .checked_sub(count as u64)
+                .ok_or_else(|| Ext4Error::new(ErrCode::EIO))?;
+            let new_sb_free = sb
+                .free_blocks_count()
+                .checked_sub(count as u64)
+                .ok_or_else(|| Ext4Error::new(ErrCode::EIO))?;
+
+            {
+                let image = self.transaction_block_for_update(transaction, bitmap_home)?;
+                self.prepare_stats.record_bitmap_io();
+                let mut bitmap = Bitmap::new(image, blocks_in_group);
+                if (bit..bit + count_usize).any(|index| !bitmap.is_bit_clear(index)) {
+                    return_error!(ErrCode::EIO, "Direct range changed during planning");
+                }
+                for index in bit..bit + count_usize {
+                    bitmap.set_bit(index);
+                }
+                if !bg.desc.update_block_bitmap_csum(
+                    sb.metadata_checksum_seed(),
+                    image,
+                    checksum_bytes,
+                ) {
+                    return_error!(ErrCode::EIO, "Invalid block bitmap checksum length");
+                }
+            }
+            bg.desc.set_free_blocks_count(new_bg_free);
+            self.transaction_stage_block_group_with_csum(transaction, &mut bg)?;
+            self.prepare_stats.record_gdt_io();
+            sb.set_free_blocks_count(new_sb_free);
+            self.transaction_stage_super_block(transaction, &sb)?;
+            self.prepare_stats.record_superblock_io();
+            let (gdt_home, _) = self.block_group_disk_pos(bgid)?;
+            return Ok(DirectRangeAllocation {
+                first,
+                allocation_homes: [bitmap_home, gdt_home, 0],
+            });
+        }
+        return_error!(ErrCode::ENOSPC, "No contiguous direct range available");
+    }
+
     /// Stage the release of one contiguous physical-block range.
     ///
     /// This changes allocation metadata only.  In particular, freed data is
@@ -764,6 +901,7 @@ impl Ext4 {
             // extent physical block numbers are absolute filesystem block numbers.
             let bitmap_block_id = bg.desc.block_bitmap_block();
             let mut bitmap_block = self.read_block(bitmap_block_id)?;
+            self.prepare_stats.record_bitmap_io();
             let old_bitmap_block = bitmap_block.clone();
             let old_bg = BlockGroupRef::new(bg.id, bg.desc);
             let old_sb = sb;
@@ -785,6 +923,7 @@ impl Ext4 {
                 return_error!(ErrCode::EIO, "Invalid block bitmap checksum length");
             }
             self.write_block(&bitmap_block)?;
+            self.prepare_stats.record_bitmap_io();
 
             // Update block group counters
             bg.desc
@@ -844,6 +983,7 @@ impl Ext4 {
             }
             return Err(init_error);
         }
+        self.prepare_stats.record_zero_io();
         Ok(pblock)
     }
 

kernel/crates/another_ext4/src/ext4/extent.rs

@@ -26,7 +26,104 @@ pub(super) struct ExtentTail {
     pub unwritten: bool,
 }
 
+pub(super) struct DirectAppendShape {
+    pub preferred_first: Option<PBlockId>,
+    pub requires_merge: bool,
+}
+
 impl Ext4 {
+    pub(super) fn direct_append_shape(
+        &self,
+        inode: &InodeRef,
+        start_lblock: LBlockId,
+        count: u32,
+    ) -> Result<Option<DirectAppendShape>> {
+        if count == 0 || count > u16::MAX as u32 || !inode.inode.uses_extents() {
+            return Ok(None);
+        }
+        let root = inode.inode.extent_root();
+        let header = root.header();
+        self.validate_extent_node(inode.id, &root)?;
+        if header.depth() != 0
+            || header.max_entries_count() as usize != root.entry_capacity()
+            || header.entries_count() as usize > root.entry_capacity()
+        {
+            return Ok(None);
+        }
+        let entries = header.entries_count() as usize;
+        if entries == 0 {
+            return Ok((start_lblock == 0).then_some(DirectAppendShape {
+                preferred_first: None,
+                requires_merge: false,
+            }));
+        }
+        let last = root.extent_at(entries - 1);
+        if last.is_unwritten() {
+            return Ok(None);
+        }
+        for index in 0..entries {
+            let extent = root.extent_at(index);
+            self.validate_data_blocks(extent.start_pblock(), extent.block_count() as u64)?;
+        }
+        let next_lblock = last
+            .start_lblock()
+            .checked_add(last.block_count())
+            .ok_or_else(|| Ext4Error::new(ErrCode::EFBIG))?;
+        if next_lblock != start_lblock {
+            return Ok(None);
+        }
+        let preferred_first = last
+            .start_pblock()
+            .checked_add(last.block_count() as PBlockId)
+            .ok_or_else(|| Ext4Error::new(ErrCode::EFBIG))?;
+        let candidate = Extent::new(start_lblock, preferred_first, count as u16);
+        let can_merge = Extent::can_append(last, &candidate);
+        let root_full = entries == header.max_entries_count() as usize;
+        if root_full && !can_merge {
+            return Ok(None);
+        }
+        Ok(Some(DirectAppendShape {
+            preferred_first: Some(preferred_first),
+            requires_merge: root_full,
+        }))
+    }
+
+    pub(super) fn stage_direct_append_extent(
+        &self,
+        inode: &mut InodeRef,
+        start_lblock: LBlockId,
+        start_pblock: PBlockId,
+        count: u32,
+    ) -> Result<()> {
+        let new_extent = Extent::new(start_lblock, start_pblock, count as u16);
+        let mut root = inode.inode.extent_root_mut();
+        let entries = root.header().entries_count() as usize;
+        if entries > 0 {
+            let last = *root.extent_at(entries - 1);
+            if Extent::can_append(&last, &new_extent) {
+                root.extent_mut_at(entries - 1)
+                    .set_block_count(last.block_count() + count);
+            } else if root.insert_extent(&new_extent, entries).is_err() {
+                return Err(format_error!(
+                    ErrCode::ENOTSUP,
+                    "Inline extent root requires a split"
+                ));
+            }
+        } else if root.insert_extent(&new_extent, 0).is_err() {
+            return Err(format_error!(
+                ErrCode::ENOTSUP,
+                "Inline extent root requires a split"
+            ));
+        }
+        let blocks = inode
+            .inode
+            .fs_block_count()
+            .checked_add(count as u64)
+            .ok_or_else(|| Ext4Error::new(ErrCode::EFBIG))?;
+        inode.inode.set_fs_block_count(blocks);
+        Ok(())
+    }
+
     fn verify_extent_block_checksum(&self, inode_ref: &InodeRef, image: &[u8]) -> Result<()> {
         let sb = self.read_super_block_cached();
         if !sb.has_read_only_compatible_feature(SuperBlock::FEATURE_RO_COMPAT_METADATA_CSUM) {
@@ -74,6 +171,7 @@ impl Ext4 {
         self.ensure_valid_pblock(inode_ref.id, pblock, "extent tree node")?;
         self.validate_data_blocks(pblock, 1)?;
         let block = self.read_block(pblock)?;
+        self.prepare_stats.record_extent_io();
         self.verify_extent_block_checksum(inode_ref, &block.data[..])?;
         Ok(block)
     }
@@ -345,6 +443,7 @@ impl Ext4 {
         // Write checksum into the tail
         block.data[tail_offset..tail_offset + 4].copy_from_slice(&csum.to_le_bytes());
         self.write_block(block)
+            .inspect(|_| self.prepare_stats.record_extent_io())
     }
 }
 
@@ -862,6 +961,15 @@ impl Ext4 {
                 inode_id
             ));
         }
+        if header.max_entries_count() as usize > ex_node.entry_capacity()
+            || header.entries_count() as usize > ex_node.entry_capacity()
+        {
+            return Err(format_error!(
+                ErrCode::EIO,
+                "extent header exceeds node capacity on inode {}",
+                inode_id
+            ));
+        }
         let entries = header.entries_count() as usize;
         if header.depth() > 0 {
             if entries == 0 {
@@ -1000,6 +1108,7 @@ mod tests {
             inode_mutation_locks: (0..crate::ext4::INODE_MUTATION_LOCK_SHARDS)
                 .map(|_| spin::Mutex::new(()))
                 .collect(),
+            prepare_stats: crate::ext4::PrepareStats::new(),
         }
     }
 
@@ -1031,6 +1140,7 @@ mod tests {
             inode_mutation_locks: (0..crate::ext4::INODE_MUTATION_LOCK_SHARDS)
                 .map(|_| spin::Mutex::new(()))
                 .collect(),
+            prepare_stats: crate::ext4::PrepareStats::new(),
         }
     }
 
@@ -1092,6 +1202,36 @@ mod tests {
         assert_eq!(err.code(), ErrCode::EIO);
     }
 
+    #[test]
+    fn validate_extent_node_rejects_header_larger_than_inline_storage() {
+        let fs = make_test_fs(1024);
+        let mut raw = [0u8; 60];
+        ExtentNodeMut::from_bytes(&mut raw).init(0, 0);
+        raw[2..4].copy_from_slice(&5u16.to_le_bytes());
+        raw[4..6].copy_from_slice(&5u16.to_le_bytes());
+        let node = ExtentNode::from_bytes(&raw);
+
+        let error = fs.validate_extent_node(5, &node).unwrap_err();
+
+        assert_eq!(error.code(), ErrCode::EIO);
+    }
+
+    #[test]
+    fn direct_append_rejects_unwritten_tail() {
+        let fs = make_test_fs(1024);
+        let mut inode = InodeRef::new(6, Box::new(Inode::default()));
+        inode.inode.extent_init();
+        let mut extent = Extent::new(0, 100, 16);
+        extent.mark_unwritten();
+        inode
+            .inode
+            .extent_root_mut()
+            .insert_extent(&extent, 0)
+            .unwrap();
+
+        assert!(fs.direct_append_shape(&inode, 16, 16).unwrap().is_none());
+    }
+
     #[test]
     fn trim_depth_zero_partial_extent_preserves_prefix() {
         let mut raw = [0u8; 60];

kernel/crates/another_ext4/src/ext4/journal.rs

@@ -146,10 +146,15 @@ impl Ext4 {
         Ok(())
     }
 
-    pub(super) fn uses_journal(&self) -> bool {
+    pub fn uses_journal(&self) -> bool {
         matches!(self.metadata_mode, MetadataMutationMode::Journal(_))
     }
 
+    pub(super) fn supports_direct_range_stage(&self) -> bool {
+        matches!(self.metadata_mode, MetadataMutationMode::Direct(_))
+            && self.block_device.supports_reliable_flush()
+    }
+
     pub fn shutdown_writable(&self) -> Result<()> {
         // Clearing RECOVER is a metadata state transition and must not race a
         // direct writer even if the VFS normally excludes writers at umount.
@@ -199,6 +204,17 @@ impl Ext4 {
         }
     }
 
+    pub(super) fn transaction_start_direct_range(
+        &self,
+        credits: usize,
+    ) -> Result<journal_transaction::Transaction<'_>> {
+        match &self.metadata_mode {
+            MetadataMutationMode::Direct(core) => core.start_direct_range(credits),
+            MetadataMutationMode::ReadOnly => Err(Ext4Error::new(ErrCode::EROFS)),
+            MetadataMutationMode::Journal(_) => Err(Ext4Error::new(ErrCode::ENOTSUP)),
+        }
+    }
+
     pub(super) fn initialize_journal(&mut self) -> Result<()> {
         if !self.block_device.supports_reliable_flush() {
             return Err(Ext4Error::new(ErrCode::ENOTSUP));