vm-memory with perisistent memory

[shenben]

Are you aware of any ongoing projects involving the integration of non-volatile memory into guest memory? Alternatively, do you have any intentions to incorporate non-volatile memory into the vm-memory project? Thanks.

[bonzini]

Non-volatile memory is just map, so there should be no problems other than issuing the appropriate msync system calls or cache write back instructions. Those could be added to Bytes<> or VolatileSlice<>.

[bhavesh51]

Hi @bonzini, @roypat

I'd like to work on this issue. Building on the direction from the earlier discussion
(persistent memory is just mmap, the missing piece is msync/cache writeback), here's a
concrete design proposal. I'd appreciate feedback before I open a PR.

Problem

VMMs that expose NVDIMM or virtio-pmem devices to guests need a way to flush dirty data
from CPU caches or the host page cache to the persistent backing medium. Currently vm-memory
has no API for this. Anyone implementing persistent memory support has to work around the
crate by calling msync directly on raw pointers, bypassing all the safety abstractions
this crate provides.

Proposed design

Why VolatileSlice is the wrong place

VolatileSlice holds only a raw NonNull<u8>, a size, and a bitmap. It has no reference
back to the parent MmapRegion or its FileOffset. Without the backing metadata,
VolatileSlice cannot determine whether the memory is file-backed persistent storage or
anonymous, and cannot issue a valid msync call (which requires page-aligned addresses
within a known mapping).

Where persistence belongs: MmapRegion

MmapRegion knows:

• Whether the region has a FileOffset (file-backed vs anonymous)
• The base address and full size of the mmap
• The mapping flags used at creation time

This makes it the natural place for persistence methods.

Proposed trait

/// Trait for memory regions that can flush data to persistent storage.
///
/// This is relevant for NVDIMM-backed and virtio-pmem-backed guest memory
/// regions where the VMM must ensure writes are durable after a guest flush
/// request (e.g. virtio-pmem VIRTIO_PMEM_REQ_TYPE_FLUSH).
pub trait Persistable {
    /// Flush `len` bytes starting at `offset` to durable storage.
    ///
    /// For file-backed mappings this calls `msync(MS_SYNC)`.
    /// Returns `Ok(())` as a no-op for non-persistent regions.
    fn persist(&self, offset: usize, len: usize) -> std::io::Result<()>;

    /// Flush the entire region to durable storage.
    fn persist_all(&self) -> std::io::Result<()>;

    /// Returns true if this region is backed by persistent storage.
    fn is_persistent(&self) -> bool;
}

Implementation for MmapRegion

impl<B: BitmapSlice> Persistable for MmapRegion<B> {
    fn persist(&self, offset: usize, len: usize) -> io::Result<()> {
        if !self.is_persistent() || len == 0 {
            return Ok(());
        }
        let end = offset.checked_add(len)
            .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "overflow"))?;
        if end > self.size() {
            return Err(io::Error::new(io::ErrorKind::InvalidInput, "out of bounds"));
        }
        let base = self.as_ptr() as usize + offset;
        let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) } as usize;
        let aligned = base & !(page_size - 1);
        let ret = unsafe {
            libc::msync(aligned as *mut libc::c_void, (base + len) - aligned, libc::MS_SYNC)
        };
        if ret == 0 { Ok(()) } else { Err(io::Error::last_os_error()) }
    }

    fn persist_all(&self) -> io::Result<()> {
        self.persist(0, self.size())
    }

    fn is_persistent(&self) -> bool {
        self.file_offset().is_some()
    }
}

Forwarding through GuestRegionMmap

impl<B: BitmapSlice> Persistable for GuestRegionMmap<B> {
    fn persist(&self, offset: usize, len: usize) -> io::Result<()> {
        self.mapping().persist(offset, len)
    }
    fn persist_all(&self) -> io::Result<()> {
        self.mapping().persist_all()
    }
    fn is_persistent(&self) -> bool {
        self.mapping().is_persistent()
    }
}

Open questions for maintainers

1. Separate trait vs extending Bytes<>: I went with a separate Persistable trait
   for clean separation of concerns. The earlier suggestion was to add to Bytes<> or
   VolatileSlice<>. Would you prefer persistence methods directly on Bytes<>?

2. Feature gate: Should this be behind a feature flag (e.g. persistent-memory) or
   always available? msync is available on all Unix, so gating may not be necessary.

3. clflush/clwb path: For real NVDIMM DAX mappings, msync is insufficient --
   x86 needs clflush/clflushopt/clwb instructions, ARM needs dc cvap. Should
   this be part of the initial PR or a follow-up?

4. GuestMemory-level API: Should there be a persist(GuestAddress, len) method on
   GuestMemory that resolves the address to a region and delegates? This would match the
   existing Bytes<GuestAddress> pattern.

About me

I work with KVM-based virtualization infrastructure (OpenShift Virtualization, KubeVirt)
and have experience with Rust systems programming, container runtimes, and VMM internals.

Happy to adjust the design based on your feedback. I'll submit a PR once we align on the
approach.