`msync()` returns `ENOMEM` (errno 12) for high-VA guest mappings created by Rosetta

[doanbaotrung]

Environment

• Host: macOS on Apple Silicon (M-series)
• Guest arch: x86_64 (run via Rosetta translation)
• Trigger: Running apt update or apt upgrade inside an x86_64 Linux guest prefix, or any guest program that msyncs a large mmap-backed cache

Problem Description

sys_msync in src/syscall/mem.c validates the guest virtual address by comparing it against g->guest_size (the primary memory region size) and g->ipa_base (the IPA base of the primary region). Addresses that fall outside that primary region are rejected with -LINUX_ENOMEM:

if (addr < g->ipa_base)
    return -LINUX_ENOMEM;

uint64_t off = addr - g->ipa_base;
/* sys_msync stays primary-only here for the same reason as madvise: msync
 * iterates regions and reaches into host_base+off to read pages back from
 * the file overlay. Widening to extra-region ranges needs a region-aware
 * iterator landing alongside the data-movement refactor.
 */
if (off > g->guest_size || length > g->guest_size - off)
    return -LINUX_ENOMEM;
uint64_t end = off + length;

Under Rosetta translation, the Rosetta runtime allocates its JIT code cache and internal slabs at high virtual addresses — typically above 0x00007FFFFFFFFFFF (the normal 48-bit user-space boundary on Linux x86_64). These mappings are added to g->regions via guest_add_mapping, so they are known to elfuse, but they have start values far beyond g->guest_size. The addr - g->ipa_base subtraction either underflows (if addr < g->ipa_base) or produces an offset that exceeds g->guest_size, causing both branches to return -LINUX_ENOMEM.

Furthermore, even when the bounds check passes for a legitimate high-VA region, the host-pointer computation in the region-iterating helpers was wrong:

sync_shared_aliases_range computed the guest host pointer as:

const uint8_t *guest = (const uint8_t *) g->host_base + src->start +
                       (wfile_start - src->offset);

This adds src->start (a raw GVA for high-VA mappings) to g->host_base, producing a completely wrong host address.

refresh_shared_region_range had the same problem:

uint64_t guest_off = r->start + (rfile_start - r->offset);
uint8_t *buf = (uint8_t *) g->host_base + guest_off;

Both helpers assumed all regions are in the primary buffer (host_base + offset), which is only correct for regions whose start is a primary-buffer-relative offset (i.e., start < guest_size).

Observed Failure

apt memory-maps its package-list cache files (the files under /var/lib/apt/lists/) with mmap(MAP_SHARED) and periodically flushes them with msync. On an x86_64 guest under Rosetta, the apt process virtual address space contains Rosetta high-VA mappings. When apt calls msync on its cache, the guest VA falls in the Rosetta high-VA range and sys_msync returns -LINUX_ENOMEM. apt surfaces this as:

E: Unable to synchronize mmap - msync (12: Cannot allocate memory)

apt update and apt upgrade both fail at package list parsing before any packages can be installed.