[DRAFT / tracking - DO NOT MERGE as one unit] PostgreSQL-on-OSv fork+ZFS integration (will split into S1..S6 on #1455/#1456/#1457 + #1423) - #1458
Draft
gburd wants to merge 65 commits into
Conversation
…pt-in)
OSv is a single-address-space unikernel, so a literal copy-on-write fork() is
not the model. This adds a thread-backed fork() emulation that covers the
useful, compatible subset of fork semantics, gated entirely behind a new
CONFIG_fork kconfig option that defaults to OFF - so a default OSv build is
unchanged (none of this code is compiled in and fork()/vfork() return ENOSYS as
before).
With CONFIG_fork enabled:
- fork()/vfork() create a child OSv thread that resumes in fork()'s caller and
returns 0 in the child / the child pid in the parent (the classic twin
return). The child runs on a private copy of the parent's user stack, so
parent and child have independent locals after the return; it gets its own
fresh OSv per-thread TLS block (own errno etc).
Implemented in libc/process/fork.cc + arch/{x64,aarch64}/fork.cc.
- execve() launches the target as a fresh OSv application (its own ELF
namespace) and does not return, making fork()+exec() work.
- waitpid()/wait4()/wait() reap a child's exit status via a pid->child
registry; SIGCHLD is raised to the parent on child exit; SIGCHLD/SIGURG/
SIGWINCH now correctly default to ignore (not poweroff).
- exit()/_exit() in a fork child ends only that child, not the whole unikernel.
- pthread_atfork prepare/parent/child handlers are now actually run around
fork() (they were a no-op stub) - glibc/musl register these internally.
- sys_clone() routes the non-CLONE_THREAD (fork) case here when enabled.
Validated: tst-fork passes 10/10 on both x86-64 and aarch64 (twin return,
private-stack isolation, fork+exec, vfork, waitpid reaping).
Documented limitations (documentation/fork.md): the child shares the parent's
heap/globals (no per-process memory isolation in one address space - a follow-up
adds per-child copy-on-write address spaces behind the same flag); deep-call-
chain child unwind and fork-as-memory-snapshot (Redis BGSAVE) are not carried by
this base; execve()'s new-ELF-namespace path has a separate pre-existing fault.
Copyright (C) 2026 Greg Burd
execve() (CONF_fork) launches the target as a fresh OSv application in its own
ELF namespace via application::run(new_program=true) and, matching Linux, does
not return to the caller. A successful exec DID launch the program (the
elf::program new-namespace construction path is fine), but after the exec'd
program finished the unikernel would hang at shutdown instead of powering off:
the loader's application::join() blocked forever on the top-level app's
_terminated flag.
Root cause is the fork child thread's lifecycle, not execve or elf::program.
fork_thread() (arch/{x64,aarch64}/fork.cc) creates the child as a normal
*attached* sched::thread. At construction the thread captures a shared_ptr to
the current application's application_runtime (sched.cc sets _app_runtime =
app->runtime()). Nothing ever join()s the fork child -- the parent reaps it
through the fork pid registry / waitpid(), not sched::thread::join() -- so the
thread object is never destroyed and its _app_runtime shared_ptr is never
released. With that reference outstanding the application_runtime's use count
never reaches zero, ~application_runtime never runs, the app's _terminated is
never set, and application::join() waits forever. (This bit every fork(), and
was most visible after fork()+execve() where the child adopts the caller app's
runtime.)
Fix: create the fork child detached and dispose it in its cleanup.
- arch/{x64,aarch64}/fork.cc: mark the child attr().detached(), so on
completion it is handed to the thread reaper (which runs its set_cleanup()),
rather than sitting forever waiting to be joined.
- libc/process/fork.cc: the child's cleanup now also calls
sched::thread::dispose(child) after the existing bookkeeping (record exit
status if it fell off the end, free the copied user stack). Disposing the
thread releases its _app_runtime reference, letting the owning application's
runtime drop to zero so join() completes and OSv powers off. This mirrors
the default detached-thread cleanup ([this]{ dispose(this); }).
Both files are compiled only under CONF_fork, so default OSv builds are
unchanged.
Tests: tst-fork test 2 now execs a real payload (/tests/payload-exit7.so, which
prints a marker and exit(7)s) instead of the previous _exit(7) sentinel that
masked whether execve actually launched anything; a return from execve() is now
treated as a failure. Added tst-execve.so (fork+execve launches the payload and
its exit code is reaped; missing path returns -1/ENOENT) and payload-exit7.so.
Validated on x86-64 (KVM disk boot): tst-fork 10/10 and tst-execve 3/3 pass with
the real exec payload and OSv shuts down cleanly (5/5 repeat runs, no hang).
Copyright (C) 2026 Greg Burd
Follow-up to the base fork() PR: gives a forked child its own address space with copy-on-write of private mappings, so a forked child has real memory isolation like Linux fork() while MAP_SHARED / shm stays shared. Also gated behind CONFIG_fork (default off): with it disabled none of this is compiled and OSv's single-address-space model and context-switch path are unchanged. With CONFIG_fork enabled: - mmu::address_space object (page-table root + vma_list); the previous global becomes 'address space 0' (kernel + init app). - Per-thread current address space; the context switch loads the target CR3 only when the address space differs (a no-op for AS0-only workloads). Kernel PML4 entries are shared across all address spaces so OSv code + the kernel heap work identically after a switch. - fork() clones the parent's vmas into a child address space: PRIVATE writable mappings are write-protected in both and copied on the first write (reusing OSv's existing COW fault machinery); MAP_SHARED / shm map the same physical pages (truly shared). execve() returns the thread to AS0. - Lock/condvar wait_records for fork-child (non-AS0) threads are allocated from the AS-shared kernel heap instead of the thread stack, so a wait_record queued on a shared kernel mutex resolves to the same physical page for any waker across address spaces (kernel-stack coherence). Validated: tst-fork-cow proves a forked child's private memory stays private while MAP_SHARED stays shared; tst-fork stays 10/10. Known limitation (documented): the child's stack is still relocated+copied rather than same-VA COW, so a child that unwinds a very deep call chain (e.g. a multi-process server forking backends) can still hit a stack-fidelity issue (tst-fork-deep); the same-VA stack fix is a further follow-up. This PR delivers memory-isolated fork for the common cases with COW proven. Copyright (C) 2026 Greg Burd
Give a forked child its parent's EXACT user stack -- same virtual addresses, private physical pages -- instead of a relocated copy with a biased SP. A relocated+biased stack leaves every saved rbp, return address and &local in the copied frames pointing at the parent's stack (off by the bias), so a deep unwind in the child dereferences the parent and faults. tst-fork-deep (fork 12 frames deep, child unwinds all the way up and _exit()s a checked value) is the test that catches this; it now passes. Gated behind CONF_fork (default n); conf_fork=0 is unchanged and builds/boots clean (fork code fully excluded, tst-mmap passes). Four coupled changes make same-VA correct, layered on the Stage 2 COW address space and the fork-child lifecycle fix (detached child + dispose(child) in the reaper): 1. core/mmu.cc: clone_address_space() PRIVATIZES the forking thread's live stack VA range into the child AS -- fresh private pages that byte-copy the parent's stack, mapped writable (not COW: OSv runs kernel code with irqs off on the app stack, a COW fault there is illegal) at the SAME VA. The parent's PTEs are untouched. 2. arch/x64/fork.cc + libc/process/fork.cc + include/osv/fork.hh: the child resumes with the caller's FULL callee-saved register context (rbx, rbp, r12-r15, rsp, rip), captured at fork() entry into osv::fork_resume_ctx and restored by the child trampoline off a scratch base register (so the restore never clobbers its own base). fork() jmps past its own epilogue, so without this the caller would resume with fork()'s internal register values -- a deep chain that keeps an accumulator in a callee-saved reg across fork() computes garbage. 3. arch/x64/arch-switch.hh: defer the fork CR3 switch to EXACTLY the rsp/rbp swap inside switch_to()'s asm. Loading the incoming AS's CR3 early, while still executing on the outgoing thread's stack (the fpu-cw/mxcsr scratch saves), resolved those shared-VA stack writes through the wrong address space and clobbered a blocked thread's live stack (root-caused via hardware watchpoint: switch_to wrote garbage into a blocked parent's on-stack user_mutex slot in AS0). 4. core/lfmutex.cc + core/mmu.cc: a wait_record on a shared kernel condvar/mutex must live in the identity-mapped kernel heap (coherent in every AS) whenever a cross-address-space wake is possible. Extend the Option-A rule from 'current thread is a fork child' to also cover 'any child address space is live' (live_child_address_spaces counter), so the parent in AS0 -- woken by a child that walks the queue through its own page tables -- is safe too. libc/process/execve.cc: on a fork child, move the thread onto its own kernel stack before switching to AS0 and running the target, so reloading CR3 to AS0 cannot alias the same-VA app stack onto the parent's physical page (which would corrupt a blocked parent). The child address space is destroyed HERE, on the kernel stack, exactly once: the reap-time cleanup (libc/process/fork.cc) then sees the thread's AS is AS0 (!= child_as) and skips its own destroy, so destroy_address_space() runs once whether the child exits normally (reaper destroys) or execs (execve destroys) -- no leak, no double-free. A pre-exec existence check keeps a failed exec returning to the caller. Validated on x86-64 (KVM): tst-fork 10/10, tst-fork-cow 6/6, tst-fork-deep 0 failures, tst-execve 3/3 with clean shutdown (no hang, no double-free at reap), on smp=1 and smp=4, 5/5 repeat runs; conf_fork=0 builds and boots clean (fork excluded, tst-mmap passes).
…nZFS + patch series
Make the in-kernel ZFS implementation selectable at build time and integrate
OpenZFS 2.4.x without maintaining a fork, layered cleanly on the current
master (which already carries the reworked pagecache/ARC bridge, block
multiqueue, io_uring, musl 1.2.1, and modern-toolchain support).
conf_zfs switch (default bsd):
- conf_zfs=bsd - legacy in-tree BSD/Illumos ZFS (default, unchanged path)
- conf_zfs=openzfs - vendored OpenZFS 2.4.x from external/openzfs
Only one is compiled per build; both produce libsolaris.so plus the
zpool/zfs userspace. The shared solaris common objects (avl/nvpair/unicode/
fm/list) are provided by OpenZFS in openzfs mode and by the in-tree copies in
bsd mode.
Fork-free OpenZFS:
- external/openzfs pinned to the PUBLISHED upstream tag zfs-2.4.2
(github.com/openzfs/zfs @ 6330a45b), a plain submodule with no OSv fork.
- The OSv platform-layer changes live as a 19-patch git series in
modules/open_zfs/patches/ and are applied at build time by the Makefile
(idempotent via the external/openzfs/.osv-patches-applied stamp) only in
openzfs mode.
- .gitignore keeps an explicit !modules/open_zfs/patches/*.patch exception so
the series is tracked despite the global *.patch ignore.
ZFS-implementation-agnostic kernel:
- cv_timedwait keeps the BSD relative-timeout semantics; a distinct
openzfs_cv_timedwait (absolute deadline) is added in bsd/porting/netport1.cc
and exported, so the same loader.elf serves both. OpenZFS objects get
-Dcv_timedwait=openzfs_cv_timedwait via OPENZFS_CFLAGS in openzfs_sources.mk.
- core/pagecache.cc keeps master's live map_arc_buf/register_pagecache_arc_funs
bridge (used by conf_zfs=bsd) and additively provides the three symbols the
OpenZFS module needs (osv_free_pages, osv_pagecache_map_arc_page,
osv_pagecache_register_arc_rele) via a separate borrowed-ARC-page path into
read_cache. The borrow class is renamed cached_page_arc_borrow to avoid
colliding with master's arc_buf_t-based cached_page_arc.
- drivers/virtio-blk.cc: give the completion thread a 256 KB stack so the ZFS
vdev_disk_bio_done path does not overrun the default kernel stack.
Also: OpenZFS 2.4.x userspace lib/cmd build rules, openzfs_osv_compat.c, the
zfs-tools module.py (drops libzutil/libshare/libzfs_core/libtpool in bsd mode),
mkfs -m pool-root pinning gated on CONF_ZFS_OPENZFS, cpiod umount prefix
normalization so the host-side image build flushes the pool, and the ZFS
validation/benchmark test suite.
The conf_zfs=openzfs build linked the legacy BSD-ZFS kstat stub (bsd/.../opensolaris_kstat.o) to satisfy kstat_create/install/delete, but every OpenZFS module was compiled against the OSv SPL kstat_t in external/openzfs/include/os/osv/spl/sys/kstat.h (~64 bytes: ks_data, ks_ndata, ks_data_size, ks_flags, ks_update, ks_private, ks_private1, ks_lock). The BSD stub allocated only a 16-byte kstat_t, so OpenZFS callers such as arc_init()/dnode_init() wrote ks_update/ks_private past the end of the allocation, corrupting the malloc free list. The next kstat_create() then returned a garbage pointer and faulted at ks_ndata = ndata (SIGSEGV -> abort with empty backtrace right after the version banner, before mkfs ran). Fix: implement OSv-native kstat_create/install/delete in openzfs_osv_compat.c using the correct OpenZFS kstat_t layout (virtual kstats; OSv has no /proc or sysctl consumer), and filter the BSD opensolaris_kstat.o out of the openzfs solaris object list in the Makefile so only the correct implementation is linked. The zfs_builder guest now boots, runs mkfs (creates + mounts pool osv), populates it via cpiod, and exports cleanly.
…mount zfs_unmount() (called by zfs destroy, zpool export, and property remounts) only unmounts a dataset if it can find it via libzfs_mnttab_find() -> libzfs_mnttab_update() -> getmntent(). On OSv that lookup went nowhere for two reasons: - MNTTAB pointed at a nonexistent path, so fopen(MNTTAB) failed before getmntany() ever ran; and - the getmntent/getmntany stubs in libzfs_util_os.c returned EOF. Datasets auto-mounted by the kernel (zfs_domount at pool/dataset create time, not through the libzfs do_mount path) were therefore invisible to libzfs, the objset stayed owned, and zfs destroy / zpool export failed with EBUSY / 'dataset is busy'. Fix, as two edits to the OpenZFS patch series: - patch 0004: MNTTAB -> /etc/mnttab (created empty at boot, so fopen succeeds and libzfs proceeds to getmntany()). - patch 0020: add lib/libzfs/os/osv/libzfs_mnttab_os.cc implementing getmntent/getmntany over the live VFS mount table via osv::current_mounts() (filtered to ZFS), replacing the EOF stubs. Wire the new C++ shim into the libzfs build, filtering the C-only warning flags out of ozfs-cflags-common that a C++ translation unit rejects. Makes zfs destroy and zpool export/import work on OSv, which the feature-coverage tests depend on.
zpool export/import intermittently aborted with Assertion failed: owner.load(...) == sched::thread::current() (core/lfmutex.cc: unlock: 221) in condvar::wait(). Root cause: the OpenZFS userland thread pool (lib/libtpool) hands jobs to worker pthreads that block in pthread_cond_wait() on a shared tp_mutex. On OSv a pthread mutex and condvar ARE the kernel lockfree::mutex + condvar, whose wait-morphing protocol transfers mutex ownership from the signalling thread to a waiter. During tpool_destroy() teardown that handoff races: a worker returns from pthread_cond_wait() and re-enters it, unlocking a tp_mutex it no longer owns -> lfmutex owner assertion. Reproduced deterministically on zpool import, whose device-scan (zutil_import.c) and mount (libzfs_mount.c) thread pools default to hundreds of workers (mount_tp_nthr = 512). Fix: run tpool jobs synchronously on OSv (0021) and force serial dataset mounting (0022). OSv threads are cheap and these jobs are short, so serial execution removes the worker/teardown machinery and the race at no practical cost. Verified 0/8 asserts (was 8/8) across create/export/ import/status cycles from a clean patch-series rebuild.
…dataset) Patch 0017 reclaimed only the mount-root znode before dmu_objset_disown. A mounted child dataset (e.g. pool/fs) leaves its own znodes live, each holding an object bonus buffer that pins a dnode. dnode_destroy never runs, the objset os_dnodes list never empties, and spa_export() blocks forever in spa_evicting_os_wait(). Reproduced deterministically by: zpool create; zfs create pool/fs; zpool export (hang). Walk z_all_znodes at unmount and inactivate every remaining znode, as vop_inactive would, so the objset drains. Verified 6/6 completions (was 0/6, hung) for create+child-fs+export+import+destroy.
OSv read_partition_table() names MBR slots 0-based (first slot is /dev/vblkN.0) and exposes a raw unpartitioned disk directly as /dev/vblkN with no child node. zfs_append_partition() unconditionally appended .1, so \x27zpool create test /dev/vblk1\x27 on a raw wiped NVMe failed with \x27cannot open /dev/vblk1.1: No such file or directory\x27. Fix (patch 0023): only append .1 when that partition node actually exists; otherwise use the whole raw disk as-is, matching Linux/FreeBSD whole-disk behavior (ZFS writes its own labels to the whole device). Also adds FINDINGS-osv-openzfs.md documenting Bug 1, Bug 2, and the page-allocator-under-memory-pressure investigation (>=4G runtime guests). Verified: zpool create on a raw disk now succeeds, pool ONLINE.
…r 0) zpool list faulted with strlen(NULL) in print_line() because zpool_prop_column_name() returned NULL for a default list column on OSv. The pool data row is correct; only the header label resolves NULL. Guard header against NULL, falling back to the ZFS missing-value glyph, so the command cannot crash.
zfs_is_readonly() was hardcoded B_FALSE, so a readonly=on dataset stayed writable on OSv. Read the readonly property at mount into a new zfsvfs->z_readonly, return it from zfs_is_readonly(), and reject the write vnops (write/create/remove/rename/mkdir/rmdir/setattr/truncate/symlink) with EROFS. Verified writes fail EROFS after remount and readonly=off restores writability.
vdev_disk_open() never set vd->vdev_has_trim, so zpool trim/autotrim reported trim not supported. Enable it optimistically (OSv has no capability query): the ZIO_TYPE_TRIM->BIO_DISCARD path reclaims space on virtio-blk with discard, and devices without discard get ENOTSUP mapped gracefully. Verified zpool trim completes 100% with discard=unmap.
…ryption/dedup/draid/TRIM/O_DIRECT/etc)
Head-to-head storage microbenchmark of the two ZFS ports on identical a bare-metal host, 8 GiB KVM guest, single-vdev (raw NVMe) and raidz2 (7x20 GiB). No Postgres/HammerDB - isolates the filesystem layer. Key finding (the ARC-bridge measure): mmap of a 512 MiB file costs BSD-ZFS ~5 MiB extra RAM vs OpenZFS ~501 MiB - BSD's unified ARC/page-cache bridge shares pages, OpenZFS double-buffers via its borrow path. Same root cause drives BSD's warm-read and cached-random-read wins. OpenZFS wins the write/CPU paths (seq write +34%, lz4 +56%, metadata +63%, fsync +11%) and uniquely offers O_DIRECT (ARC bypass, ~raw-ceiling, BSD has no direct=). Harness (scripts/bench/): pure-C zfs-bench.c (a g++ .so pulls libstdc++ symbols OSv can't resolve at load; C avoids GLIBCXX/CXXABI/__isoc23). Runs off the zfs_builder bootfs - no usr.img cpiod populate. rebuild-bench.sh, run-bench.sh, run-all.sh drive it; results.tsv is the raw data. Copyright (C) 2026 Greg Burd
…en_zfs/ Per review: external/ is reserved for things compiled into the kernel (libfdt, acpica). The vendored OpenZFS tree is a module dependency, so move the submodule from external/openzfs to modules/open_zfs/openzfs alongside its patch series and module.py. Updates: .gitmodules path (still points at the published upstream github.com/openzfs/zfs at tag zfs-2.4.2, no fork), the OZFS/OPENZFS make variables (Makefile + bsd/sys/cddl/openzfs_sources.mk), the parse-time patch-apply step (stamp path, -d guard, and the git -C ... apply path, whose relative patch prefix changes from ../../modules/open_zfs/patches to ../patches now that the submodule sits one level deeper), and the doc comments that named the old path.
The per-child COW address space (clone_pt_level<1>) shared any 2 MB large
page verbatim between parent and child instead of copy-on-writing it:
if (ppte.large()) { child_pt[i] = ppte; continue; } // 2MB: share as-is
so a forked child writing a private large mapping scribbled the parent(s)
too. This hit large malloc (>= 2 MB, mmap-backed), large MAP_ANON regions,
and any 2 MB-backed private writable mapping in the application slots.
Fix: for a WRITABLE large PD entry, split it to 4 K in the parent in place
(the existing split_large_page primitive) and fall through to the normal 4 K
clone path, which already makes the correct per-page decision -- private
writable -> COW, MAP_SHARED -> stays shared, read-only -> shared as-is. A
read-only large page can never diverge, so it is still shared verbatim. This
reuses the whole 4 K COW fault machinery (clone_pt_level0 +
handle_cow_write_fault); the only cost is one 4 K page table per split large
page, acceptable under the opt-in CONF_fork.
Verified on x86-64 (KVM microvm):
- a standalone musl-PIE fork test now isolates a private >2 MB malloc across
fork (child write does not change the parent; parent write does not change
the child), while a MAP_SHARED region stays mutually visible;
- tst-fork 10/10, tst-fork-cow 6/6, tst-fork-deep and tst-execve all pass
(no regressions);
- conf_fork=0 builds clean with the whole fork path (this change included)
compiled out (clone_address_space absent from loader.elf).
Note: this fixes large-page (application-slot) COW. OSv's small-object
malloc heap lives in the shared kernel identity map (mem_area::mempool), which
fork() shares by design; isolating that is a separate change.
All gated behind CONF_fork (default off).
Copyright (C) 2026 Greg Burd
Stock fork-per-backend PostgreSQL corrupts memory on OSv because OSv's small-object malloc heap lives in the kernel identity map (mem_area::mempool, VA >= 0x400000000000), which is shared verbatim across every address space and cannot be copy-on-write cloned on fork(). A forked child writing to a heap object inherited from its parent scribbles the SHARED heap. Give application allocations a home in an ordinary anonymous mmap region in the app-slot VA range (below phys_mem), so clone_address_space()'s existing COW machinery isolates the whole app heap per child. virt_to_phys() already page-walks addresses below phys_mem, so arena pages stay DMA-usable. The arena is a segregated free-list whose bookkeeping lives entirely in kernel BSS (never in arena pages), so managing it never faults an arena page and never recurses into malloc during fork's own page-table work. std_malloc routes an app thread's allocations to the arena (decided by is_app at malloc time); free()/realloc() dispatch purely by address range, so cross-thread alloc/free is always handled by the allocator that owns the address. All gated behind CONF_fork (default n): with fork disabled none of this is compiled and the heap behaves exactly as before. tst-pgfork.c proves a forked child's writes to inherited AND new small/large heap objects do not leak into the parent. Author: Greg Burd
Progress on the fork heap arena: - arena allocator reworked to use a preemption-disabling spinlock (not a sleeping mutex, which is illegal when malloc runs preempt-disabled) with the page-touching header write done outside the lock. - clone_address_space() pre-reserves the child address_space object and moves the child vma-list construction (anon_vma allocations) OUT of the vmas_mutex critical section, and snapshots parent vmas under the lock into a pre-grown vector -- avoiding malloc under the fork lock (the malloc-during-fork trap). - sched::thread objects and coherent wait_records forced to the identity kernel heap (kernel_heap_scope) so cross-address-space kernel structures are not COW-private. Known blocker (documented in /tmp/fork-arena-result.txt): with the arena active, the first fork() still hits an illegal page fault (page_fault: preemptable() assert) during/around clone -- a COW write fault occurring in a preemption/irq-disabled window. Arena allocator validated standalone (tst-realloc passes). conf_fork=0 verified clean (no fork_arena symbols, boots). Author: Greg Burd
… the fork lock Further hardening of the fork heap arena (still gated on CONF_fork, default n): - arena allocator is now fully lock-free (Treiber-stack free lists + atomic bump pointer): no preemption-disabling lock is held while touching a chunk, so a post-fork copy-on-write page fault on a chunk is always serviceable (OSv forbids faulting with preemption/interrupts disabled). - clone_address_space(): defer flush_tlb_all() and the live_child_address_spaces bump until AFTER releasing vmas_mutex, and build the child vma_list from a snapshot outside the lock -- a write that COW-faults a kernel .bss global (or any malloc) must not run while vmas_mutex is held for write, or the COW fault handler self-deadlocks re-acquiring it. Remaining blocker: COW-cloning the arena subtree specifically still triggers an illegal (preemption-disabled / recursive) page fault at the first fork(); sharing the arena instead lets fork() run (5/5 early tst-fork checks) but loses heap isolation. Documented in /tmp/fork-arena-result.txt. Author: Greg Burd
A thread's TLS block and tiny/large syscall stacks hold state the kernel reads and WRITES from preemption/interrupt-disabled contexts (TLS __thread vars; the kernel runs syscalls on the syscall stack). If they were allocated from the COW fork arena (an app thread creating a thread after the arena is live), fork would write-protect them and the next preempt-disabled write would take an illegal COW page fault. Force these allocations to the identity kernel heap via kernel_heap_scope, matching the thread-object and wait_record treatment. Author: Greg Burd
…cycle + execve-continuation allocs to identity heap Root cause (found via qemu+gdb hardware watchpoint): the kernel_heap_scope guard around aligned_alloc in setup_tcb was being ELIDED by the compiler. force_kernel_heap was a plain __thread unsigned; GCC models aligned_alloc (and operator new) as not reading global memory, so the paired ++/-- around a single such call was dead-code-eliminated -> app-thread TLS/thread objects landed in the COW arena -> writing preempt_counter (in a COW arena page) during fork triple-faulted. Fixes (all gated CONF_fork): - force_kernel_heap -> volatile, so the scope's inc/dec survive around builtin-modeled allocators (verified in disassembly + a minimal repro). - wrap the whole sched::thread construction in make() in a kernel_heap_scope so the thread object, _detached_state, and _wakeup_link._helper (all touched by the scheduler cross-AS with preemption off) go to the identity heap. - __cxa_thread_atexit_impl linked_destructor nodes -> identity heap. - execve() continuation strings (s_path/s_args/s_env): allocate their backing storage under kernel_heap_scope so it survives destroy_address_space() of the child COW AS (was reading a freed arena buffer -> 'executable too short /'). Validated: tst-pgfork PASS 9/9 (child small-heap writes do NOT leak to parent - the arena's core purpose works). tst-fork now passes 4/10; fork+execve (test 2) still hits a nested page-fault under investigation.
…hes the arena A forked child's fork+execve (tst-fork test 2) aborted with 'exception_depth <= 1' during ELF demand-paging. The exec'd program's file-backed pages fault in through vm_fault -> pagecache/ROFS, which allocates cached_page bookkeeping. On an app thread those allocations route to the COW fork arena; first-touching a fresh arena page WHILE servicing a fault (already holding vma_list_mutex, possibly one exception deep) takes a second page fault, exceeding exception_depth and asserting. vm_fault services KERNEL work: pagecache pages and filesystem demand-paging buffers are shared kernel infrastructure that must be identity-mapped and never a COW arena page. Wrap the whole handler in a kernel_heap_scope (gated CONF_fork) so no arena page is allocated or first-touched during fault servicing. Restores tst-fork 10/10 with the arena enabled.
…loc)
The detached-thread reaper kept its pending zombies in a std::list<thread*>,
whose push_back() in add_zombie() allocates a list node. Two problems with
that under the fork arena:
1. add_zombie() runs from a terminating thread. For a fork child that is an
APPLICATION thread executing in the child's COW address space, so the
std::list node is malloc'd into the fork arena -- landing on a
COW-private page at an arena VA. The reaper drains the list from AS0,
where that same arena VA resolves to a DIFFERENT physical page: it reads
a garbage 'thread*' and join()s a bogus pointer forever. OSv then never
runs the child's cleanup, the child's application_runtime reference never
drops, application::join()'s wait_until(_terminated) never completes, and
the guest hangs at shutdown instead of powering off.
2. The node allocation can also fault a fresh heap page from a
preemption-disabled termination window.
Switch _zombies to a boost::intrusive::list threaded through a new
thread::_zombie_link hook. The link lives inside the thread object (identity
kernel heap), so add_zombie() performs NO allocation: coherent across address
spaces and safe from a non-preemptable context.
… heap async::run_later()/timer task nodes (one_shot_task, percpu_timer_task) were allocated from the COW fork arena. The async worker walks its intrusive task list with preempt_lock held (preemption disabled); touching a COW arena page there faults, and OSv forbids page faults in non-preemptable context (assert(sched::preemptable()) in page_fault). Force these nodes onto the identity kernel heap via fork_arena::kernel_heap_scope, matching the existing wait_record / child-registry kernel-heap pattern. Surfaced by booting stock musl PostgreSQL 18 with CONF_fork: the postmaster's first forked child (checkpointer) tripped this the moment the async worker fired a TCP-stack run_later callback. Clears that wall; shared-memory attach across fork() then works (verified: child reads PG ProcGlobal shared state).
fork_arena::init() reserved the 512 MiB arena VA with map_anon(mmap_fixed) WITHOUT populating it, so pages faulted in lazily on first touch. Under real concurrent PostgreSQL load fork_arena::alloc() gets entered from an IRQs-off / preemption-off context (mid-exception, kernel work), and first-touching a freshly bump-carved page there DEMAND-FAULTS -- which page_fault forbids (assert(ef->rflags & rflags_if) / assert(preemptable)) -> abort. Add mmu::mmap_populate to the map_anon flags: all 512 MiB is RAM-backed at init, so alloc's first write always hits a present page and can never fault. That makes alloc safe to call from ANY context. clone_address_space() still COW-clones the whole vma per child; the child only WRITE-faults (COW break) from app context with irqs/preemption on, so the original fault-context invariant is preserved on the child path. 512 MiB fully backed is acceptable for a per-app fork heap; boot is unaffected (~3-4 s) and the whole fork suite stays green (tst-fork 10/10 x5, tst-pgfork 9/9, tst-fork-cow/deep/execve pass). All gated CONF_fork; conf_fork=0 unchanged (arena compiles to an empty TU, exits 0). Also bump apps to the pg18-fork test module (musl PG18 fork-per-backend image).
…tion bug A fork child that spins deep on its same-VA stack while being heavily preempted (touching .data/.bss globals, its own stack, FPU/SSE, and taking syscalls + COW faults) has its register context corrupted on the preemptive context-switch resume path -- a callee-saved register (rbp) is reconstructed from a .text immediate and a pointer resolves into the same-VA stack, giving a page fault outside the application. Deterministic on -smp 1 and -smp 2. The existing fork tests only fork short-lived children preempted a handful of times, so they never exercised this window. The manifest entry is auto-derived from the tests list; the test itself is unconditional (fork() just fails when built conf_fork=0). Author: Greg Burd
…eempt test + mmap-in-child repro Three related changes from investigating the PostgreSQL fork-child wall: 1. libc/signal.cc: the global "waiters" std::list (one per signal, populated by wait_for_signal / drained by unwait_for_signal) is manipulated cross-address- space -- a fork child and its parent both push/remove on the same shared list. Its list nodes were allocated from the COW fork arena, so a child's node became a COW-private page whose next/prev links were inconsistent with the shared sentinel seen from the other AS; a later list::remove() traversal (reached via sigprocmask in the checkpointer child) followed a dangling link and page-faulted with interrupts disabled -> assert(rflags & IF) in page_fault. Force the identity kernel heap for those node alloc/free, the same rule already applied to thread objects and wait_records. Proven with gdb: this was one of the two PG crash variants after "InitAux shmem read OK"; the fix eliminates it. Gated CONF_fork. 2. tests/tst-fork-preempt.cc: reframed honestly. A long-lived fork child spun deep on its same-VA stack while heavily preempted (compute + .data/.bss globals + stack + FPU + syscalls) PASSES -- proving the preemptive context switch itself preserves a fork child's register/stack context (this refutes the earlier hypothesis that the deferred-CR3 same-VA switch was the wall; see report). Green on -smp 1 and -smp 2. 3. tests/tst-fork-child-mmap.cc: standalone repro (NOT in the suite -- it aborts the unikernel) of a separately-found real bug: a fork child's own mmap() then write SIGSEGVs, because only the page-FAULT path is per-child-AS aware while the mmap/munmap ALLOCATION path (mmu::allocate / map_anon via the global vma_list + vma_range_set) is not. See /tmp/pg-preempt-fix.txt for the full analysis. Author: Greg Burd
…e -> first stock PostgreSQL query served on OSv
Route every fork-shared kernel structure that a DIFFERENT address space
(a forked PG backend, the RX/IRQ thread, or the reaper) touches onto the
identity kernel heap instead of the COW fork arena, matching the proven
rule already applied to struct file, ramfs nodes, thread objects and the
signal waiters list.
The DSM wall: PG's dynamic-shared-memory POSIX backend creates a segment
with shm_open(O_CREAT|O_EXCL)+ftruncate+mmap(MAP_SHARED) in the postmaster
and ATTACHES it by name (shm_open(O_RDWR)) in every forked backend during
InitPostgres. OSv keeps named segments in a kernel-static registry
(posix_shm_objects, name->fileref). The std::unordered_map OBJECT is in
shared BSS, but its nodes / std::string keys / fileref control blocks are
heap-allocated -- an app thread's allocations land in the COW fork arena,
giving each process a divergent private map. A name the postmaster inserts
is then invisible to a forked backend -> shm_open(name, O_RDWR) = ENOENT
("could not open shared memory segment"). shm_file::_pages (the physical
huge-page map) had the same gap. Fix: kernel_heap_scope on the registry
mutations and shm_file::page().
Post-fork mmap vmas (map_file/map_anon): a fork child's mmap-created vma is
disposed cross-AS by destroy_address_space() on the reaper (AS0); an
arena-resident vma faults there. Force it onto the identity heap like the
clone-path vmas.
The ramfs wall: ramfs_enlarge_data_buffer's data buffer + segment-map node
were arena-allocated on the writer thread, so a backend reading a file the
postmaster grew saw a divergent segment map -> ramfs_read bounds assert.
The net-RX wall: net_channel _pollers/_epollers and classifier channels are
populated by a backend adding its socket to epoll but WALKED by the virtio-
net RX path (wake_pollers/post_packet) in AS0 with preemption off -> arena
VAs unmapped there -> page fault while !preemptable(). And mbufs: uma_zalloc
backs the BSD net stack; a backend allocates a send-path mbuf for its query
response that the RX thread drops (tcp_input->sbdrop) -> arena mbuf =
inconsistent send buffer -> panic("sbdrop"). Route both to identity.
The thread kernel stack: init_stack's malloc was not under a kernel_heap_
scope, so a forked backend ran on an arena stack that the reaper frees from
AS0 in ~thread -> page fault on an arena address. Force identity, like the
TCB/TLS and syscall stacks alongside it.
Result: stock PostgreSQL 18.4 serves real queries on OSv --
select 1 -> 1
create table q(x int); insert into q values(1),(2),(3); select sum(x) -> 6
3 concurrent psql (3 forked backends) -> 11, 22, 33
Regression test tests/tst-fork-posix-shm.cc mirrors the DSM create/attach
across fork (reproduced ENOENT on the arena-node path, passes with the fix;
also verifies MAP_SHARED write-back coherence child->parent). All fork tests
0 failures; shm/mmap/net tests green; conf_fork=0 compiles byte-identical.
Next wall (distinct, symbolized, NOT fixed here): after ~2 forked backends
are reaped, the next backend fork() -> thread ctor -> application::
get_current() GP-faults on a corrupt _app_runtime (fork+reap lifecycle).
Author: Greg Burd
…-> stock PostgreSQL serves SUSTAINED connections on OSv The fork+reap sustained-serving wall: PG served the first ~6 sequential backends then GP-faulted in the NEXT fork's new-thread ctor at application::get_current() -> get_shared() -> shared_from_this(), escalating under KVM to "exception nested too deeply". Root cause (gdb-proven, candidate c): the per-application application_runtime object AND its shared_ptr control block were allocated by an app thread at postmaster start, so std_malloc routed them into the COW fork arena (VA 0x3000..). Every backend thread's _app_runtime points there; the thread ctor dereferences runtime->app on every fork. clone_address_space COW-clones the arena per fork, so across fork/reap cycles the forking thread reads a DIVERGENT arena copy whose `app` reference is garbage (gdb: _app_runtime._M_ptr == 0x3000000004a0 in the arena, runtime->app == 0x202.. wild, varying run to run) -> get_shared() faults on a bogus application*. Fix: allocate application_runtime + control block on the identity kernel heap (make_shared under fork_arena::kernel_heap_scope) so it is byte-identical in every address space and never diverges -- the same rule as the DSM registry nodes, mbufs, thread objects and thread kernel stack. A second wall this uncovered (~50 connections): virtio-net's TX net_req is alloc'd by a backend AS in xmit_prep but delete'd cross-AS by txq::gc as the host completes the send; an arena net_req made free()-by-address read a garbage chunk header in the reaping AS (assert h->magic == chunk_magic). Route the new net_req to the identity heap too. Both changes are #if CONF_fork; the non-fork build keeps the byte-identical original (app.cc #else restores `new application_runtime`). Regression test tst-fork-serial: sustained overlapping fork/work/exit/reap loop (30 iterations) exercising get_current() on every new-thread ctor -- the OS-level guard for the fixed path (the definitive reproducer is PG itself; noted honestly in the test header). Result (fixed pg18-fork, KVM -m 4G -smp 2): 100/100 sequential + 40/40 concurrent (5x8 backends) + 12/12 create/insert/select rounds served, guest healthy -- was 6 before the wall. Next distinct wall (NOT fixed): heavy pgbench -i bulk-load storage/shared-buffer coherence, symbolized in the report. Author: Greg Burd
…or semaphores, waitqueues and epoll watcher lists -> stock PostgreSQL survives a heavy pgbench -i bulk load on OSv
The storage / shared-buffer coherence wall under `pgbench -i` (parallel index
build / checkpoint large write) was a ramfs segment DATA BUFFER landing in one
forked backend's private COW address space:
ramfs_enlarge_data_buffer() grows a file's segment with malloc(). Once the
segment exceeds the fork arena's 2 MiB max_alloc it falls through to
malloc_large()'s size>=huge_page && !contiguous branch, which reserves an
APP-SLOT (VA < 0x400000000000) anonymous mmap in the CURRENT process's address
space. With per-child COW address spaces that buffer exists ONLY in the
backend that enlarged the file; a sibling backend read()ing/write()ing the
same ramfs file then memcpy's (uiomove) against an app-slot VA unmapped in its
own page tables:
page fault outside application, addr: 0x2000....
RIP: memcpy_repmov_ssse3 ; pread/pwrite -> vfs_file::read/write -> ramfs
i.e. the exact fault under `pgbench -i -s 5`. A kernel_heap_scope does not
help large allocations: it only skips the fork arena; malloc_large still uses
app-slot map_anon for huge non-contiguous requests.
Fix (fs/ramfs/ramfs_vnops.cc): allocate the segment data buffer as physically-
contiguous IDENTITY-mapped memory (memory::alloc_phys_contiguous_aligned, VA >=
0x400000000000, in the kernel PML4 slots clone_address_space() shares verbatim),
so every address space reaches the same file image. Plain free() dispatches by
address to free_large (identity path), so the existing free(data) sites are
correct from any address space.
Investigating the concurrent-writer path uncovered three more instances of the
same "kernel structure a cross-AS waker dereferences must be identity-mapped"
rule (all #if CONF_fork; non-fork byte-identical):
- libc/sem.cc: a pshared POSIX semaphore's posix_semaphore object (its _val,
_mtx and _waiters list) was heap-allocated on the creating app thread -> COW
arena. A waiter (sem_wait) blocked on its private _waiters and a poster
(sem_post) in another backend bumped a private _val -> lost wakeup, and
post_unlocked() GP-faulted dereferencing a stack wait_record cross-AS. Force
the object onto the identity kernel heap.
- core/semaphore.cc: the semaphore wait_record itself is queued on the shared
_waiters list and read by a cross-AS poster. Route it through the same
fork_child_needs_heap_wait_record() gate lockfree::mutex uses (identity heap
when a cross-AS wake is possible; on-stack fast path in AS0).
- include/osv/waitqueue.hh + core/waitqueue.cc: wait_object<waitqueue> queued a
stack wait_record on the shared _waiters_fifo (epoll's activity waiters, etc.)
that a cross-AS waker walks. Give it a coherent_wait_record, like condvar.
- fs/vfs/kern_descrip.cc: file::epoll_add allocated f_epolls (the per-file list
of watching epoll instances) on the registering backend's COW arena; the
RX/netisr thread walks so->fp->f_epolls cross-AS to wake watchers. Keep the
vector and its growth on the identity kernel heap.
Regression test tests/tst-fork-shared-write.cc (registered in
modules/tests/Makefile): a forked child ftruncate()s + writes a large pattern to
a ramfs file (forcing the >2 MiB segment buffer into its AS), then a SIBLING
forked child and the parent read it back. Reproduces the "page fault outside
application" memcpy abort on HEAD; passes with the fix.
Results (KVM -m 4G -smp 2, pg18-fork, ramfs /data):
pgbench -i -s 5 (heavy bulk load): completes -- 500k rows, vacuum, primary keys.
pgbench -c 1 -T 20 (sustained heavy TPC-B write): 41,042 txns, 2053 tps, 0 failed.
pgbench -S -c 4 (concurrent read): sustains ~45-50k tps for a bounded window.
All fork regression tests: 0 failures (tst-fork, -cow, -deep, -serial,
-arena-freelist, -preempt, -irq-cow, -conn-socket, -socket, -pgfork, -execve,
tst-fork-shared-write NEW). conf_fork=0 builds clean (CONF_fork==0).
HONEST STATUS: the storage/shared-buffer coherence wall this targeted is FIXED
(pgbench -i completes; sustained single-writer load completes). Sustained
CONCURRENT load still hits two distinct, precisely-symbolized cross-child-COW
walls that are NOT storage coherence: (1) a post-fork MAP_SHARED write-visibility
race (pre-existing tst-fork-posix-shm part 3, fails on clean HEAD, passes only
when gdb/instrumentation serializes the CPUs), and (2) a park_timers boost
intrusive-list double-insert assert in sched::cpu::reschedule_from_interrupt
during epoll_file::wait under concurrency. Both are in the #if CONF_fork
scheduler/timer/AS-clone machinery, the next investigation for a full HammerDB run.
PR cloudius-systems#1456 relevance: MAP_SHARED / cross-AS coherence across per-child COW address
spaces is core to it; these are five more instances of the shipped rule (struct
file, signal table, thread objects, arena free-list, DSM registry, mbufs, app
runtime, net_req): any kernel structure a forked child or a cross-AS waker reads
must live on the identity kernel heap, never the COW fork arena.
…, per-backend pid + SIGURG latch routing, and park_timers migration -> stock PostgreSQL sustains CONCURRENT multi-client load on OSv
Sustained concurrent pgbench now completes with 0 failed transactions
(pgbench -c 4/-c 8 read-write, -S -c 8 read-only) -- the gate to HammerDB.
Four cross-AS coherence walls fixed, all #if CONF_fork (non-fork byte-identical):
(A) epoll_file map/_activity node allocations landed in the COW fork arena and
were freed cross-AS by the RCU quiescent-state thread (AS0) -> fork_arena
chunk-magic abort under sustained epoll load. Force them onto the identity
kernel heap (core/epoll.cc EPOLL_KH).
(B) rcu_defer's on-stack wait_record, linked into *percpu_waiting_defers, was
walked cross-AS by the RCU thread with preemption disabled -> page fault in
non-preemptable context (exception_depth assert) for a fork-child caller.
Use coherent_wait_record (identity heap under fork) (core/rcu.cc).
(C) OSv reported one pid (OSV_PID) for every forked backend, so PostgreSQL's
SetLatch saw owner_pid == MyProcPid for cross-backend latches and took the
LOCAL self-pipe path -> the waiting backend was never woken and a
heavyweight-lock wait wedged forever (0 tps, 0 failed). getpid() now
reports each fork child's distinct pid (runtime.cc, libc/process/fork.cc
pid<->AS registry), and kill() routes a fork-child-targeted signal handler
into the target backend's address space so latch_sigurg_handler pokes the
right self-pipe (libc/signal.cc).
(D) park_timers' per-CPU parked-threads list was not migration-coherent: a
parked thread migrated (load_balance / thread::pin) while still linked on
its source cpu tripped the intrusive-list double-insert. Track the owning
cpu (thread::_parked_cpu), unlink at every migration point, guard push_back
with is_linked() (core/sched.cc, include/osv/sched.hh). Corrects 1041518.
(E) /dev/urandom was unseeded at startup (yarrow accumulates entropy lazily),
so an early read blocked/EINTR-looped and PostgreSQL backends failed with
'could not generate random cancel key' (pre-existing, racy, not fork). Read
directly from RDRAND when the CSPRNG is unseeded (drivers/random.cc; not
fork-gated).
New regression tests (tst-fork-timer-park-stress reproduces A+B;
tst-fork-shared-race the post-fork MAP_SHARED path; tst-fork-urandom E +
per-child pid). Full fork suite 0 failures; conf_fork=0 compiles clean +
non-fork sched/epoll/mmap green. tst-fork-posix-shm part 3 (WALL 1, pre-existing
one-shot MAP_SHARED visibility race) still intermittent; does not block
concurrent pgbench.
… zfs Restructure the in-kernel ZFS build into the module layout requested in PR cloudius-systems#1423 review (issue cloudius-systems#1201), mirroring how java is provided by openjdk*-from-host: - modules/zfs placeholder; module.py selects the impl by conf_zfs (openzfs -> open_zfs, else -> bsd_zfs). Still ships the shared libsolaris.so via usr.manifest. - modules/bsd_zfs provides = ["zfs"]. Owns the legacy BSD/Illumos ZFS build rules in bsd_zfs_sources.mk (the solaris compat list, the zfs core list, and the conf_zfs=bsd object/flag selection) moved out of the top-level Makefile. - modules/open_zfs provides = ["zfs"]. openzfs_sources.mk moved here as open_zfs_sources.mk and extended with the conf_zfs=openzfs object/flag selection (solaris += $(openzfs-all), kstat filter, OpenZFS CFLAGS). The top-level Makefile now includes modules/bsd_zfs/bsd_zfs_sources.mk unconditionally (defines the shared solaris/zfs lists) and, for conf_zfs=openzfs, additionally includes modules/open_zfs/open_zfs_sources.mk. The provider modules register the "zfs" capability during placeholder import, exactly like the java placeholder, so no include collision occurs. Build-structure refactor only; the same objects build for each mode. Kernel impl-agnostic bits (openzfs_cv_timedwait, the -DCONF_ZFS_OPENZFS redirect, the OpenZFS patch-apply) stay in the kernel/top-level Makefile. Validated under KVM: conf_zfs=bsd -> EXIT=0, cpiod finished, pool populated, libsolaris.so.
… pool at root In open_zfs mode the ZFS pool was not usable as the root filesystem: the guest mounted an empty pool at / and powered off with "Failed to load object: /hello" (issue reported by wkozaczuk on PR cloudius-systems#1423). bsd_zfs worked end to end. Root cause: the per-object flag that makes mkfs pick the OpenZFS pool-root mountpoint convention, $(out)/tools/mkfs/mkfs.o: CXXFLAGS += -DCONF_ZFS_OPENZFS was placed in the early conf_zfs=openzfs block, ABOVE the line where `out` is defined (out = build/$(mode).$(arch)). At that point $(out) expands to the empty string, so the target-specific variable bound to the target `/tools/mkfs/mkfs.o` instead of the real `build/release.x64/tools/mkfs/mkfs.o`, and the define never reached the compile. mkfs.cc therefore always took its #else (BSD) branch: zpool create -f -R /zfs osv # no -m / With BSD ZFS the pool-root mountpoint defaults to / so the pool mounts at the -R altroot /zfs, and cpiod --prefix /zfs/ populates the osv dataset; the loader then mounts osv and finds /hello. OpenZFS instead defaults the pool-root mountpoint to /<pool> (/osv), so under -R /zfs the pool mounted at /zfs/osv. cpiods /zfs/... writes then landed on the builders ramfs (nobody was mounted at /zfs) and were lost on shutdown, leaving the on-disk osv dataset empty apart from the osv/zfs child mountpoint stub. At boot the mount + pivot both succeeded, but every lookup in the root ZAP returned ENOENT, so /etc/fstab and /hello were invisible. Fix: define a plain make variable (conf_zfs_openzfs) in the early block and attach the -DCONF_ZFS_OPENZFS per-object flag AFTER `out` is defined, gated on that variable. mkfs.o now compiles with the define, so open_zfs uses: zpool create -f -R /zfs -m / osv # mountpoint / -> mounts at /zfs and cpiod populates the osv dataset exactly as BSD does. The pool now mounts at / and the app runs. Kernel stays impl-agnostic; the only consumer of CONF_ZFS_OPENZFS is the userspace mkfs build tool, and the change is gated on conf_zfs=openzfs so conf_zfs=bsd is byte-for-byte unchanged. Validated under KVM, both modes booting native-example end to end from the ZFS root: conf_zfs=bsd -> zfs: mounting osv from device /dev/vblk0.1; /hello runs conf_zfs=openzfs -> ZFS: root mounted ok; /dev,/proc,/sys remount; /hello runs Write+readback of a file at the ZFS root and its survival across a reboot confirm the mount is a real, persistent ZFS root.
…fs=openzfs
The six ZFS tests that drive the OpenZFS userspace C API via
dlopen("libzfs.so") -- tst-zfs-recordsize, tst-zfs-direct-io,
tst-zfs-encryption, tst-zfs-db-sim, tst-zfs-trim and misc-zfs-mmap-bench --
plus the libzfs* user-space libraries they need, only exist in
conf_zfs=openzfs. They were, however, added to the manifests for any fs=zfs
build, which broke conf_zfs=bsd two ways:
1. usr.manifest emitted /usr/lib/libzfs_core.so (and libzutil/libshare/
libtpool) gated on $(filter zfs,$(fs_type)). conf_zfs=bsd does not build
those four libraries, so the image build failed at upload time with
"ERROR: file upload failed: ... No such file or directory:
'./libzfs_core.so'".
2. The dlopen tests were in the always-on `tests` list, so a conf_zfs=bsd
test.py run collected them from usr.manifest even though libzfs.so /
libzfs_core.so are not present -- they could only ever print
"SKIP: cannot load libzfs.so" (noise), and tst-zfs-trim in particular
looked like it stalled.
Split those six tests into zfs-openzfs-only-tests and append them (and the
zfs-user-libs manifest emission) only when conf_zfs=openzfs, matching how the
top-level Makefile builds the libraries. tst-zfs-crucible-stress, misc-zfs-io
and misc-zfs-arc do not dlopen libzfs and stay in both modes.
conf_zfs reaches this Makefile through the environment (scripts/build exports
every name=value build argument); default it to bsd here to match the
top-level Makefile when the module Makefile is invoked directly.
The system-lua/luarocks fallback added to modules/lua/Makefile is a build convenience unrelated to the selectable BSD/OpenZFS ZFS work in this PR. Restore the file to its upstream/master state so it is not carried by cloudius-systems#1423; it can be proposed on its own.
…is busy") Building a ZFS image ends with the zfs_builder running /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/; /zfs.so set compression=off osv; /zpool.so export osv and printing "cannot export 'osv': pool is busy" at the very end. mkfs auto-mounts the pool root (osv at /zfs) and the osv/zfs child (at /zfs/zfs), and cpiod writes into them. zpool export's own zpool_disable_datasets() unmounts datasets it finds in /etc/mnttab, but OSv's mount shim never populates mnttab, so it finds nothing to unmount; the kernel then refuses the export with EBUSY because the still-mounted datasets hold the objsets (spa refcount != 0). The message is cosmetic -- the image is fully written -- but it looks like a failure. Drain the pool explicitly before exporting: unmount the datasets by mountpoint (children first) with the OSv-native /tools/umount.so, which calls umount(2) -> VFS_UNMOUNT -> the real zfs_umount -> dmu_objset_disown, dropping the spa refcount to zero so the export succeeds cleanly. This is mode-independent and avoids the bsd "zfs unmount" command, which needs statfs2mnttab (absent from the builder image). umount.so is already built into tools; add it to the zfs_builder bootfs so it is available there.
…irtio-blk req cross-AS coherence -> stock PostgreSQL on ZFS-on-NVMe forks its aux processes on OSv Two cross-AS coherence fixes uncovered running stock PostgreSQL 18.4 with /data on a ZFS pool (fs=zfs conf_zfs=openzfs conf_fork=1), which the prior ramfs image never exercised: 1. libc/pthread.cc: the process-global atfork_handlers std::vector backing store landed in the COW fork arena (register_atfork runs on an app thread); a forked child read a DIVERGENT copy. Additionally, apps run via osv::run() before the main app (zpool.so/zfs.so in the boot cmdline, import_extra_zfs_ pools) register atfork handlers via __register_atfork and then EXIT, leaving dead handler pointers into unloaded objects. When PostgreSQL then fork()s, __osv_run_atfork_prepare() called a garbage/dead pointer -> instruction-fetch SIGSEGV. Fix: (a) allocate the vector on the identity kernel heap under CONF_fork (ATFORK_KH), and (b) skip any handler whose address is not owned by a currently-loaded ELF object (elf::object_containing_addr). 2. drivers/virtio-blk.cc: a forked backend issues a ZFS read/write; make_request does new blk_req(bio) on the backend thread, so the request lands in that backend COW arena. The single blk completion thread (req_done->drain_queue in AS0) then delete req -> fork_arena::free reads a divergent arena chunk header (magic mismatch) -> abort. Fix: allocate blk_req on the identity kernel heap under CONF_fork (same rule as the virtio-net TX net_req fix).
…L heap, libsolaris.so statics, large arrays) -> stock PostgreSQL completes recovery + reaches ready on ZFS-on-NVMe on OSv The prior ZFS-era fork work (18f59ac3) let PostgreSQL fork its aux processes on ZFS-on-NVMe but the startup process then hung: a LOST WAKEUP in the ZFS synchronous-I/O completion path across the fork copy-on-write address-space boundary. Root cause (gdb-proven): ZFS runs as libsolaris.so loaded into the COW-cloned application VA slot (slot 32), so ALL its state -- .data/.bss statics (buf_hash_table, arc_anon/mru/mfu, the dbuf hash), its SPL/kmem heap objects (zio_t.io_cv, zil_commit_waiter_t.zcw_cv, dmu_tx/dbuf/dnode/arc_buf_hdr and their embedded locks/condvars), its 8 MB ARC/dbuf hash arrays, and the block-I/O bio -- gets a private COW copy per forked child. A forked PG process blocks on a primitive in its copy while ZFS kernel threads in AS0 (the virtio-blk completion thread, txg_sync, dp_sync_taskq, the zil lwb writer) signal / dereference a DIFFERENT physical copy: the wakeup is lost (all vCPUs idle) or AS0 faults on a stale pointer. Fixes (all #if CONF_fork; non-fork builds byte-identical): - fs/vfs/kern_physio.cc: alloc_bio() on the identity kernel heap (the block completion thread in AS0 reads bio->bio_caller1/bio_done to drive zio_interrupt -> zio_done -> cv_broadcast; the bio must be one coherent object). - opensolaris_kmem.c: route zfs_kmem_alloc + kmem_cache_alloc (the whole SPL/ZFS heap) onto the identity kernel heap -- ZFS objects are shared kernel state, not per-process app state. free() dispatches by address so this is free-safe. - core/fork_arena.cc + exported_symbols/osv_libsolaris.so.symbols: C-linkage fork_kernel_heap_push/pop accessors so the libsolaris.so module can route its allocations to the identity heap. - core/mmu.cc + include/osv/mmu.hh + core/elf.cc: share libsolaris.so writable PT_LOAD segments (.data/.bss) verbatim (never COW) across every fork child, so its global ZFS state (buf_hash_table, arc_anon, dbuf hash, ...) is coherent in every address space. The ELF loader registers each such segment. - core/mempool.cc: large allocations (>= 2 MB, mapped_malloc_large) made under force_kernel_heap are registered fork-shared too -- catches ZFS 8 MB ARC/dbuf hash arrays, which otherwise land COW in the app mmap slot. Result: PG now completes full crash recovery (was: hung immediately after database-system-was-shut-down) and reaches database-system-is-ready-to-accept- connections on ZFS-on-NVMe. A further wall remains in the ZIL lwb synchronous- write completion (fsync -> zil_commit -> zil_commit_waiter cv_wait; lwb stuck ISSUED) -- documented for follow-up. Author: Greg Burd
…ntity kernel heap so forked-backend writes are not deadlocked A ZFS taskqueue (struct taskqueue: its tq_mutex, tq_queue list heads and tq_threads array) is SHARED kernel infrastructure: the AS0 txg_sync / zio pipeline threads take tq_mutex and splice tasks onto tq_queue, while forked PostgreSQL backends enqueue onto the SAME taskqueue. subr_taskqueue.c allocated it with plain calloc(), which std_malloc routes to the per-process fork COW arena. After the first fork() the arena page holding tq_mutex is COW-write-protected in AS0, so an AS0 sync thread issuing a write ZIO (spa_sync -> dbuf_sync_list -> zio_nowait -> zio_issue_async -> taskq_dispatch_ent -> taskqueue_enqueue -> mtx_lock(&tq->tq_mutex)) takes a COW WRITE fault. vm_fault grabs the kernel vma_list_mutex for write and deadlocks against a forked backend holding it for read across a demand fault; txg_sync wedges and forked-backend dirty data never reaches the disk (the virtio-blk WRITE counter never moves). gdb evidence: tx_sync (spa_sync/dbuf_write/taskqueue_enqueue) blocked in rwlock::wlock on a lock xadd against a mutex at 0x3000000b4fd0 (FORK-ARENA COW, PTE writable=ro); vma_list_mutex held WRITER_LOCK, txg syncing/synced frozen across samples. Fix: wrap the struct taskqueue + tq_threads allocations in the identity kernel-heap scope (fork_kernel_heap_push/pop) so the taskqueue is one physical object shared by every address space (never COW). Single-process OpenZFS is unaffected (no fork, no COW). Gated CONF_fork: conf_fork=0 is byte-identical (the TQ_KHEAP macros expand to no-ops). Signed-off-by: Greg Burd <greg@burd.me>
…_size after write zfs_vop_write() (the OSv vop_write bridge) calls zfs_write(), which updates the ZFS logical file size (zp->z_size) but NOT the OSv vnode cached size (vp->v_size). zfs_vop_read()/zfs_vop_cache() bound reads with vp->v_size, so a file-extending write followed by a read of the newly written region returns end-of-file even though the bytes are in the ARC and on disk. PostgreSQL hits this immediately: a backend extends a relation (writes block 0) and a later read fails with "unexpected data beyond EOF in block 0 of relation ...", so no committed row is visible and no workload can run. The fix (a new OSv platform-layer patch in modules/open_zfs/patches/, NOT a direct edit of the pinned submodule) refreshes vp->v_size from zp->z_size after every successful regular-file write. This is a single-process correctness bug (independent of fork/COW); it was latent because earlier bring-up used write-then-close-reopen or reads within the initial size, never a write-extend-then-read on the same live vnode. Verified: create table + insert + checkpoint + select round-trips (rows visible), and the data survives a guest crash + reboot + pool re-import. Signed-off-by: Greg Burd <greg@burd.me>
OSv shmctl(IPC_STAT) unconditionally returned 0 and shmat(dup(fd)) mapped
whatever unrelated file happened to occupy that fd number. A stale SysV shmid
recorded on disk (e.g. PostgreSQL postmaster.pid after an unclean shutdown) was
therefore mistaken for a live segment across a reboot: shmget key registry is
per-boot RAM, so after a reboot the old id no longer names a shm_file, but
PGSharedMemoryIsInUse() probing that id got SHMSTATE_ANALYSIS_FAILURE and PG
refused to start with "pre-existing shared memory block (key .., ID ..) is
still in use". This blocked PostgreSQL crash recovery on OSv.
Add shm_fd_is_segment() (dynamic_cast<mmu::shm_file*> of the fd) and reject
shmids that are not live shm segments with EINVAL in shmctl(IPC_STAT/IPC_RMID)
and shmat; PGSharedMemoryAttach then correctly concludes SHMSTATE_ENOENT and PG
proceeds with automatic recovery. Wrong even for single-process PostgreSQL
(any restart after a crash); not fork-specific.
Verified: after a guest crash + reboot PostgreSQL runs redo ("automatic
recovery in progress") and reaches "ready to accept connections", and the
pre-crash committed rows are readable.
Signed-off-by: Greg Burd <greg@burd.me>
…ession drivers/virtio-blk.cc: add four volatile global counters (g_blk_submitted / g_blk_completed and the VIRTIO_BLK_T_OUT-only g_blk_wr_submitted / g_blk_wr_completed) incremented in make_request() and drain_queue(). These are the read-via-gdb instrument used to prove forked-backend writes reach the block device (the WRITE counter must MOVE across insert+checkpoint). Small, always-on, benign. tests/tst-fork-zfs-write.cc (+ modules/tests/Makefile, in zfs-only-tests): a forked child writes+fsyncs a 40 KiB file on a ZFS mount (/tmp) and the parent, after reaping the child, must read back the exact bytes at the grown size. Reproduces both write-path walls (taskqueue COW deadlock -> hang/no-persist; v_size beyond-EOF -> short read) on a buggy kernel and passes with the fixes. Gated CONF_fork (SKIPs otherwise). Signed-off-by: Greg Burd <greg@burd.me>
The OpenZFS image-populate step (create_zfs_filesystem in scripts/build and the
cpiod builder in scripts/upload_manifest.py) OOMs at 512M when building the
PG-on-ZFS image; 4G is fine on the build host. scripts/build now honours an
optional ${zfs_builder_mem} override (default 4G).
Signed-off-by: Greg Burd <greg@burd.me>
…so an AS0 writer can wake a forked-backend reader rwlock::rlock() registers a pending reader as a lockfree::linked_item<thread*> that is a LOCAL on the waiting thread stack, pushed onto the shared _read_waiters queue. With per-child COW stacks, when a forked PostgreSQL backend blocks as a reader its read_waiter lives at a COW-private app-stack VA (0x200000...); an AS0 writer releasing the lock walks _read_waiters in wake_pending_readers() and dereferences that VA through AS0 page tables -> SIGSEGV (Aborted in rwlock::wake_pending_readers+82 <- rwlock::wunlock <- taskqueue_thread_loop), which crashed a pgbench bulk COPY at ~20%. Mirror the existing lfmutex/condvar coherent_wait_record mechanism: when fork_child_needs_heap_wait_record() (a forked-child caller, or any live child AS while the AS0 parent waits), allocate the linked_item from the identity kernel heap (fork_arena::kernel_heap_scope) so its VA is coherent in every address space; the waiter frees it after waking. Default OSv / AS0-only keeps the zero-overhead on-stack fast path. Gated CONF_fork; conf_fork=0 byte-identical. Verified: pgbench -i -s 50 (5,000,000-row bulk COPY) now completes 100% where it previously aborted at 20%. Signed-off-by: Greg Burd <greg@burd.me>
…y kernel heap so a forked backend's rename survives cross-AS drele() dentry_alloc() already wraps its dentry + d_path allocations in a fork_arena::kernel_heap_scope so shared dentry-cache infrastructure stays off the per-address-space COW fork arena. dentry_move() (the rename path) did NOT: its dp->d_path = strdup(path) ran unguarded, so a rename issued by a forked PostgreSQL backend allocated the new d_path in that backend's COW-private fork arena. When the last reference to that dentry later dropped in a DIFFERENT address space (AS0 postmaster or a sibling backend), drele() -> free(dp->d_path) routed the pointer to fork_arena::free(), which read the chunk header at that arena VA through the freeing AS's page tables. After the COW divergence that VA holds a different physical page there, so the header magic mismatched and recover() aborted: Assertion failed: h->magic == chunk_magic (core/fork_arena.cc: recover: 257) cloudius-systems#3 fork_arena::free cloudius-systems#4 drele (dp=...) at fs/vfs/vfs_dentry.cc:229 <- free(dp->d_path) cloudius-systems#8 vfs_file::close <- close(fd) PostgreSQL triggers this during a checkpoint that recycles WAL segments (rename of a pg_wal file), which is exactly the "N recycled" checkpoint that crashed a forked backend on the RAID-Z pool. Fix: wrap dentry_move()s d_path strdup in the same kernel_heap_scope as
… AS0 so the postmaster launches parallel workers (parallel-query hang) A forked PostgreSQL backend that signals the postmaster used kill(pid, sig) with pid == OSV_PID (2) -- the postmaster is the top-level app / AS0, whose getpid() is OSV_PID. The CONF_fork kill() path resolved a cross-process target via osv::fork::as_for_pid(pid) and returned ESRCH when that was null. AS0 is never entered into g_pid_as (only fork children are register_pid()'d), so as_for_pid(OSV_PID) == nullptr and kill(OSV_PID, sig) from a child returned ESRCH -- the signal was dropped. That is the lost wakeup behind the parallel-query hang. A parallel-query leader (itself a forked backend) launches workers via RegisterDynamicBackgroundWorker, which sets a shared request slot and then does kill(PostmasterPid, SIGUSR1) to make the postmaster fork the workers. With the signal dropped (ESRCH), the postmaster -- parked in its ServerLoop WaitEventSetWait/epoll_wait -- never sees the request, never forks a worker (debug1 logging showed ZERO "starting background worker" lines), and the leader waits forever in WaitForParallelWorkersToAttach. Block I/O was fully drained; this is a pure coordination lost-wakeup, same cross-AS/per-fork-child-pid routing class as the shipped SIGCHLD/SIGURG-latch fixes. (It also explains the noted SIGHUP-to-postmaster ESRCH from pg_reload_conf().) Fix (libc/signal.cc, #if CONF_fork): exclude OSV_PID from the cross-process target resolution -- treat it like the self/broadcast case so target_as stays nullptr and the handler runs in AS0 (mmu::kernel_address_space()), which is exactly where the postmaster's handler must run to poke its own latch/self-pipe and wake its ServerLoop. Never returns ESRCH for the top-level app. The later (pid == OSV_PID) wake_up_signal_waiters()/handler-in-AS0 logic is unchanged. conf_fork=0 does not compile this block (byte-identical). Validated: with max_parallel_workers_per_gather=2, a parallel Seq Scan Gather `select count(*) from pgbench_accounts` returns 10000000 (== the non-parallel reference), and EXPLAIN ANALYZE shows "Workers Planned: 2 / Workers Launched: 2" with both workers processing rows. Previously: 0 workers launched, hang. Signed-off-by: Greg Burd <greg@burd.me>
…_simple + zfs_space on OSv so ZIL replay works (sync=standard durability)
The OSv OpenZFS platform layer left zfs_write_simple() and zfs_space() as
ENOTSUP stubs. Both are on the ZIL replay path that runs on pool import after
a crash when a dataset has sync=standard (ZIL active, here on a SLOG vdev):
zfs_replay_write() (TX_WRITE, txtype 9) -> zfs_write_simple()
zfs_replay_truncate() (TX_TRUNCATE, txtype 10) -> zfs_space(F_FREESP)
Consequence on the RAID-Z (4xEBS raidz1 + NVMe SLOG + L2ARC): a kill -9 mid-txg
with sync=standard, then re-import, logged
"ZFS replay transaction error 95 [ENOTSUP], dataset pgdata, txtype 9/10"
and dropped every logged intent-log record (ZIL gave no durability); an earlier
build variant instead SPL-PANICked in abd_alloc_linear
(VERIFY3U(size <= SPA_MAXBLOCKSIZE)) on a log record it could not replay.
Either way sync=standard was not usably crash-recoverable through the ZIL on OSv
-- forcing sync=disabled and losing the SLOG's purpose.
Patch 0029 implements both the OSv way, reusing primitives that already work
here (so this is a single-process correctness fix, no fork/COW dependency,
correct for every config -- deliberately NOT CONF_fork-gated):
* zfs_write_simple: one-iovec struct uio over the kernel buffer +
zfs_write(zp, &zuio, O_SYNC, NULL) (the zfs_vop_write shape / FreeBSD's
vn_rdwr(UIO_SYSSPACE, IO_SYNC)); refresh vp->v_size from zp->z_size.
* zfs_space: F_FREESP wrapper delegating to the already-present
zfs_freesp() (zfs_znode_os.c), identical to FreeBSD's zfs_space.
As an upstream-openzfs source change this is a modules/open_zfs/patches/ entry
(git-apply'd to the pinned submodule at build time), NOT a submodule edit.
Validated on the RAID-Z: sync=standard, insert marker + checkpoint + settle,
kill -9, reboot -> NO replay error, NO panic, ZIL TX_WRITE/TX_TRUNCATE replay,
PG crash recovery completes, and the row survives crash+reboot.
Signed-off-by: Greg Burd <greg@burd.me>
…bling fork backends (PG shared_buffers catalog-read-zero) Stock PostgreSQL (shared_memory_type=mmap, the default) puts shared_buffers in one big anonymous MAP_SHARED segment the postmaster creates BEFORE forking backends. Only a few of its pages are touched pre-fork; the system-catalog index/relcache pages are first-faulted by an individual backend AFTER its fork. On OSv this produced the multi-backend catalog-read-zero wall: backend cloudius-systems#1 reads a catalog-index page fine, but sibling backends cloudius-systems#2+ (zero writes in between) read the SAME shared page as ZEROS ("pg_authid_rolname_index contains unexpected zero page", "cache lookup failed"), so HammerDB can't even authenticate vuser cloudius-systems#2. Hypervisor-independent (KVM and Firecracker identical). Root cause (core/mmu.cc): an anon MAP_SHARED page not present in the parent's page table at fork time is cloned as an EMPTY child PTE (clone_pt_level0 can only share what is already present). When a sibling backend later faults that VA, initialized_anonymous_page_provider::map() calls memory::alloc_page() and installs a FRESH, PRIVATE, ZERO page in that backend's AS only. Every AS that first-touches the page post-fork gets a different private zero page -- they never converge on the ONE physical page MAP_SHARED promises. Fix: back an anon MAP_SHARED vma with a shared_anon_page_provider that keys a process-global registry (identity kernel heap, like shm_file::_pages) on the absolute page VA. An anon MAP_SHARED region lives at the SAME VA in every fork AS (clone_address_space keeps identical VAs), so the VA uniquely identifies the shared page: the first AS to fault it allocates+records it; every later AS (sibling backend or parent) that faults the same VA maps THAT frame. All siblings converge. The provider tags the PTE pte_shared so per-AS teardown (free_child_pt_level0) does not free the jointly-owned frame -- without this, the first backend to exit freed the shared page and later siblings read freed/zeroed memory (the "second backend reads zeros" symptom). A searched (addr==NULL) mapping is relocated by allocate() after construction, so the provider base is re-keyed via anon_vma::update_shared_base() once the real VA is known. Regression test tests/tst-fork-shared-catalog.cc mirrors PG exactly: parent creates an anon MAP_SHARED region, forks THREE siblings; child A writes a page, siblings B and C (and the parent) must read A's data, not zeros. Reproduces the zero-read on HEAD; PASSES with the fix at -smp 2 and -smp 4 (5/5 runs, real concurrency, not gdb-serialized). Full fork suite still 0 failures (tst-fork/cow/deep/child-mmap/file-mmap/posix-shm/shared-write/shared-race/ serial/sigchld-latch/arena-freelist/irq-cow/preempt/socket/conn-socket/ timer-park-stress). All #if CONF_fork; non-fork byte-identical. Author: Greg Burd <greg@burd.me>
…kq quiescence OSv taskq_wait() enqueued a single barrier task and drained only that barrier. With an 8-worker system_taskq a free worker runs the barrier while the other 7 are still mid dnode_sync/dbuf_sync, so taskq_wait returned early and the syncing thread raced the still-running workers -> dbuf/dirty-record/refcount corruption (arc_write_done VERIFY3S refcount underflow + lfmutex owner-mismatch panics). ZFS documents this contract (dmu_objset.c PORTING note). Add taskqueue_drain_all() which waits until the queue is empty AND no worker is active (illumos semantics), and wire taskq_wait() to it. conf_fork unaffected.
…tity linear map + heap-allocate reclaimer waiter node for fork children Under CONF_fork, ZFS large allocations (>= huge_page, or the non-contiguous fallback) made under fork_arena::kernel_heap_scope (force_kernel_heap) went to map_anon(), which lands in the CURRENT thread address space app mmap slot (VA 0x2000..). A forked PostgreSQL backend making such an allocation (e.g. an 8k zio_buf / dbuf db_data) put the VA only in that child page tables; AS0 txg_sync / zio-completion threads and sibling backends faulted outside application on db_data. Force force_kernel_heap large allocations onto free_page_ranges (linear map, phys_mem 0x4000.., a kernel PML4 slot mapped verbatim in every AS) and never fall back to map_anon. Verified buffers move 0x2000 -> 0x4000. Companion: reclaimer_waiters::wait() wait_node is a stack local dereferenced by the AS0 reclaimer; for a fork-child waiter its COW-private stack VA GPFd wake_waiters(). Heap-allocate the node for fork children (identity, coherent), same pattern as coherent_wait_record. Non-fork path byte-identical. Signed-off-by: Greg Burd <greg@burd.me>
Describe the test host generically and reword internal comment tags to plain notes. No functional change.
…-swap asm (no post-swap %rbp stack read)
The fork COW context switch (switch_as branch) re-tested switch_as and reloaded
the FPU control words (fpucw/mxcsr) in C++ AFTER the inline asm swapped rsp/rbp
AND CR3 to the incoming fork child. The compiler spills switch_as and the
fpucw/mxcsr locals to %rbp-relative stack slots, and %rbp at that point already
points at the incoming child stack under the just-loaded child CR3. So the
post-swap code
mov -0x50(%rbp),%rdx ; cmp %rdx,-0x48(%rbp) (the switch_as re-test)
movzwl -0x52(%rbp),%eax ; mov %ax,-0x34(%rbp) (the else-branch reload)
reads and writes the child COW stack in an irq-off, non-preemptable window. A
COW write-fault there trips assert(sched::preemptable()) at arch/x64/mmu.cc:38,
and a mis-read switch_as takes the wrong branch and can ldmxcsr a garbage MXCSR
(reserved bits set) -> #GP, so the resumed child never runs again (a forked
backend parks in switch_to and is never rescheduled). The prior FPU fix
(1ae2602f9) hardened the reloaded VALUES for the switch_as==true path but left
the branch decision and the else-branch operands as %rbp-relative reads, still
resolved through the incoming child AS. Dormant when the surrounding code
compiles exactly like c8f9c82b; exposed once code layout shifts the fork timing.
Fix: emit emms + fldcw + ldmxcsr from kernel-identity-mapped .rodata statics
INSIDE the switch_as asm, right at the 1: resume label, then return. RIP-relative
.rodata is identically mapped in every address space (never COW), so nothing after
the CR3 swap touches a %rbp-relative (stack) slot: no COW fault, no mis-read. The
canonical control words are correct for every OSv thread (issue cloudius-systems#1020; switch_to
does no fxsave/fxrstor, so no per-thread FPU state is preserved across a switch
regardless), so this is semantically identical to the old fldcw(fpucw)/ldmxcsr(mxcsr)
for the AS-crossing case. The shared post-asm reload now runs only for the
non-crossing (same-AS) path, whose %rbp is the resumed threads own coherent stack.
…e self-pipe coherent Two coupled CONF_fork fixes so a forked PostgreSQL backend's own latch and ProcSignalBarrier self-signals reach it, without breaking the child-exit notification. kill(): a fork backend's OWN SIGURG (latch) and SIGUSR1 (ProcSignalBarrier) self-signal must run its handler in the caller's own copy-on-write address space, so the handler sets the pending flag / pokes the self-pipe in the copy the backend actually reads. Restrict the self-routing to SIGURG/SIGUSR1: the child-exit SIGCHLD that fork emulation raises with kill(getpid(), SIGCHLD) from a dying child must NOT be diverted, or the parent's reaper runs against the child's torn-down mapping and faults, and the parent never reaps the exited child (it then never finishes startup). as_for_pid() also gates to a live registered child, so a child mid-teardown falls back to the top-level table. pipe2(): allocate the pipe_buffer and pipe_file on the identity kernel heap so the self-pipe backing a latch is coherent across every fork address space (a cross-backend wakeup writes the pipe from another address space while the waiting backend polls the same pipe). Same rule as the epoll and file containers. Both are inside #if CONF_fork, so a non-fork build is byte-identical. Validated: with the routing present, boot-to-ready 10/10 on a fresh recordsize=8k pool; CREATE DATABASE + DROP DATABASE completes 5/5 (was an indefinite wait for the emitter's own ProcSignalBarrier); pgbench -c8 0 failed transactions.
Two independent bugs prevented building the fork-enabled kernel
(conf_fork=1) on GCC 11.x, and also broke a clean conf_fork=0 build on
both GCC 11 and GCC 13:
1. core/mmu.cc defined anon_vma::~anon_vma() unconditionally, gating only
the body with #if CONF_fork, while include/osv/mmu.hh declares the
destructor only under #if CONF_fork. When CONF_fork is 0 the class
does not declare the destructor but the .cc still defines one, which
is an ISO C++ violation ("definition of implicitly-declared
destructor") rejected by both GCC 11 and GCC 13. Move the #if CONF_fork
to wrap the whole destructor definition so it matches the declaration.
2. Makefile auto-generates the kernel configuration with
$(shell make -f conf/Makefile -j1 config), which did not forward
conf_fork. GNU make does not propagate a command-line variable
override into a $(shell ...)-invoked sub-make, so the Kconfig option
(def_bool driven by the conf_fork env var) always resolved to
CONF_fork=0. A conf_fork=1 build therefore produced CONF_fork=0 and
silently failed to compile the fork sources. Forward conf_fork to the
config sub-make.
Both changes are gated on CONF_fork / conf_fork. conf_fork=0 output is
unchanged: at CONF_fork=1 the mmu.cc change is a preprocessor no-op
(the -g0 core/mmu.o is byte-identical), and forwarding an empty conf_fork
yields the same generated config as before.
Verified: fork+ZFS kernel compiles and links clean and the resulting
image boots (ZFS root mounted) on both GCC 11.5 and GCC 13.3.
Signed-off-by: Greg Burd <greg@burd.me>
…o a forked backend does not fault in elf::resolve_pltgot OSv resolves a dynamic object's PLT/GOT lazily: the first call to an imported function traps into elf::object::resolve_pltgot(), which looks the symbol up and writes the resolved address into the object's .got.plt slot. Under CONF_fork a forked child runs in a copy-on-write clone of the parent address space, and two facets of the lazy resolver are not coherent across that clone, which surfaces as a NULL dereference inside elf::resolve_pltgot in a forked child (e.g. a PostgreSQL backend under a heavy stored-procedure workload). Root cause and fix, both CONF_fork-gated (conf_fork=0 emits identical code): 1) COW-stale GOT. The writable file-backed segment that holds .data/.got/.got.plt is demand-paged. The dynamic linker RELOCATES that GOT in memory at load (object::relocate / relocate_pltgot rewrite each slot by the object base and install the resolver trampoline pointer and the object pointer in pltgot[1]). Those writes diverge the page from its on-disk bytes. If such a relocated page is not resident when clone_address_space() clones the parent, the child inherits an empty leaf PTE and its first access re-reads the ORIGINAL, unrelocated bytes from the file (pltgot[1]=0, raw jump slots). The child's lazy resolver then dereferences a NULL elf::object and faults. Fix: mmap_populate writable file-backed segments at load so every relocated page is resident in the parent; clone_address_space then COW-shares the relocated pages and the child always reads the correct GOT. OSv does not swap and a written file-private page becomes anonymous, so a populated page stays resident through fork. 2) Symbol-set node in the child COW arena. When a forked child is first to resolve a cross-object slot, resolve_pltgot() inserts into the shared elf::object's _used_by_resolve_plt_got set. Without care the set node is allocated from the child's private COW arena while the set lives on the shared identity heap; a sibling or the top-level address space later walking that set dereferences a node mapped only in the child's (possibly already reclaimed) address space. Fix: allocate the set node under a kernel_heap_scope so it lands on the identity heap, coherent across every fork address space. Adds tests/tst-fork-pltgot.cc: a forked child is first to resolve a batch of libc/libm PLT symbols the parent never called, plus a parent-and-reaped-children cross-object stress of the shared symbol set. Passes at -smp 4 and -smp 8; boot-to-ready 10/10 unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / tracking — DO NOT MERGE AS ONE UNIT
This is the full PostgreSQL-on-OSv integration branch (
integ/pg-fork-zfs, 67 commits). It is filed as a draft tracking PR to make the completed fork-completeness + ZFS-coherence work visible with its dependency chain, NOT for review as one 67-commit unit.It will be split into the logical follow-on stack (S1..S6 below) once its bases merge. Until then this branch is the integration reference that proves stock, unmodified PostgreSQL boots, forks its aux processes, serves rows, and sustains concurrent load on OSv over OpenZFS-on-NVMe.
Depends on (must merge first)
The fork trilogy (opt-in,
CONF_fork, off by default):process: implement thread-backed fork()/vfork()/execve()/waitpid()mm: per-child copy-on-write address space for fork()(stacked on process: implement thread-backed fork()/vfork()/execve()/waitpid() (opt-in, off by default) #1455)libc: honor POSIX default-ignore disposition for SIGCHLD/SIGURG/SIGWINCHAnd, for the ZFS-coherence + PG-on-ZFS fixes:
zfs: selectable in-kernel ZFS (BSD or OpenZFS 2.4.2) via upstream submodule + patch seriesThis branch already contains the #1423 commits and the fork-trilogy commits inline (it was built on top of them); the split PRs below will be rebased to sit on top of the merged bases so each is independently reviewable.
The intended split (S1..S6), in dependency order
Every fork-side commit on this branch is prefixed by its destination
(
[fork-stack / CONF_fork],fork:,fork arena:,mm/fork:,mmu:,process:), and every ZFS-side commit by[#1423 / OpenZFS]/zfs(openzfs):. The split follows those groupings:S1 — AS-aware mmap allocation path (stacks on #1455 + #1456)
Thread
address_space*through the mmu allocation path (allocate/map_anon/find_hole/evacuate/protect/unmap/mprotect/msync + vma_range_set), defaulting to
the kernel AS so the non-fork path stays byte-identical; give a child AS its own
vma_range_set. Fixes fork-child mmap landing in the global vma_list invisiblyto the child fault handler (the W-mmap wall). Also
fork: preserve file_vma type in clone_address_space.Key commits:
mmu: make the mmap allocation path address-space aware,fork: preserve file_vma type in clone_address_space (fix wild-branch).S2 — Per-child COW arena + same-VA COW stack (stacks on #1456)
Private COW-able heap arena for fork children; COW 2 MB large pages; same-VA COW
stack for deep call chains; lock-free arena allocator; eager arena population;
identity-heap TLS/syscall stacks; intrusive zombie reaping (no alloc on the
lifecycle path).
Key commits: the
mm/fork:trio + the ninefork arena:commits.S3 — fd inheritance across fork (stacks on S1/S2)
Child gets its own reference on inherited fds; keep the shared fd slot when the
owner closes an fd a live child still holds (backend connection-socket wall).
Key commits:
fork: give the child its own reference on inherited fds,fork: keep the shared fd slot when the owner closes an fd a live child inherited.S4 — Per-process signals + timer parking (stacks on #1457)
Per-process signal dispositions + deliver blocked SIGCHLD to the handler
(postmaster PM_RUN wall); park app-thread timers off the per-CPU list across
address spaces (IRQ-context COW wall).
Key commits:
fork: per-process signal dispositions + deliver blocked SIGCHLD,fork: park app-thread timers off the per-CPU list across address spaces.S5 — Cross-AS coherence (kernel structures) (stacks on S1..S4)
Route kernel wait-records / list nodes / latches that a forked backend and AS0
must both see onto the identity kernel heap: rwlock read-waiters, signal-waiters,
rcu_defer wait_records, epoll containers + watcher lists, semaphores, waitqueues,
renamed-dentry d_path,
kill(OSV_PID)routing to AS0 for parallel workers,anonymous MAP_SHARED coherence for
shared_buffers, POSIX-shm/DSM/ramfs/netRX+TX coherence, thread-stack coherence, application_runtime + atfork handlers.
Key commits: the
fork: cross-AS coherence ...set + the[fork-stack / CONF_fork] route ... onto the identity heapset.S6 — ZFS-coherence + PG-on-ZFS durability fixes (stacks on S5 and #1423)
Route ZFS taskqueue + large kmem allocations to the identity linear map/heap so
forked-backend writes don't deadlock; cross-AS coherence for the ZFS
I/O-completion path (bio, zio/SPL heap, libsolaris.so statics); virtio-blk req
coherence. Plus the
[#1423 / OpenZFS]durability patches (0028 vnode v_sizerefresh, 0029 zfs_write_simple/zfs_space for ZIL replay, 0030 single-threaded
inline ARC eviction, taskq_wait quiescence) — these amend #1423, not the fork
stack, per the standing rule.
Split & landing rule (from the program's standing rules)
shipping fork PRs (process: implement thread-backed fork()/vfork()/execve()/waitpid() (opt-in, off by default) #1455/mm: per-child copy-on-write address space for fork() (opt-in, stacked on #1455) #1456/libc: honor POSIX default-ignore disposition for SIGCHLD/SIGURG/SIGWINCH #1457) to strengthen them.
pr/openzfs-draft).(S1..S6 above), gated
CONF_fork; the non-fork OSv path stays byte-identical.Blocks / part of
Part of the PostgreSQL-on-OSv North Star (Milestone 1: stock unmodified PG at
parity with PG-on-Linux over ZFS). This branch is where the M1 walls were
root-caused and fixed; the app side is the osv-apps
postgres18-musldemo(held on this stack landing on master).
Status: WIP / DO-NOT-MERGE as one unit. Will be split into S1..S6 and each
sub-PR opened for review once #1455 / #1456 / #1457 and #1423 merge. Filed as a
draft so the dependency chain and the completed work are visible.