Speedup identity, urandom, and clock_gettime

This introduces an EL1-only shim_data block holding a host-published
cache: identity slots (pid/ppid/uid/euid/gid/egid/tid), urandom-eligible
fd bitmap, a 4 KiB urandom ring with head/tail/lock, and a 32-bit
attention bitmask. The EL1 shim assembly serves identity and urandom
1-byte reads inline without trapping to the host; the existing HVC #5
forwarder is taken only when attention is raised, when a non-urandom fd
is consulted, or when the ring needs a host-side refill.

Measured at 1 M iterations under the new tests/bench-hot-syscalls.c :
  getpid/getppid/getuid/geteuid/getgid/getegid/gettid :   47 ns/op
  clock_gettime via __kernel_clock_gettime vDSO       :    3.7 ns/op
  read(/dev/urandom, 1 byte)                          :  134 ns/op
  clock_gettime via SVC fallback                      : 2056 ns/op

The vDSO clock_gettime trampoline now seeds CLOCK_{MONOTONIC,REALTIME}
anchors back-to-back from a single SVC fallback, so the fast path serves
either clockid after one warm-up call. The X9/ELR_EL1 gate runs before
the host wall-clock samples so the anchor inherits no positive bias from
the seeding round trip.

Integrity boundary around the new cache:
- The shim_data block is mapped MEM_PERM_RW_EL1_ONLY (AP[2:1]=00) by
  both bootstrap and execve so EL0 cannot read or store the bytes
  directly. /proc/self/maps reports PROT_NONE for [shim-data] to
  match what guest dereferences would observe.
- gva_translate_perm refuses MEM_PERM_EL1_ONLY descriptors on
  guest-behalf access in both the L2 block and L3 page walk paths.
  read(fd, shim_data_gva, n) now returns EFAULT instead of letting
  the host spoof the cache.
- elf_map_segments takes an explicit infra reserve range and rejects
  PT_PHDR copies or PT_LOAD segments whose page-aligned write extent
  intersects it, closing a host-side overwrite path through the ELF
  loader that bypassed page-table permissions.
- A new EL1 data-abort recover handler in shim.S catches strb faults
  inside named urandom write ranges (caused by a racing EL0 munmap
  or mprotect), drops the inner exception frame, releases the ring
  lock, and returns EFAULT to EL0.

Cred publish is bracketed so concurrent fast-path readers see a
consistent snapshot. The attention word splits into ATTN_BIT_SIGTIMER
(0x1), ATTN_BIT_CRED (0x2), and ATTN_BIT_TRACE (0x4). CRED_BRACKETED
ORs the CRED bit, runs the setuid/setgid mutator, publishes the four
cred slots, then ANDs the CRED bit off. shim_globals_attn_or uses
__ATOMIC_SEQ_CST so the mutator's publish stores cannot become globally
visible before the attention bit on weakly-ordered ARM64; the AND clear
stays __ATOMIC_RELEASE because release pairs with the shim LDAR for the
publish-then-clear order. vdso_attention_or mirrors the same ordering.

Signal and itimer path support the lane discipline:
- attention_guest is now _Atomic so signal_init's NULL clear during
  the execve reset window pairs with attention_raise's acquire load
  on any sibling thread.
- signal_set_itimer writes expiry and interval before the release
  store of .active, matching the field order already used by the
  virt and prof setters. Consumers that ACQUIRE-load .active without
  holding sig_lock now never observe armed=true with stale fields.
- New signal_attention_needed() OR-reads the three guest itimer
  .active fields plus an unblocked-deliverable signal hint so the
  HVC epilogue's recompute decides accurately whether the next call
  may stay on the fast path.

The fd-table publication paths that feed the urandom bitmap are
serialized so a pathological sibling close+reopen on the same guest
fd cannot make the EL1 fast path consult a stale bit:
- fd_refresh_urandom_bitmap snapshots (type, linux_flags) AND publishes
  the bitmap bit inside the same fd_lock critical section.
- fd_alloc_opened_host and duplicate_guest_fd install linux_flags,
  dir, seals, and the urandom bit only after re-acquiring fd_lock and
  confirming the slot's (type, host_fd) tuple still matches the just-
  allocated values. On mismatch (the slot was reallocated by a
  sibling) the install is skipped and any cloned DIR* is closed to
  avoid a leak.
- The host-side urandom cache replaces its single global mutex with
  a per-fd lock embedded in urandom_cache_t, initialized by io_init()
  from syscall_init. Concurrent urandom reads on different fds no
  longer serialize on one mutex.
- sys_readv on /dev/urandom now triggers shim_globals_refill_urandom_ring
  on the slow path, matching sys_read so readv consumers do not leave
  the shim ring drained.

Author: jserv

Makefile

@@ -23,6 +23,7 @@ SRCS := \
     core/elf.c \
     core/stack.c \
     core/vdso.c \
+    core/shim-globals.c \
     core/bootstrap.c \
     core/rosetta.c \
     core/sysroot.c \
@@ -160,6 +161,24 @@ $(BUILD_DIR)/test-pthread: tests/test-pthread.c | $(BUILD_DIR)
 	@echo "  CROSS   $< (with -lpthread)"
 	$(Q)$(CROSS_COMPILE)gcc -D_GNU_SOURCE -static -O2 -o $@ $< -lpthread
 
+# test-shim-cred-race spawns a pthread reader while the main thread
+# toggles setresuid; the reader spins on the identity fast path.
+$(BUILD_DIR)/test-shim-cred-race: tests/test-shim-cred-race.c | $(BUILD_DIR)
+	@echo "  CROSS   $< (with -lpthread)"
+	$(Q)$(CROSS_COMPILE)gcc -D_GNU_SOURCE -static -O2 -o $@ $< -lpthread
+
+# test-shim-urandom-smp spawns N pthreads racing on a shared FD_URANDOM
+# slot to exercise the shim's LDXR/STXR head-advance under contention.
+$(BUILD_DIR)/test-shim-urandom-smp: tests/test-shim-urandom-smp.c | $(BUILD_DIR)
+	@echo "  CROSS   $< (with -lpthread)"
+	$(Q)$(CROSS_COMPILE)gcc -D_GNU_SOURCE -static -O2 -o $@ $< -lpthread
+
+# test-shim-urandom-toctou races mprotect(PROT_NONE) against urandom
+# reads to exercise the EL1 data abort recovery path. Needs pthreads.
+$(BUILD_DIR)/test-shim-urandom-toctou: tests/test-shim-urandom-toctou.c | $(BUILD_DIR)
+	@echo "  CROSS   $< (with -lpthread)"
+	$(Q)$(CROSS_COMPILE)gcc -D_GNU_SOURCE -static -O2 -o $@ $< -lpthread
+
 # test-fuse-basic runs a guest daemon thread and consumer in one process
 $(BUILD_DIR)/test-fuse-basic: tests/test-fuse-basic.c | $(BUILD_DIR)
 	@echo "  CROSS   $< (with -lpthread)"

src/core/bootstrap.c

@@ -20,6 +20,7 @@
 
 #include "core/bootstrap.h"
 #include "core/rosetta.h"
+#include "core/shim-globals.h"
 #include "core/stack.h"
 #include "core/startup-trace.h"
 #include "core/vdso.h"
@@ -31,6 +32,7 @@
 #include "syscall/internal.h"
 #include "syscall/path.h"
 #include "syscall/proc.h"
+#include "syscall/signal.h"
 
 #include "debug/log.h"
 
@@ -95,20 +97,25 @@ static void register_elf_segment_regions(guest_t *g,
     }
 }
 
-/* Publish shim, shim-data, heap, stack-guard, and stack regions to the
+/* Publish shim, shim-data, heap, stack-guard, and stack regions to
  * /proc/self/maps view, and invalidate the null page and stack-guard PTEs.
- * Shared by guest_bootstrap_prepare and guest_bootstrap_rosetta_post_reset;
- * the caller registers ELF or rosetta segments separately because those
- * differ between aarch64 and rosetta guests.
+ * Shared by guest_bootstrap_prepare and guest_bootstrap_rosetta_post_reset; the
+ * caller registers ELF or rosetta segments separately because those differ
+ * between aarch64 and rosetta guests.
  */
 static void register_runtime_regions(guest_t *g, size_t shim_bin_len)
 {
     guest_region_add(g, g->shim_base, g->shim_base + shim_bin_len,
                      LINUX_PROT_READ | LINUX_PROT_EXEC, LINUX_MAP_PRIVATE, 0,
                      "[shim]");
+    /* shim_data is mapped privileged-only (AP[2:1]=00) in the page tables; the
+     * EL1 shim has full RW but EL0 cannot read or write. Report PROT_NONE in
+     * /proc/self/maps so guest tooling treats it as inaccessible, matching what
+     * dereferencing the GVA actually does (translation fault -> EL0 SIGSEGV
+     * path).
+     */
     guest_region_add(g, g->shim_data_base, g->shim_data_base + BLOCK_2MIB,
-                     LINUX_PROT_READ | LINUX_PROT_WRITE, LINUX_MAP_PRIVATE, 0,
-                     "[shim-data]");
+                     LINUX_PROT_NONE, LINUX_MAP_PRIVATE, 0, "[shim-data]");
 
     if (g->brk_base < g->brk_current) {
         guest_region_add(g, g->brk_base, g->brk_current,
@@ -247,8 +254,11 @@ static bool load_interpreter(guest_t *g,
     }
 
     boot->interp_base = g->interp_base;
+    uint64_t infra_lo = g->interp_base - INFRA_RESERVE;
+    uint64_t infra_hi = g->interp_base;
     if (elf_map_segments(&boot->interp_info, boot->interp_resolved,
-                         g->host_base, g->guest_size, boot->interp_base) < 0) {
+                         g->host_base, g->guest_size, boot->interp_base,
+                         infra_lo, infra_hi) < 0) {
         log_error("failed to map interpreter segments");
         if (interp_host_temp)
             unlink(boot->interp_resolved);
@@ -278,20 +288,28 @@ static bool build_boot_regions(mem_region_t *regions,
      */
     if (!append_boot_region(regions, nregions, g->shim_base,
                             g->shim_base + shim_bin_len, MEM_PERM_RX) ||
+        /* shim_data is EL1-only: the guest must not directly read or write the
+         * identity cache, attention flag, urandom bitmap, or ring, any of which
+         * would let it spoof its own syscall results. The EL1 shim itself has
+         * full RW. /proc/self/maps still lists [shim-data] (region tracking is
+         * independent of EL0 access), but EL0 dereferences fault to the SIGSEGV
+         * path.
+         */
         !append_boot_region(regions, nregions, g->shim_data_base,
-                            g->shim_data_base + BLOCK_2MIB, MEM_PERM_RW) ||
+                            g->shim_data_base + BLOCK_2MIB,
+                            MEM_PERM_RW_EL1_ONLY) ||
         !append_boot_region(regions, nregions, VDSO_BASE, VDSO_BASE + VDSO_SIZE,
                             MEM_PERM_RX)) {
         return false;
     }
 
-    /* Rosetta guests never load the x86_64 ELF or its interpreter into
-     * guest memory; rosetta itself reads the target via fd 3 once it is
-     * running. Adding those segments to the page-table builder would emit
-     * ghost L2/L3 entries at the binary's x86_64 link address (typically
-     * 0x400000) pointing into uninitialized primary-buffer GPAs. The
-     * rosetta image's own segments are registered by rosetta_prepare's
-     * separate region append in the bootstrap caller.
+    /* Rosetta guests never load the x86_64 ELF or its interpreter into guest
+     * memory; rosetta itself reads the target via fd 3 once it is running.
+     * Adding those segments to the page-table builder would emit ghost L2/L3
+     * entries at the binary's x86_64 link address (typically 0x400000) pointing
+     * into uninitialized primary-buffer GPAs. The rosetta image's own segments
+     * are registered by rosetta_prepare's separate region append in the
+     * bootstrap caller.
      */
     if (!g->is_rosetta) {
         if (!append_elf_segment_regions(regions, nregions, &boot->elf_info,
@@ -370,12 +388,12 @@ int guest_bootstrap_prepare(guest_t *g,
         (unsigned long long) boot->elf_info.load_max,
         want_rosetta ? "x86_64-via-rosetta" : "aarch64");
 
-    /* Rosetta is statically linked at 0x800000000000 (128 TiB), beyond the
-     * 36 and 40-bit IPA ranges. Request 48-bit IPA up-front so the
-     * page-table builder can reach the rosetta segments. HVF clamps to its
-     * supported size; on M1 hosts the upstream hyper-linux audit confirms
-     * 48 is honoured even though the auto-detect default returns 36, so
-     * the request is non-fatal in either direction.
+    /* Rosetta is statically linked at 0x800000000000 (128 TiB), beyond the 36
+     * and 40-bit IPA ranges. Request 48-bit IPA up-front so the page-table
+     * builder can reach the rosetta segments. HVF clamps to its supported size;
+     * on M1 hosts the upstream hyper-linux audit confirms 48 is honoured even
+     * though the auto-detect default returns 36, so the request is non-fatal in
+     * either direction.
      */
     uint32_t req_ipa = want_rosetta ? 48 : 0;
     t0 = startup_trace_now_ns();
@@ -397,8 +415,8 @@ int guest_bootstrap_prepare(guest_t *g,
     if (want_rosetta) {
         /* Rosetta path: no x86_64 ELF segments are loaded into guest memory
          * (rosetta itself does that lazily once it starts running). brk and
-         * stack use the same defaults the aarch64 path falls back to when
-         * the binary sits at low VAs; the x86_64 binary's load_max would be
+         * stack use the same defaults the aarch64 path falls back to when the
+         * binary sits at low VAs; the x86_64 binary's load_max would be
          * meaningless here because nothing of it actually lives in primary
          * buffer GPA space.
          */
@@ -412,8 +430,11 @@ int guest_bootstrap_prepare(guest_t *g,
         boot->elf_load_base =
             (boot->elf_info.e_type == ET_DYN) ? PIE_LOAD_BASE : 0;
         t0 = startup_trace_now_ns();
+        uint64_t infra_lo = g->interp_base - INFRA_RESERVE;
+        uint64_t infra_hi = g->interp_base;
         if (elf_map_segments(&boot->elf_info, elf_host_path, g->host_base,
-                             g->guest_size, boot->elf_load_base) < 0) {
+                             g->guest_size, boot->elf_load_base, infra_lo,
+                             infra_hi) < 0) {
             log_error("failed to map ELF segments");
             return -1;
         }
@@ -664,9 +685,49 @@ int guest_bootstrap_create_vcpu(guest_t *g,
     HV_CHECK(hv_vcpu_set_sys_reg(vcpu, HV_SYS_REG_SP_EL0, sp_ipa));
     HV_CHECK(hv_vcpu_set_sys_reg(vcpu, HV_SYS_REG_SP_EL1, el1_sp));
 
-    /* CNTKCTL_EL1.EL0VCTEN | EL0PCTEN: allow EL0 to read CNTVCT_EL0 /
-     * CNTPCT_EL0. Required by the vDSO clock_gettime fast path (and is the
-     * default on native Linux), without which the guest gets 0 back from MRS.
+    /* Round-trip a sentinel through TPIDR_EL1 before installing the real
+     * value. Validates only the hv_vcpu_{set,get}_sys_reg pre-run round
+     * trip, not preservation across hv_vcpu_run -- the test-shim-identity
+     * microbench is the end-to-end check for that.
+     */
+    if (shim_globals_self_test(vcpu) < 0)
+        return -1;
+    /* TPIDR_EL1 -> shim_globals base, CONTEXTIDR_EL1 -> tid (== pid for the
+     * initial main thread). gettid fast path reads CONTEXTIDR_EL1 directly.
+     */
+    if (shim_globals_install_per_vcpu(vcpu, g, proc_get_pid()) < 0)
+        return -1;
+
+    /* Zero the shim-globals region and publish the initial identity so the very
+     * first getpid / getuid / etc. SVC #0 hits the cache instead of returning
+     * the all-zero seed. Future setuid/setgid paths refresh creds via
+     * cred_publish_after; fork-child has its own publish on the inherited
+     * identity.
+     */
+    shim_globals_init(g);
+    shim_globals_set_trace_enabled(g, verbose);
+    shim_globals_publish_pid(g, proc_get_pid(), proc_get_ppid());
+    shim_globals_publish_creds(g, proc_get_uid(), proc_get_euid(),
+                               proc_get_gid(), proc_get_egid());
+    /* Pre-fill the entropy ring so the first read(/dev/urandom) from the guest
+     * is served by the shim fast path with no cold-start HVC for refill.
+     */
+    shim_globals_refill_urandom_ring(g);
+    /* Register the singleton guest pointer so signal_queue and the itimer
+     * setters can raise the attention flag without threading g through every
+     * call site. signal_init clears this defensively; the first registration
+     * must run after both proc_init and shim_globals_init.
+     */
+    signal_set_shim_globals_guest(g);
+    /* Same singleton pattern but for the fd-table hooks that update the urandom
+     * bitmap. Must run before any FD_URANDOM-typed slot is allocated; bootstrap
+     * finishes before any guest syscall runs.
+     */
+    shim_globals_set_singleton(g);
+
+    /* CNTKCTL_EL1.EL0VCTEN | EL0PCTEN: allow EL0 to read {CNTVCT,CNTPCT}_EL0.
+     * Required by the vDSO clock_gettime fast path (and is the default on
+     * native Linux), without which the guest gets 0 back from MRS.
      */
     HV_CHECK(hv_vcpu_set_sys_reg(vcpu, HV_SYS_REG_CNTKCTL_EL1, 0x3ULL));
 

src/core/elf.c

@@ -208,8 +208,16 @@ int elf_map_segments(const elf_info_t *info,
                      const char *path,
                      void *guest_base,
                      uint64_t guest_size,
-                     uint64_t load_base)
+                     uint64_t load_base,
+                     uint64_t infra_lo,
+                     uint64_t infra_hi)
 {
+    /* Half-open intersection test for [a, a+alen) and [b, b+blen). When
+     * infra_lo == infra_hi the caller opted out (early bring-up before
+     * guest_t is wired up); the host-side writes that follow still get
+     * the existing guest_size bound check.
+     */
+    bool infra_active = infra_lo < infra_hi;
     FILE *f = fopen(path, "rb");
     if (!f) {
         perror(path);
@@ -264,6 +272,17 @@ int elf_map_segments(const elf_info_t *info,
         fclose(f);
         return -1;
     }
+    if (infra_active && phdr_dest < infra_hi &&
+        phdr_dest + ph_total > infra_lo) {
+        log_error(
+            "%s: program headers at 0x%llx overlap infra reserve "
+            "[0x%llx, 0x%llx)",
+            path, (unsigned long long) phdr_dest, (unsigned long long) infra_lo,
+            (unsigned long long) infra_hi);
+        free(ph_buf);
+        fclose(f);
+        return -1;
+    }
     memcpy((uint8_t *) guest_base + phdr_dest, ph_buf, ph_total);
 
     /* Copy PT_LOAD contents after AT_PHDR is in place; ET_DYN segments are
@@ -308,15 +327,34 @@ int elf_map_segments(const elf_info_t *info,
             return -1;
         }
 
-        /* Zero the full page-aligned segment extent, not only p_memsz.
-         * Linux guarantees zero-filled tail bytes in the last mapped page,
-         * and some dynamic linkers allocate from that page tail before they
-         * request more memory. Leaving stale bytes there leaks state across
-         * execve and corrupts the new image.
+        /* The host memset zeros PAGE_ALIGN_UP(memsz) bytes, not just memsz,
+         * so the infra-overlap check has to use the same rounded extent.
+         * Without the rounding here, a segment that ends just below
+         * infra_lo passes the check and still spills up to PAGE_SIZE-1
+         * bytes of zero into the infra reserve via the page tail.
          */
         uint64_t zero_len = PAGE_ALIGN_UP(memsz);
         if (gpa + zero_len > guest_size)
             zero_len = guest_size - gpa;
+        if (infra_active && gpa < infra_hi && gpa + zero_len > infra_lo) {
+            log_error(
+                "%s: segment at 0x%llx+0x%llx (zero-extent 0x%llx) overlaps "
+                "infra reserve [0x%llx, 0x%llx)",
+                path, (unsigned long long) gpa, (unsigned long long) memsz,
+                (unsigned long long) zero_len, (unsigned long long) infra_lo,
+                (unsigned long long) infra_hi);
+            free(ph_buf);
+            fclose(f);
+            return -1;
+        }
+
+        /* Zero the full page-aligned segment extent (zero_len computed above
+         * with guest_size and infra_reserve checks). Linux guarantees
+         * zero-filled tail bytes in the last mapped page, and some dynamic
+         * linkers allocate from that page tail before they request more
+         * memory. Leaving stale bytes there leaks state across execve and
+         * corrupts the new image.
+         */
         memset((uint8_t *) guest_base + gpa, 0, zero_len);
 
         /* Overlay initialized bytes after zeroing so BSS and page tail remain

src/core/elf.h

@@ -109,13 +109,20 @@ int elf_load(const char *path, elf_info_t *info);
  * Also copies program headers into guest memory for AT_PHDR.
  * load_base is added to all virtual addresses (0 for ET_EXEC at link addr,
  * non-zero for ET_DYN loaded at a chosen base).
+ * infra_lo and infra_hi delimit the runtime infra reserve (page-table pool,
+ * shim text, shim_data, vDSO). Any PT_LOAD or PT_PHDR copy whose destination
+ * intersects [infra_lo, infra_hi) is rejected: those writes go through
+ * host_base directly and would otherwise bypass the EL1-only page-table
+ * protection on shim_data. Pass 0,0 only when the guest_t is not yet built.
  * Returns 0 on success, -1 on failure.
  */
 int elf_map_segments(const elf_info_t *info,
                      const char *path,
                      void *guest_base,
                      uint64_t guest_size,
-                     uint64_t load_base);
+                     uint64_t load_base,
+                     uint64_t infra_lo,
+                     uint64_t infra_hi);
 
 /* Resolve a PT_INTERP path against a sysroot directory.
  * Tries three strategies: