Skip to content

Commit dcfa196

Browse files
authored
feat: parallel batch scanner and input binding tokens (#125)
1 parent 8686ab1 commit dcfa196

9 files changed

Lines changed: 1117 additions & 9 deletions

File tree

AGENTS.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -130,14 +130,14 @@ make clean # Remove all build directories
130130

131131
```text
132132
include/DetourModKit/ # Public headers -- one per module
133-
scanner.hpp # AOB pattern scanning (whole-process executable/readable regions + module-scoped and host-EXE cascade; AOB syntax supports full-byte, full-wildcard, and per-nibble wildcard tokens; the region walk carries a pattern-length overlap across adjacent accepted regions so a signature straddling a protection split is still found; the prologue-recovery fallback recovers both E9 near-jump and FF25 indirect-jump overwritten prologues) with a SIMD-tiered prefilter and verify (the self-provided dmk_memchr prefilter and the verify both tier SSE2/AVX2 at runtime, plus an opt-in AVX-512F+AVX-512BW verify tier behind the DMK_ENABLE_AVX512 build option); in-code constant extraction (read_code_constant, Zydis-backed); string-reference xref resolver (find_string_xref: locate an immutable literal, then its unique RIP-relative reference -- a fast lea/mov shape scan by default, or that scan plus an opt-in Zydis broad sweep for cmp/push/no-REX shapes; returns the load site, the enclosing function, or the cached global pointer slot)
133+
scanner.hpp # AOB pattern scanning (whole-process executable/readable regions + module-scoped and host-EXE cascade; AOB syntax supports full-byte, full-wildcard, and per-nibble wildcard tokens; the region walk carries a pattern-length overlap across adjacent accepted regions so a signature straddling a protection split is still found; an opt-in fork-join batch scanner (scan_regions_batch whole-process, scan_module_batch one image, in scanner_parallel.cpp) resolves a batch of compiled patterns concurrently, sharing them read-only across a transient worker pool; the prologue-recovery fallback recovers both E9 near-jump and FF25 indirect-jump overwritten prologues) with a SIMD-tiered prefilter and verify (the self-provided dmk_memchr prefilter and the verify both tier SSE2/AVX2 at runtime, plus an opt-in AVX-512F+AVX-512BW verify tier behind the DMK_ENABLE_AVX512 build option); in-code constant extraction (read_code_constant, Zydis-backed); string-reference xref resolver (find_string_xref: locate an immutable literal, then its unique RIP-relative reference -- a fast lea/mov shape scan by default, or that scan plus an opt-in Zydis broad sweep for cmp/push/no-REX shapes; returns the load site, the enclosing function, or the cached global pointer slot)
134134
hook_manager.hpp # SafetyHook wrapper (inline, mid, and VMT hooks)
135135
async_logger.hpp # Lock-free MPMC queue logger
136136
logger.hpp # Synchronous singleton logger
137137
win_file_stream.hpp # Win32 shared-access file stream (CreateFile backend)
138138
config.hpp # INI configuration with callback setters
139139
config_watcher.hpp # Filesystem watcher (ReadDirectoryChangesW) for INI hot-reload
140-
input.hpp # Input polling (keyboard/mouse/XInput) + opt-in mouse-wheel capture and gamepad passthrough suppression
140+
input.hpp # Input polling (keyboard/mouse/XInput) + opt-in mouse-wheel capture and gamepad passthrough suppression; generation-checked BindingToken handles let a high-frequency consumer query a binding without the per-call name hash, failing closed on any reshape
141141
input_codes.hpp # Unified InputCode type and named key tables (incl. WheelUp/Down/Left/Right)
142142
srw_shared_mutex.hpp # Opaque Windows SRWLOCK SharedLockable wrapper; use for reader/writer locks that need native Windows semantics without exposing Windows headers
143143
memory.hpp # Memory read/write, sharded region cache, seh_read<T>, seh_resolve_chain/seh_read_chain<T>, plausible_userspace_ptr, PE module range
@@ -159,6 +159,7 @@ src/ # Implementation files. One .cpp per module, except a
159159
# cohesive module may split into sibling TUs sharing one
160160
# public header: scanner.cpp (scan engine) +
161161
# scanner_cascade.cpp (cascade resolver) +
162+
# scanner_parallel.cpp (fork-join batch scanner) +
162163
# code_constant.cpp (Zydis decode) +
163164
# string_xref.cpp (string-reference resolver; confines its
164165
# Zydis broad sweep like code_constant.cpp) via
@@ -327,6 +328,7 @@ The raw `Scanner::find_pattern(start_address, region_size, pattern)` overloads d
327328
- **Reverse-RTTI / self-heal tests:** `tests/test_rtti_dissect.cpp` covers the `rtti_dissect.hpp` surface (L1 `identify_pointee_type` through L4 `solve_fingerprint`), including the typed `identify_pointee_typed` error matrix (`BadSlotAddress` / `UnreadableSlot` / `NoRtti`) and the `identify_pointee_type_or` candidate-fallback composition (primary short-circuit, ordered fallback selection, and the load-bearing first-error-preserved guarantee). It rebuilds the `SyntheticVtable` COL/TD/vtable fixture in the test exe's PE range (so the shared prelude's module bound-check accepts it) and adds a `syn_heap_object` helper that `VirtualAlloc`s an object outside every module range, exercising the pointer-to-object branch and the cross-DLL resolvability case. The suite replaces the throwing global `operator new`/`delete` with a malloc/free pair plus a counter so the `_AllocatesNothing` tests can assert the heal and fingerprint paths are allocation-free (each warms the module-range cache first, then measures a second call; the heal case is measured on the window-scan path, not just the nominal short-circuit); the aligned `operator new` forms are left at their defaults so over-aligned allocations never cross-free. This replacement is process-wide for the test binary but harmless to other suites (malloc/free is a valid implementation and only this suite reads the counter).
328329
- **Reverse-RTTI by-name tests:** `tests/test_rtti_reverse.cpp` covers `rtti.hpp`'s reverse resolvers (`vtable_for_type`, `vtables_for_type`, `TypeIdentity`). It builds synthetic COL/TypeDescriptor/vtable fixtures in the test exe's data segment (so the prelude's module bound-check accepts them) and resolves them through a tight pool range for speed, with one case driving the real PE-section walk via `host_module_range()`. It exercises single- and multiple-inheritance (one name -> many sub-object vtables), the `COL.offset == 0` primary selection, ambiguous-name fail-closed, and the warm-cache identity path.
329330
- **Code-constant tests:** `tests/test_code_constant.cpp` covers `Scanner::read_code_constant`. It plants known x86-64 encodings (immediate, `[reg+disp]`, negative disp8, RIP-relative) into a committed page and a `ModuleRange` over it, asserting the decoded value, narrowing, RIP-relative absolute resolution, the always-decode (never the stale `nominal`) contract, and the typed failures (`DecodeFailed` / `UnexpectedShape` / `OperandOutOfRange`).
331+
- **Parallel batch scanner tests:** `tests/test_scanner_parallel.cpp` is a deliberate second suite for the scanner module (`scanner_parallel.cpp`), kept separate from `test_scanner.cpp` to isolate the threading concerns, mirroring the `test_memory_chain.cpp` split. It plants unique LCG-derived signatures into committed pages and covers `scan_regions_batch` / `scan_module_batch`: empty-batch, per-item input-order preservation regardless of address order, the fail-closed cases (null / empty pattern, zero occurrence, no match), per-item Nth-occurrence, `ScannerKind` selection (a data-only page is invisible to `Executable` but reached by `Readable`), module-range confinement and the invalid-range fail-closed path, and the worker-count knob (`1` serial, a fixed pool, an over-count clamp, and auto) all returning identical results across a many-pattern sweep that exercises the concurrent path.
330332
- **String-reference xref tests:** `tests/test_string_xref.cpp` covers `Scanner::find_string_xref` across both phase-2 modes. It builds three synthetic images: a single RWX page (`SyntheticImage`) for the default narrow shape scan; a split code/data image (`SplitImage`: an execute-readable code page plus a readable `PAGE_READWRITE` data page) so the broad Zydis sweep only ever decodes the code page; and a four-page `GappedCodeImage` whose executable region is split by a reserved interior page, the only fixture that makes `collect_executable_windows` return more than one window. Cases cover the narrow `lea` / `mov` resolves, the broad-only shapes (`cmp [rip+d], imm`, `push [rip+d]`, a no-REX `lea`, and the REX `lea` superset), the invariant that broad mode keeps default all-offset shape-scan coverage, the decode-failure byte-restart recovery, the `Utf16le` widening and the `require_terminator` prefix guard, the `EnclosingFunction` prologue back-scan (hit and miss), the `StringPointerSlot` store-xref mode (resolves the cached-pointer slot of a `lea reg, [rip+string]; mov [rip+slot], reg` pair, with the r8..r15 REX.R reconstruction, the register-mismatch / out-of-window / mov-load / broad-only `StoreNotFound` rejections), cross-window resolve and ambiguity accumulation, the invariant that the non-executable data page is never swept as code, and every fail-closed error (`EmptyQuery` / `InvalidRange` / `StringNotFound` / `StringAmbiguous` / `NoReference` / `AmbiguousReference` / `FunctionNotFound` / `StoreNotFound`) plus the `constexpr noexcept` totality of `string_xref_error_to_string`.
331333
- **Anchor registry tests:** `tests/test_anchors.cpp` covers `anchors.hpp` (`resolve`, `resolve_all`, `anchor_status_to_string`), one case per resolvable kind (Manual literal, CodeOperand, RipGlobal, VtableIdentity fail-closed) plus the `CallArgHome` `Unsupported` path and the parallel-report capacity behaviour. It also covers the optional post-resolve validator (accept, reject-fails-closed, context pass-through both ways, and the Manual / CallArgHome exemptions) and the `Quorum` kind (agreement, disagreement, one-signal-fails, null sub-anchor, rejected nesting, within-tolerance accept and reject, negative-tolerance fail-closed, the quorum's own validator applied to the corroborated value, and quorum propagation through `resolve_all`). The quorum-agreement cases pair genuinely independent sub-anchors (a `CodeOperand` plus a `Manual`), since the independence gate now rejects the dual-`Manual` shape; dedicated cases pin the `QuorumNotIndependent` rejections (pointer-equal, dual-Manual, same-backend-config) versus distinct-candidate-array acceptance, the opt-in `validate_manual` and `require_validator` policies (with the Quorum exemption), and `assess_quality`. The per-game scan profile (`profile.hpp`) has its own suite, `tests/test_profile.cpp`: `apply_profile` broad-widening, `order_candidates` permutations, `candidate_order_to_string` totality, and the fail-closed (non-substituting) deny-list at the anchor and quorum levels.
332334
- **Test fixture pattern:** Each suite uses a `::testing::Test` subclass with `SetUp()`/`TearDown()` for temp file cleanup. Temp file paths must include the process ID (`_getpid()`) and a counter to avoid collisions when CTest runs tests in parallel as separate processes.
@@ -363,7 +365,7 @@ PATH="/c/msys64/mingw64/bin:$PATH" ./build/mingw-debug/tests/DetourModKit_tests.
363365

364366
| Module | Thread safety | Hot-path mechanism |
365367
|--------|--------------|-------------------|
366-
| Scanner | Stateless -- inherently safe | N/A (startup only) |
368+
| Scanner | Stateless -- inherently safe; the opt-in batch scanner (`scan_regions_batch` / `scan_module_batch`) shares immutable `CompiledPattern`s read-only across a transient fork-join worker pool (no per-pattern mutation; `compile_anchor()` must precede the batch), writes each result slot from one worker via an atomic cursor, and joins before returning | N/A (startup only; the batch scanner is setup/control-plane, never callback-safe) |
367369
| HookManager | SRWLOCK-backed reader/writer locks; two-phase shutdown (disable under shared lock, clear under exclusive lock), both phases walking `m_hook_creation_order` in reverse so inline/mid hooks layered on one address unwind newest-first instead of in bucket order; `m_mutator_gate` blocks new mutators (including all VMT operations) during teardown; CAS on `m_shutdown_called` serializes shutdown/remove_all_hooks; double-checked fast-fail on `m_shutdown_called` in all mutators; every public mutation or teardown entry point also fails closed (`HookError::ReentrantCallRejected`, a false/zero result, or a logged no-op for void lifecycle calls) when the per-thread reentrancy guard is set, so a call from inside a `with_*` callback returns or no-ops instead of recursively acquiring the non-recursive lock; the batch toggles (`enable_hooks`/`disable_hooks`/`*_all`) collect their log lines under the lock and emit them after release so a synchronous sink flush or a blocking async overflow never stalls an exclusive acquirer; destructor fallback (when `DMK_Shutdown()` was not called) acquires `m_mutator_gate` exclusively, flips `m_shutdown_called`, drains readers via exclusive `m_hooks_mutex`, then clears the maps -- under loader lock it pins the module and swaps each map's contents into heap storage allocated via `new (std::nothrow)` so the storage outlives the destructor without ever draining, mirroring the leak-on-loader-lock discipline used in `Logger::shutdown_internal` and `ConfigWatcher::~ConfigWatcher` | `shared_lock` SRWLOCK reader for `with_inline_hook()` |
368370
| Logger | `atomic<shared_ptr>` for lock-free async reads; `shutdown_internal` and `disable_async_mode` are safe across repeated shutdown / enable_async_mode cycles: when the writer thread has to be detached under loader lock, the module is pinned and the `shared_ptr<AsyncLogger>` is moved into a per-call permanent cell (normal path: `new (std::nothrow)`; fallback path: non-CRT permanent storage), so a heap allocation failure cannot drop the last handle while the writer may still be running | Single atomic load on log level check |
369371
| AsyncLogger | Lock-free MPMC queue (Vyukov-style); post-join drain on shutdown (at most one message per producer can be lost in the nanosecond race between drain and force-zero -- accepted trade-off to avoid atomic overhead on every enqueue); a producer wakes a parked writer through a seq_cst pending-count/flag handshake (`m_pending_messages` is made non-zero before the queue slot is published, the writer publishes `m_writer_waiting` before checking that count and blocking, and the producer notifies under `m_flush_mutex` only when the flag is set), so the busy-writer hot path stays lock-free and syscall-free yet a push can never strand a message until the flush-interval timeout; timestamp caching in write batches | Atomic sequence numbers per slot; flag-gated writer wakeup |
@@ -382,6 +384,7 @@ These are called at 60+ fps from game hook callbacks. Never add allocations, exc
382384

383385
- `InputPoller::is_binding_active(index)` -- `shared_lock` (uncontended SRWLOCK reader, guards the `m_active_states` pointer swap on reshape) + single `memory_order_relaxed` load
384386
- `InputPoller::is_binding_active(name)` -- `shared_lock` + hash lookup + `memory_order_relaxed` load per matching binding (typically 1-3)
387+
- `InputPoller::is_binding_active(token)` -- `shared_lock` + generation compare + `memory_order_relaxed` load per cached binding index (no name hash); a stale token (generation mismatch after a reshape) fails closed before any index read. `acquire_binding_token(name)` itself is setup/control-plane (it copies the index set and may allocate); mint once, query per frame
385388
- `HookManager::with_inline_hook()` -- `shared_lock` SRWLOCK reader
386389
- `Logger::log()` level check -- single atomic load
387390
- `Logger::log()` async enqueue -- atomic shared_ptr load + lock-free queue push

0 commit comments

Comments
 (0)