fix(scanner): repair prologue-fallback cascade recovery

[tkhquang]

Summary

The cascade resolver's hooked-prologue fallback had two correctness bugs that made recovery of an already inline-hooked target either fail or resolve to a wrong address. This fixes both and hardens the formatter config so doc-blocks are not mangled.

• The fallback dropped the candidate's | anchor offset, so a |-anchored Direct candidate recovered an address short by that offset. The offset is now carried through the fallback resolve, so recovery matches the unhooked direct scan exactly.
• The E9 destination was required to lie inside a loaded module, which rejected every SafetyHook trampoline (VirtualAlloc'd outside every image). The destination is now gated on a committed, execute-readable page, recovering real trampolines while still rejecting unmapped or data-only targets.
• Tooling: .clang-format now exempts hang-indented Doxygen doc-blocks from reflow via CommentPragmas, so the formatter no longer mangles them while still reflowing plain // comments.

Adds regression tests for trampoline recovery outside any module, data-only destination rejection, and |-anchor offset preservation. Updates the docs and the public contract. No public API change.

Summary by CodeRabbit

• Documentation

  • Clarified address-validation semantics for cascade resolution and prologue fallback logic.
  • Refined code-formatting guidelines for documentation blocks.

• Bug Fixes

  • Improved prologue fallback to properly validate executable addresses, allowing trampoline destinations outside the loaded module while maintaining safety checks.

• Tests

  • Added comprehensive test coverage for prologue fallback scenarios including anchor offset handling and cross-module trampoline validation.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR refactors the cascade resolver's hooked-prologue fallback to relax jump destination validation from module-containment to executable-address checks, adds an is_executable_address() helper, refactors the prologue pattern builder to operate on pre-parsed patterns, updates resolution logic to honor anchor-offset semantics, and adds comprehensive test coverage with RAII memory utilities.

Changes

Prologue Fallback Destination Relaxation

Layer / File(s)	Summary
Executable address validation helper src/scanner_internal.hpp, src/scanner.cpp	New internal function is_executable_address() validates whether an address is on committed, executable memory using VirtualQuery and page protection flags, providing a reusable gate for prologue-fallback destination validation.
Hooked prologue pattern builder refactor src/scanner_cascade.cpp	Refactors pattern builder to accept pre-parsed CompiledPattern instead of token string reparsing, introduces PROLOGUE_PATCH_BYTES and append_hex_byte helpers, enforces literal-tail validation, and integrates new builder into scan_candidates_hooked_prologue. Removes #include <windows.h>.
Prologue fallback resolution with new destination validation src/scanner_cascade.cpp	Updates acceptance criteria to validate jump destinations as executable addresses instead of module-resident pointers, and reconstructs match addresses using anchor-offset semantics (match_addr + original->offset) before calling resolve_candidate_match.
Comprehensive test coverage with RAII infrastructure tests/test_scanner.cpp	Introduces VirtualFreeDeleter and VirtualPagePtr for automatic memory cleanup, updates existing prologue-fallback tests to use RAII, and adds new tests validating executable-destination acceptance, data-only-page rejection, anchor-offset preservation, and out-of-module trampolines.
API and design documentation include/DetourModKit/scanner.hpp, docs/misc/aob-signatures.md	Updates API documentation to reflect new destination validation semantics (executable address check, not module containment) and clarifies anchor-offset handling in resolved addresses.
Formatting configuration and tooling .clang-format, AGENTS.md	Adds CommentPragmas regex to preserve Doxygen doc-block wrapping and updates tooling guidance to document formatter exemption behavior and manual edit expectations.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• tkhquang/DetourModKit#69: Both PRs modify the scanner prologue-fallback/cascade resolver behavior and its tests—one hardens prologue-fallback ambiguity via bounded hit counting, while this one refactors hooked-prologue fallback and relaxes destination validation constraints.
• tkhquang/DetourModKit#80: Both PRs modify the hooked-prologue fallback behavior in the cascade resolver, including what constitutes an applicable/acceptable rebuilt E9 pattern and corresponding tests/docs.
• tkhquang/DetourModKit#67: This PR fixes hooked-prologue cascade resolution to use anchored match_addr + offset, while that PR corrects find_pattern/scan_executable_regions to apply pattern.offset exactly once with related offset-behavior tests.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main correctness fix in the changeset—repairing the prologue-fallback cascade recovery logic with specific focus on the recovered detour mechanism.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.clang-format:
- Around line 15-21: The current CommentPragmas regex ("CommentPragmas: '^ IWYU
pragma:|^ {3,}'") is too broad and prevents reflow for any comment line with 3+
leading spaces; narrow it to only match Doxygen hang-indented continuation lines
by changing the value of CommentPragmas to something that targets the Doxygen
continuation prefix (e.g. a pattern matching three leading spaces followed by an
asterisk and optional space). Update the CommentPragmas entry in .clang-format
(the CommentPragmas setting) to that tighter regex (for example: '^ IWYU
pragma:|^ {3}\\*\\s?') so ordinary indented // comments are not exempted.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 17a7580c-ffb2-4cbe-88a4-d9f47c15be87

📥 Commits

Reviewing files that changed from the base of the PR and between 44112e2 and f3d867e.

📒 Files selected for processing (8)

• .clang-format
• AGENTS.md
• docs/misc/aob-signatures.md
• include/DetourModKit/scanner.hpp
• src/scanner.cpp
• src/scanner_cascade.cpp
• src/scanner_internal.hpp
• tests/test_scanner.cpp