Skip to content

Commit fb416e3

Browse files
bghgaryCopilot
andauthored
Apps/ModuleLoadTest: add boot-time module load regression test (#1666)
[Created by Copilot on behalf of @bghgary] Adds a new `Apps/ModuleLoadTest` harness that asserts BabylonNative does not load unexpected native modules on boot. Motivating case: catching regressions like `dbghelp.dll` being introduced (currently loaded by bx's `DbgHelpSymbolResolve` static initializer). ### How it works - **Pre-static-init baseline.** A platform-specific callback captures the loaded-module set before any C++ static initializer in this binary runs. A `main()`-entry baseline would miss `dbghelp.dll`, since bx's static initializer fires before `main()`. - Windows: TLS callback in `.CRT$XLB`. - macOS: `__attribute__((constructor(101)))` function ordered before normal static initializers. - Linux: `.init_array` entry via the same constructor priority mechanism. - **Post-boot snapshot.** The harness drives BN to a steady boot state (graphics device up, all polyfills + plugins initialized, one frame rendered) and snapshots again. - **Asymmetric assertion.** We fail only on unexpected *new* modules. Missing-from-delta is environmental variance (GPU SKU, OS patch, launch environment, config) and is not a regression. - **Debug / debugger SKIP.** Debug config and debugger-attached runs print a SKIP and exit 0 — they load a materially different module set and would produce confusing FAILs otherwise. - **Launch-env allow-list.** GPU driver ICDs (NVIDIA/Intel/AMD on Windows, Mesa software-renderer versioned libs on Linux) and VS-injected DLLs (`kernel.appcore.dll`, `microsoft.internal.warppal*`) are filtered via `IsAllowedOptionalModule` so devs see the same verdict from a VS Ctrl-F5 run as CI does from a plain `cmd` / terminal launch. ### Platforms - **Windows** — primary implementation in `App.Win32.cpp`. - **macOS** — `App.Apple.mm`; golden list seeded with the ARM64 paravirt runner's delta (`appleparavirtgpumetaliogpufamily`, `iogpu`). - **Linux** — `App.X11.cpp`; runs under `xvfb-run` in CI. Golden list seeded with the stable Mesa/X11/DRI set (21 entries); versioned Mesa libs (`libgallium-*.so`, `libllvm.so.*`) are matched via prefix allow-list. CI wires the test into `build-windows.yml`, `build-macos.yml`, and `build-linux.yml` (the Linux step wraps with `xvfb-run`). ### Local verification (Windows) - RelWithDebInfo: PASS. - Release: PASS (identical delta to RelWithDebInfo). - Debug: SKIP (by design). - VS Ctrl-F5 (no debugger): PASS (after launch-env filter). - VS F5 (debugger): SKIP (by design). ### CI status Fully green across all three platforms (Win32, macOS, Ubuntu). The `GetExpectedBootModules()` lists are seeded from live CI runs; future drift (e.g. new runner images, additional configs) will surface as the same kind of fail-with-delta this test is designed to produce, at which point we append to the golden list and re-push. Current Windows list includes `dbghelp.dll` (pre-existing) so the test passes — removing it is a separate follow-up. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 063ea09 commit fb416e3

14 files changed

Lines changed: 1149 additions & 0 deletions

.github/workflows/build-linux.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,13 @@ jobs:
6767
ulimit -c unlimited
6868
xvfb-run ./UnitTests
6969
70+
- name: Module Load Test
71+
if: ${{ !inputs.enable-sanitizers }}
72+
run: |
73+
cd build/Linux/Apps/ModuleLoadTest
74+
ulimit -c unlimited
75+
xvfb-run ./ModuleLoadTest
76+
7077
- name: Upload Rendered Pictures
7178
if: always()
7279
uses: actions/upload-artifact@v6

.github/workflows/build-macos.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,9 @@ jobs:
5151
- name: Build UnitTests macOS
5252
run: cmake --build build/macOS --target UnitTests --config RelWithDebInfo
5353

54+
- name: Build ModuleLoadTest macOS
55+
run: cmake --build build/macOS --target ModuleLoadTest --config RelWithDebInfo
56+
5457
- name: Enable Core Dump
5558
run: sudo sysctl -w kern.corefile=%N.core.%P
5659

@@ -60,6 +63,13 @@ jobs:
6063
ulimit -c unlimited
6164
./UnitTests
6265
66+
- name: Run ModuleLoadTest macOS
67+
if: ${{ !inputs.enable-sanitizers }}
68+
run: |
69+
cd build/macOS/Apps/ModuleLoadTest/RelWithDebInfo
70+
ulimit -c unlimited
71+
./ModuleLoadTest
72+
6373
- name: Upload UnitTests Core Dumps
6474
if: failure()
6575
uses: actions/upload-artifact@v6

.github/workflows/build-win32.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,13 @@ jobs:
136136
cd build\${{ steps.vars.outputs.solution_name }}\Apps\UnitTests\RelWithDebInfo
137137
UnitTests
138138
139+
- name: Module Load Test
140+
if: ${{ !inputs.enable-sanitizers }}
141+
shell: cmd
142+
run: |
143+
cd build\${{ steps.vars.outputs.solution_name }}\Apps\ModuleLoadTest\RelWithDebInfo
144+
ModuleLoadTest
145+
139146
- name: Stage UnitTests Crash Dump
140147
if: failure()
141148
shell: powershell

Apps/CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ endif()
99

1010
if((WIN32 AND NOT WINDOWS_STORE) OR (APPLE AND NOT IOS AND NOT VISIONOS) OR (UNIX AND NOT ANDROID AND NOT APPLE))
1111
add_subdirectory(UnitTests)
12+
add_subdirectory(ModuleLoadTest)
1213
endif()
1314

1415
if((WIN32 AND NOT WINDOWS_STORE AND GRAPHICS_API STREQUAL D3D11) OR (APPLE AND NOT IOS AND NOT VISIONOS))

Apps/ModuleLoadTest/CMakeLists.txt

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
set(SOURCES
2+
"Source/App.h"
3+
"Source/App.cpp"
4+
"Source/ModuleSnapshot.h")
5+
6+
if(WIN32 AND NOT WINDOWS_STORE)
7+
list(APPEND SOURCES
8+
"Source/App.Win32.cpp"
9+
"Source/ModuleSnapshot.Win32.cpp")
10+
elseif(APPLE AND NOT IOS)
11+
list(APPEND SOURCES
12+
"Source/App.Apple.mm"
13+
"Source/ModuleSnapshot.macOS.mm")
14+
find_library(JAVASCRIPTCORE_LIBRARY JavaScriptCore)
15+
set(ADDITIONAL_LIBRARIES PRIVATE ${JAVASCRIPTCORE_LIBRARY})
16+
elseif(UNIX AND NOT ANDROID)
17+
list(APPEND SOURCES
18+
"Source/App.X11.cpp"
19+
"Source/ModuleSnapshot.Linux.cpp")
20+
else()
21+
# Unsupported platform (UWP, Android, iOS) — mobile/store bundling
22+
# changes module-load semantics. Silently skip.
23+
return()
24+
endif()
25+
26+
add_executable(ModuleLoadTest ${SOURCES})
27+
28+
target_link_libraries(ModuleLoadTest
29+
PRIVATE AppRuntime
30+
PRIVATE Blob
31+
PRIVATE Canvas
32+
PRIVATE Console
33+
PRIVATE GraphicsDevice
34+
PRIVATE NativeEngine
35+
PRIVATE NativeEncoding
36+
PRIVATE Window
37+
PRIVATE XMLHttpRequest
38+
${ADDITIONAL_LIBRARIES})
39+
40+
add_test(NAME ModuleLoadTest COMMAND ModuleLoadTest CONFIGURATIONS Release RelWithDebInfo)
41+
42+
# See https://gitlab.kitware.com/cmake/cmake/-/issues/23543
43+
add_custom_command(TARGET ModuleLoadTest POST_BUILD
44+
COMMAND ${CMAKE_COMMAND} -E $<IF:$<BOOL:$<TARGET_RUNTIME_DLLS:ModuleLoadTest>>,copy,true> $<TARGET_RUNTIME_DLLS:ModuleLoadTest> $<TARGET_FILE_DIR:ModuleLoadTest> COMMAND_EXPAND_LISTS)
45+
46+
set_property(TARGET ModuleLoadTest PROPERTY FOLDER Apps)
47+
source_group(TREE ${CMAKE_CURRENT_SOURCE_DIR} FILES ${SOURCES})
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
#include "App.h"
2+
#include "ModuleSnapshot.h"
3+
4+
#include <Babylon/DebugTrace.h>
5+
#include <Babylon/Graphics/Device.h>
6+
7+
#import <Metal/Metal.hpp>
8+
9+
#include <sys/sysctl.h>
10+
#include <sys/types.h>
11+
#include <unistd.h>
12+
13+
#include <iostream>
14+
#include <string_view>
15+
16+
#import <Foundation/Foundation.h>
17+
18+
namespace ModuleLoadTest
19+
{
20+
// Apple equivalent of IsDebuggerPresent() — non-invasive.
21+
// https://developer.apple.com/library/archive/qa/qa1361/_index.html
22+
bool IsBeingTraced()
23+
{
24+
int name[4] = { CTL_KERN, KERN_PROC, KERN_PROC_PID, getpid() };
25+
struct kinfo_proc info{};
26+
size_t size = sizeof(info);
27+
if (sysctl(name, 4, &info, &size, nullptr, 0) != 0)
28+
{
29+
return false;
30+
}
31+
return (info.kp_proc.p_flag & P_TRACED) != 0;
32+
}
33+
34+
const ModuleSnapshot& GetExpectedBootModules()
35+
{
36+
// Seeded from CI on macos-latest (ARM64 paravirtualized GPU runner).
37+
// Append entries here as new configs surface deltas.
38+
static const ModuleSnapshot kModules{
39+
"appleparavirtgpumetaliogpufamily",
40+
"iogpu",
41+
};
42+
return kModules;
43+
}
44+
45+
// On macOS the interesting per-SKU variation is the Metal/GPU driver
46+
// bundles (AMDMTLBronzeDriver, AppleIntelKBLGraphicsMTLDriver, ...), which
47+
// dyld loads by path under /System/Library/Extensions/. Base names still
48+
// differ between Apple Silicon and Intel. Seed a broad carve-out.
49+
bool IsAllowedOptionalModule(std::string_view name)
50+
{
51+
static constexpr std::string_view kPrefixes[] = {
52+
// Metal GPU driver bundles
53+
"amdmtl",
54+
"appleintel",
55+
"applem1",
56+
"applem2",
57+
"applem3",
58+
"nvmtl",
59+
// Ambient/IOSurface layer
60+
"iosurface",
61+
};
62+
for (const auto& prefix : kPrefixes)
63+
{
64+
if (name.size() >= prefix.size() && name.compare(0, prefix.size(), prefix) == 0)
65+
{
66+
return true;
67+
}
68+
}
69+
return false;
70+
}
71+
std::optional<Babylon::Graphics::Configuration> CreateGraphicsConfig()
72+
{
73+
// MTL::Device must outlive RunBoot. Park in function-local static
74+
// storage so it lives for the duration of the process.
75+
static NS::SharedPtr<MTL::Device> device = NS::TransferPtr(MTL::CreateSystemDefaultDevice());
76+
if (!device)
77+
{
78+
std::cout << "ModuleLoadTest: SKIP - no Metal device available." << std::endl;
79+
return std::nullopt;
80+
}
81+
82+
Babylon::DebugTrace::EnableDebugTrace(true);
83+
Babylon::DebugTrace::SetTraceOutput([](const char* trace) { NSLog(@"%s", trace); });
84+
85+
Babylon::Graphics::Configuration config{};
86+
config.Device = device.get();
87+
config.Width = 600;
88+
config.Height = 400;
89+
return config;
90+
}
91+
}
Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
#include "App.h"
2+
#include "ModuleSnapshot.h"
3+
4+
#include <Babylon/DebugTrace.h>
5+
#include <Babylon/Graphics/Device.h>
6+
7+
#include <Windows.h>
8+
9+
#include <iostream>
10+
#include <string_view>
11+
12+
namespace ModuleLoadTest
13+
{
14+
namespace
15+
{
16+
LRESULT WINAPI WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
17+
{
18+
return ::DefWindowProc(hWnd, msg, wParam, lParam);
19+
}
20+
}
21+
22+
bool IsBeingTraced()
23+
{
24+
return ::IsDebuggerPresent() != FALSE;
25+
}
26+
27+
// Expected set of modules loaded during BabylonNative boot, as a delta
28+
// from the baseline snapshot captured by the TLS callback before any C++
29+
// static initializer in this binary has run. Base names only, lower case.
30+
//
31+
// This list targets optimized builds (Release and RelWithDebInfo, which
32+
// produce identical module sets). Debug builds load additional debug CRT
33+
// and diagnostic DLLs and are not supported — main() returns a SKIP in
34+
// that config. CI runs only in RelWithDebInfo (see CMakeLists.txt's
35+
// `add_test ... CONFIGURATIONS Release RelWithDebInfo`).
36+
//
37+
// Launch-environment noise (e.g. VS's Start Without Debugging injecting
38+
// kernel.appcore.dll) is filtered via IsAllowedOptionalModule so devs
39+
// see the same verdict from Ctrl-F5 as CI does from a plain cmd launch.
40+
//
41+
// The assertion is asymmetric: we FAIL on modules not in this list (a new
42+
// module was pulled in — the regression signal we want), and we IGNORE
43+
// modules in this list that did not load (environmental variance across
44+
// GPU SKUs, Windows patch levels, VS vs cmd launch, etc.). This lets the
45+
// list be a permissive superset that works on both dev machines and CI.
46+
//
47+
// Motivating example: a dependency quietly pulling in dbghelp.dll on boot.
48+
// dbghelp.dll lives in System32, so a path-based OS filter would miss it;
49+
// we therefore golden-list the full delta (including OS modules) and only
50+
// allow a narrow name-pattern carve-out for GPU driver ICDs whose exact
51+
// names differ per runner SKU (see IsAllowedOptionalModule).
52+
const ModuleSnapshot& GetExpectedBootModules()
53+
{
54+
// Seeded from a local RelWithDebInfo run on Windows 11 x64 with D3D11
55+
// and Chakra. CI may add more entries for other Win32 configs
56+
// (V8/JSI/D3D12) — those should be appended as the draft PR runs.
57+
static const ModuleSnapshot kModules{
58+
"bcryptprimitives.dll",
59+
"cfgmgr32.dll",
60+
"crypt32.dll",
61+
"cryptbase.dll",
62+
"cryptnet.dll",
63+
"d3d10warp.dll",
64+
"d3d11.dll",
65+
"d3d11_3sdklayers.dll",
66+
"d3d12.dll",
67+
"d3d12core.dll",
68+
"d3d12sdklayers.dll",
69+
"d3dscache.dll",
70+
// TODO: bgfx loads dbghelp.dll at boot (callstack/crash helper). Drop this
71+
// entry once bgfx stops pulling it in.
72+
"dbghelp.dll",
73+
"dcomp.dll",
74+
"devobj.dll",
75+
"directxdatabasehelper.dll",
76+
"drvstore.dll",
77+
"dwmapi.dll",
78+
"dxcore.dll",
79+
"dxgi.dll",
80+
"dxgidebug.dll",
81+
"dxilconv.dll",
82+
"iertutil.dll",
83+
"imagehlp.dll",
84+
"msasn1.dll",
85+
"msctf.dll",
86+
"netutils.dll",
87+
"ntmarta.dll",
88+
"powrprof.dll",
89+
"profapi.dll",
90+
"rsaenh.dll",
91+
"setupapi.dll",
92+
"shcore.dll",
93+
"shell32.dll",
94+
"srvcli.dll",
95+
"umpdc.dll",
96+
"userenv.dll",
97+
"uxtheme.dll",
98+
"version.dll",
99+
"windows.storage.dll",
100+
"winmm.dll",
101+
"wintrust.dll",
102+
"wintypes.dll",
103+
"wldp.dll",
104+
};
105+
return kModules;
106+
}
107+
108+
// Name patterns for modules whose presence in the delta is allowed but
109+
// not required. Covers two classes:
110+
//
111+
// * GPU driver ICDs, whose exact base name depends on which GPU the
112+
// runner has (NVIDIA/Intel/AMD/ATI).
113+
// * Environmental/ambient DLLs that some launchers (e.g. Visual Studio's
114+
// Start Without Debugging) inject into the process but which a plain
115+
// cmd.exe launch does not. These are not introduced by BabylonNative.
116+
//
117+
// Prefixes match at the start of the base name; exacts match the whole
118+
// name.
119+
bool IsAllowedOptionalModule(std::string_view name)
120+
{
121+
static constexpr std::string_view kPrefixes[] = {
122+
// GPU driver ICDs
123+
"nv", // NVIDIA (nvoglv64.dll, nvapi64.dll, nvd3dum.dll, ...)
124+
"ig", // Intel (igdumd64.dll, igxelpicd64.dll, ...)
125+
"amd", // AMD (amdvlk64.dll, amdxc64.dll, ...)
126+
"atio", // AMD/ATI (atio6axx.dll, ...)
127+
// Ambient WARP variants
128+
"microsoft.internal.warppal",
129+
};
130+
static constexpr std::string_view kExact[] = {
131+
// GPU driver ICD
132+
"dxil.dll",
133+
// VS Start-Without-Debugging launch environment
134+
"kernel.appcore.dll",
135+
};
136+
for (const auto& prefix : kPrefixes)
137+
{
138+
if (name.size() >= prefix.size() && name.compare(0, prefix.size(), prefix) == 0)
139+
{
140+
return true;
141+
}
142+
}
143+
for (const auto& exact : kExact)
144+
{
145+
if (name == exact)
146+
{
147+
return true;
148+
}
149+
}
150+
return false;
151+
}
152+
std::optional<Babylon::Graphics::Configuration> CreateGraphicsConfig()
153+
{
154+
::SetConsoleOutputCP(CP_UTF8);
155+
156+
// bgfx D3D12 implementation requires an HWND to avoid a device refcount
157+
// leak on shutdown. Create a hidden window to satisfy that requirement.
158+
// Parked in function-local static storage so the handle lives for the
159+
// duration of the process.
160+
static WNDCLASSEX wc = { sizeof(WNDCLASSEX), CS_CLASSDC, WndProc, 0L, 0L,
161+
::GetModuleHandle(nullptr), nullptr, nullptr, nullptr, nullptr,
162+
"BabylonNativeModuleLoadTest", nullptr };
163+
::RegisterClassEx(&wc);
164+
static HWND hWnd = ::CreateWindow(wc.lpszClassName, "BabylonNativeModuleLoadTest",
165+
WS_OVERLAPPEDWINDOW, -1, -1, -1, -1, nullptr, nullptr, wc.hInstance, nullptr);
166+
167+
Babylon::DebugTrace::EnableDebugTrace(true);
168+
Babylon::DebugTrace::SetTraceOutput([](const char* trace) {
169+
::OutputDebugStringA(trace);
170+
::OutputDebugStringA("\n");
171+
});
172+
173+
Babylon::Graphics::Configuration config{};
174+
config.Window = hWnd;
175+
config.Width = 600;
176+
config.Height = 400;
177+
return config;
178+
}
179+
}

0 commit comments

Comments
 (0)