[ci] Windows test support#4885
Draft
gamesh411 wants to merge 33 commits into
Draft
Conversation
gamesh411
changed the title
gamesh411
force-pushed
the
windows-test-support
branch
14 times, most recently
from
8081c64 to
7219423
Compare
8 tasks
The codechecker_common/compatibility/multiprocessing.py shim provided platform-conditional imports (multiprocess on macOS/Win, stdlib on Linux). Replace with direct multiprocess imports everywhere since multiprocess is now a hard dependency on all platforms. multiprocess is a fork of stdlib multiprocessing that uses dill instead of pickle, enabling serialization of closures and complex objects needed for spawn-based workers. No behavioral change: multiprocess uses fork by default on Linux (same as stdlib multiprocessing). The start method is not changed in this commit.
On non-Linux platforms, fork() is unsafe due to macOS Obj-C runtime constraints (Security.framework loaded via SQLAlchemy -> asyncio -> ssl -> libcrypto triggers fork-safety crashes). Windows has no fork() at all. Python 3.14 will change the default start method on all platforms. Changes: - Set multiprocess start method to 'spawn' on non-Linux (cli.py) - Add _build_worker_server() to reconstruct HTTPServer in each spawn worker from serializable args (server.py) - Pass listening socket via DupFd to spawn workers (server.py) - Add logging initializers to Pool workers in analysis_manager, pre_analysis_manager, and log_parser (spawn workers do not inherit parent's logging configuration) Linux behavior is unchanged: fork is still used, no import overhead, no worker reconstruction needed.
- Normalize gcc/clang output quotes in log parser tests (macOS clang uses different quoting than Linux gcc) - Skip ld-logger specific tests on non-Linux (LD_PRELOAD is Linux-only) - Use os.sep for path assertions instead of hardcoded '/' - Add platform guards for Linux-specific checker tests
The JSON compilation database can use either 'command' (string) or 'arguments' (list) format. tu_collector only handled 'command', failing silently on the 'arguments' format. Parse both formats correctly by joining the arguments list when 'command' is absent.
- Use subprocess.run() instead of Popen.communicate() for git commands (clearer error handling) - Handle git template path differences across platforms - Fix Makefile test targets for portable invocation
- Replace blind sleep(5) with port-readiness polling (socket connect loop to localhost:3000) - Capture mock server stdout/stderr to log file for debugging - Use sys.executable for subprocess to ensure correct Python - Use 127.0.0.1 instead of localhost in OAuth test config (avoids DNS resolution delays on some platforms) - Add flush=True to startup print for immediate visibility
- Add HTTP readiness check as fallback for file-based detection: if the server port responds to HTTP (even 404), consider it ready. This handles cases where the log file message is delayed due to output buffering or slow worker initialization. - Add fatal error detection: fail fast if server output contains DB initialization failures instead of waiting for full timeout. - Allow CI to override worker counts via CC_TEST_API_WORKERS and CC_TEST_TASK_WORKERS env vars for tuning on slow infrastructure. - Pass port to wait_for_server_start for HTTP probe. The HTTP readiness timeout is subject to tuning based on CI infrastructure performance. Current observations: macOS GitHub Actions runners have ~40ms per-file I/O (133x slower than local NVMe), causing spawn worker imports to take ~36s each.
Task management tests previously used fixed sleep() calls to synchronize with async task state transitions. This is inherently racy: too short and the task has not been picked up, too long and it has already completed. Replace with poll-based approach: - _poll_status() polls getTaskInfo every 1s until expected state or timeout (120s on macOS, 30s on Linux) - For RUNNING checks: create long (600s) task, poll until RUNNING, cancel when done - For COMPLETED/FAILED: create short (1s) task, poll until terminal state (worker is already warm from prior tests) - Completion waits in filter tests use poll loops too This makes tests deterministic regardless of worker startup time while remaining fast on Linux (poll returns in 1-2s).
Add macOS runner (macos-latest) to CI: - Analyzer job: matrix with ubuntu-24.04 and macos-latest - Web (macOS) job: sqlite-only, 1+1 workers (CC_TEST_API_WORKERS=1 CC_TEST_TASK_WORKERS=1) to reduce spawn startup overhead - Tools job: matrix with macOS, skip Linux-only steps (build-logger needs LD_PRELOAD, bazel-compile-commands needs Bazel setup) macOS jobs use continue-on-error to avoid blocking Linux CI. Worker count rationale: on macOS CI runners (3 CPU, 7 GB RAM), each spawn worker imports 887 .pyc files at ~40ms each (~36s per worker). With 1+1 workers, server startup takes ~42s vs ~84s with 2+2. This is a CI infrastructure constraint (VM storage latency), not a code issue — locally startup is 1.6s. Adds install-deps-macos.sh for brew-based dependency installation.
gamesh411
force-pushed
the
windows-test-support
branch
from
7219423 to
be130dd
Compare
Add analyzer and web (sqlite) test jobs on windows-latest. Both use continue-on-error since Windows support is exploratory. Uses bash shell and choco for LLVM/cppcheck installation. Worker count set to 1+1 (same rationale as macOS: spawn worker import overhead on potentially slow CI storage). Python 3.10 to match other CI jobs. Depends on the spawn-based server from prior commits for the web tests to have any chance of working (Windows has no fork()).
gamesh411
force-pushed
the
windows-test-support
branch
from
be130dd to
f74ee25
Compare
python-ldap has no binary wheels for Windows and requires OpenLDAP C headers to build from source. Use pytest.importorskip to skip the test module cleanly when ldap is not installed.
gamesh411
force-pushed
the
windows-test-support
branch
from
bec7169 to
8ee7cf5
Compare
gamesh411
force-pushed
the
windows-test-support
branch
4 times, most recently
from
d63f80d to
6211588
Compare
gamesh411
force-pushed
the
windows-test-support
branch
from
6211588 to
3d9de0e
Compare
Co-authored-by: Endre Fülöp <gamesh411@users.noreply.github.com>
Move the Windows jobs out of the slow full matrix into a dedicated windows-iter workflow that only runs on the windows-test-support branch. Uses pytest-timeout (--timeout/--timeout-method=thread) so a hanging test fails fast with a thread stack dump instead of consuming the whole job timeout, and bumps checkout@v4/setup-python@v5. Co-authored-by: Endre Fülöp <gamesh411@users.noreply.github.com>
…tdin
The functional tests enable server auth and rely on HOME redirection so the
CodeChecker client finds the throw-away credentials written by login(). On
Windows os.path.expanduser('~') ignores HOME, so the client could not find
the credentials and fell back to an interactive getpass() prompt, hanging the
whole job until the timeout.
- Point CC_PASS_FILE/CC_SESSION_FILE into the test workspace in test_env()
(honored first on every platform), and have login()/logout() write the
credentials to that same path.
- Pass stdin=subprocess.DEVNULL to the test subprocess calls so any future
interactive prompt fails fast instead of hanging.
Co-authored-by: Endre Fülöp <gamesh411@users.noreply.github.com>
The official LLVM Windows package does not ship the 'diagtool' binary that CodeChecker uses to enumerate clang-diagnostic-* checkers. Without it these three checker-handling tests cannot succeed, so skip them when diagtool is not found (the same approach already used for python-ldap). Co-authored-by: Endre Fülöp <gamesh411@users.noreply.github.com>
CC_FORCE_SYNC_STORE (used by the functional tests) wrapped massStoreRun in a signal.SIGUSR1-based watchdog. SIGUSR1 does not exist on Windows, so even referencing it crashed the store with 'module signal has no attribute SIGUSR1'. Fall back to a plain synchronous store when the signal is unavailable; POSIX behavior is unchanged. Co-authored-by: Endre Fülöp <gamesh411@users.noreply.github.com>
Apply the same stdin=subprocess.DEVNULL guard to the check and store subprocess calls in check_and_store so no functional-test subprocess can block on an interactive prompt on Windows. Co-authored-by: Endre Fülöp <gamesh411@users.noreply.github.com>
gamesh411
force-pushed
the
windows-test-support
branch
from
4054fdd to
294f169
Compare
CodeChecker log / check -b relies on LD_PRELOAD which is unavailable on Windows. On Windows, check_and_store now generates compile_commands.json from the test source files and runs 'CodeChecker analyze' directly. Also restore full analyzer test suite in windows-iter (was filtered to ClangTidy-only for debugging).
gamesh411
force-pushed
the
windows-test-support
branch
4 times, most recently
from
fb62c4e to
d24f8d8
Compare
The background task worker process referenced signal.SIGHUP for graceful shutdown handling. SIGHUP does not exist on Windows, crashing the worker on startup. This caused massStoreRun to hang forever (no worker to process the task, pipe.recv() blocks indefinitely). Guard SIGHUP setup with hasattr check. On Windows, use terminate() for worker shutdown instead of SIGHUP.
gamesh411
force-pushed
the
windows-test-support
branch
from
d24f8d8 to
9b133aa
Compare
The store client packed source files into the zip via
os.path.join('root', f.lstrip('/')). On Windows, f is a drive-absolute
path (e.g. 'D:\\a\\f.h'); lstrip('/') leaves it unchanged and
os.path.join discards the 'root' prefix, so files land under the wrong
zip member name. The server reconstructs the expected location with
path_for_fake_root() (os.path.relpath(f, '/')), so the content is never
found, FileContent rows are missing, the report insert fails the
FOREIGN KEY constraint, and all reports are silently skipped, leaving
getRunResults empty.
Use os.path.relpath(f, '/') on the client to mirror the server logic.
Identical to the previous behaviour for POSIX absolute paths.
Homebrew's llvm@14 has no built-in knowledge of the macOS SDK location. Recent runner images dropped the implicit header path it fell back on, so libc++ headers using '#include_next <ctype.h>' fail with 'ctype.h' file not found, breaking analyzer and web (analysis) tests. Export SDKROOT from the active SDK in install-deps-macos.sh (written to GITHUB_ENV) so the build and test steps can compile. Verified locally: clang@14 fails without SDKROOT and succeeds with it. Also split the Windows web unit-test discovery into separate server/client pytest sessions (shared 'unit' package name collides in one session).
The CC_PASS_FILE/CC_SESSION_FILE redirect was added so login works on
Windows, where os.path.expanduser ignores HOME. Applying it on POSIX too
regressed products/test_config_db_share: that test starts a second server
and switches the client to its workspace by reassigning only HOME. With
the credential files pinned via CC_PASS_FILE/CC_SESSION_FILE to the first
workspace, the second server's session token was written to the wrong
file, so the client was unauthorized (getRunData returned None -> 401).
On POSIX, expanduser('~') honors HOME, so the HOME-based resolution
already works; gate the override to Windows to restore master behavior.
Two residual macOS failures from clang@14 vs the macOS 15 SDK: - python-ldap (and other pip C extensions) build universal2 by default, but clang@14 does not support '_Float16' for the x86_64 target, so the SDK's <math.h> fails to compile. Set ARCHFLAGS=-arch arm64 so native builds target only the host arch of the Apple Silicon runner. - test_buildcmd_escaping rebuilt the compile command without '-c', so it linked; clang@14 cannot link against the macOS 15 SDK. The test only checks that the escaped command compiles, so use '-fsyntax-only' (no link) on every platform, matching the test's documented intent.
The web unit suites are platform-independent and now pass on Windows (server: 31 passed/1 skipped, client: 3 passed). Run them as a real gate alongside the analyzer unit tests and the functional smoke test.
init_worker() called setup_logger() in every worker process. Under fork (Linux) workers already inherit the parent's logging configuration, so re-running logging.config.dictConfig() in each worker is unnecessary and was observed to cause intermittent analysis failures (CodeChecker analyze returning exit code 3 in test_compile_uniqueing). Only configure logging in workers when the start method is not fork (i.e. spawn on macOS/Windows, where the configuration is genuinely not inherited).
The GCC analyzer uses -fdiagnostics-format=sarif-stderr, so anything gcc
writes to stderr ends up in the SARIF stream. On the macOS runner, gcc@13
defaults to an older deployment target than the installed SDK, so the
Xcode toolchain's clang-based assembler prints a
'-Woverriding-deployment-version' warning to stderr. That warning was
appended after the SARIF JSON, making the file invalid JSON and failing
analyze_and_parse's gcc tests ('... .sarif is not a valid json file.
Extra data: line 2 column 1'). Pin MACOSX_DEPLOYMENT_TARGET to the SDK
version so no override (and no warning) is emitted. Reproduced and
verified locally with gcc@13 13.4.0.
python-ldap failed to link with 'ld: library ldap_r not found': the Homebrew llvm@14 clang (on PATH for the analyzer) cannot link against the current macOS SDK's .tbd libraries. Apple's clang handles the SDK natively, so set CC/CXX to it for pip builds. The analyzer selects its binary independently of $CC, so analysis still uses clang@14. Verified locally that Apple clang links -lldap_r -llber against the SDK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add analyzer and web (sqlite) test jobs on windows-latest. Both use continue-on-error since Windows support is exploratory. Uses bash shell and choco for dependency installation.