ci: auto-heal corrupt boost conan cache on self-hosted runners - #691
Closed
frstrtr wants to merge 6 commits into
Closed
ci: auto-heal corrupt boost conan cache on self-hosted runners#691frstrtr wants to merge 6 commits into
frstrtr wants to merge 6 commits into
Conversation
Add a conan-cache-guard composite action that runs "conan cache check-integrity boost/*" before each conan install and purges the boost package on any manifest mismatch, so the install rebuilds it clean. Fixes the recurring intermittent boost reds (missing asio/log headers, libboost_process/coroutine "expected but not built") caused by a partially-failed conan build leaving an incomplete boost package in the shared ~/.conan2 that persists and poisons later jobs. Wired before every conan install across build.yml, coin-matrix.yml, release.yml, codeql-analysis.yml (presence/build-mode conditions mirrored).
…ervice PATH) Windows self-hosted (VM217) has no bash on the service PATH, so the single shell: bash guard step errored with 'bash: command not found'. Split the composite action by runner.os: bash on POSIX, pwsh on Windows with the equivalent check-integrity/remove and exit 0 to preserve the never-fails invariant. Guard now runs on every lane including win-217.
…WinPS 5.1, no pwsh Core)
…olicy blocks unsigned powershell bodies) cmd has no execution-policy gate, mirroring the windows-2022 fork-fallback. if errorlevel 1 purges the corrupt boost package; exit /b 0 keeps the step non-failing on a healthy or empty cache.
…rupt The per-job profile pin (sed compiler.cppstd=20) sets what conan install REQUESTS but not what it SELECTS: an intact gnu17-built boost in the reused self-hosted cache passes check-integrity, and Conan compatibility fallback re-selects it instead of rebuilding a cppstd=20 boost -> ABI mismatch reddens later jobs (boost.asio thread_pool_/partial_redirect_error, cobalt assertion). Guard now purges boost on any cached compiler.cppstd != 20, making the cppstd=20 pin durable at the cache layer. Mirrored on the Windows cmd leg.
…ache The cppstd!=20 conditional purge is correct, but on a cold VM905 cache all 8 Linux runners share one ~/.conan2 and purge+rebuild boost concurrently, racing (Reference already exists / boost headers deleted mid-build). Wrap the whole check+purge+rebuild in an flock so exactly one job rebuilds a clean cppstd=20 boost from source and the rest wait then cache-hit. Windows (single VM217 runner) has no herd, leg unchanged.
Owner
Author
|
Superseded by #700 (build.yml) + #702 (release.yml): per-job CONAN_HOME=${{ runner.temp }}/conan2 gives every job an isolated Conan cache, so concurrent jobs on the shared c2pool-build host no longer tear each others ~/.conan2. That removes the shared-cache race this guards decontamination step tried to police — and the guards conan-remove was itself destructive/racy against sibling jobs building in the same shared cache. No shared cache to decontaminate means no backstop needed. Closing; reopen if a torn cache recurs under the isolated-home model. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Recurring intermittent boost/conan reds across CI + release builds. The self-hosted runners reuse
~/.conan2across jobs; a partially-failed conan build leaves an incomplete boost package that persists and poisons later jobs. Different pieces go missing per run (same class):boost/asio/detail/timed_cancel_op.hpp: No such file(bch/abla: MSVC-portable 128-bit muldiv (fixes c2pool-bch Windows build) #688 dgb)boost/log/detail/header.hpp: No such file(bch smoke, master c79e4be)libboost_process.a: No such fileboost_coroutine "expected to be built but not built"(dash smoke)Intermittent because it depends on which runner/cache state a job lands on.
Fix
New composite action
.github/actions/conan-cache-guardruns before everyconan install:check-integrityvalidates the cached package against its manifest, so it catches ANY missing header/lib — not just the named ones — and auto-purges so the followingconan installrebuilds boost clean. No-op on a healthy/empty cache; declares its ownshell: bashso it is cross-platform (Linux/mac/Windows-217). Never fails the build itself.Wired before all conan install sites: build.yml (6), coin-matrix.yml (1), release.yml (2), codeql-analysis.yml (1) — presence/build-mode conditions mirrored.
Verification
check-integrity boost/*RC=0.check-integritydetects the corruption class (manifest-vs-disk mismatch) andconan remove boost/*purges cleanly.Reviewer: @Integrator — no self-merge; awaiting operator push-approval.