ci: widen persistent self-hosted Conan home to coin-matrix, CI and gate lanes

frstrtr · frstrtr · commit dfcbb0b51dd1 · 2026-07-19T13:28:32.000Z
The Conan cache-lock flake is not CodeQL-specific. Job 88179349495 (Coin matrix / ltc smoke) and 88188892260 (CI / Linux x86_64) fail the same way: actions/cache reports a hit, the restore of the multi-hundred-MB conan2 tree onto VM905 then times out ("Failed to restore: The operation cannot be completed in timeout"), the home is left empty, and all eight runners fall into a simultaneous cold from-source rebuild whose I/O storm blows the 20s sqlite lock timeout ("Conan failed to acquire database lock"). Apply the same three-part fix everywhere the pattern appears: * CONAN_HOME on self-hosted becomes $HOME/.conan2-ci/$RUNNER_NAME -- persistent, so the cache-service restore is off the hot path, and keyed on the runner instance, which executes one job at a time, so it is never contended and keeps the per-job isolation of #704/#710. github-hosted forks keep RUNNER_TEMP/conan2 + actions/cache unchanged. * actions/cache conan2 steps are gated to runner.environment == github-hosted. * conan install gets a bounded 3x retry; remote downloads are the only network step left. Also scopes coin-matrix orphaned-tempfile cleanup to $CONAN_HOME instead of the shared ~/.conan2, which it was mutating out from under peer jobs, and repoints the CodeQL home from .conan2-codeql to the same per-runner path (that one was still shared across runners on concurrent refs). Windows (VM217) lanes are untouched: no restore flake reported there.
diff --git a/.github/workflows/bch-gate-g3a-regtest-block-production.yml b/.github/workflows/bch-gate-g3a-regtest-block-production.yml
@@ -34,7 +34,21 @@ jobs:
       - uses: actions/checkout@v6
 
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Check gate entrypoint presence
         id: presence
@@ -72,7 +86,8 @@ jobs:
         if: steps.presence.outputs.exists == '1'
         run: conan profile show -pr:a=ci/conan/linux-gcc13.profile
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         if: steps.presence.outputs.exists == '1'
         uses: actions/cache@v5
         with:
diff --git a/.github/workflows/bch-gate-g3b-genuine-activation.yml b/.github/workflows/bch-gate-g3b-genuine-activation.yml
@@ -31,7 +31,21 @@ jobs:
       - uses: actions/checkout@v6
 
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Check gate entrypoint presence
         id: presence
@@ -69,7 +83,8 @@ jobs:
         if: steps.presence.outputs.exists == '1'
         run: conan profile show -pr:a=ci/conan/linux-gcc13.profile
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         if: steps.presence.outputs.exists == '1'
         uses: actions/cache@v5
         with:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -48,7 +48,21 @@ jobs:
       # 1.90.0 torn-header flakes). RUNNER_TEMP is only valid at step scope, so
       # export it via $GITHUB_ENV -- a job-level `env: runner.temp` fails to parse.
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Install system dependencies
         if: runner.environment == 'github-hosted'
@@ -74,7 +88,8 @@ jobs:
       - name: Show committed Conan profile
         run: conan profile show -pr:a=ci/conan/linux-gcc13.profile
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         uses: actions/cache@v5
         with:
           path: ${{ runner.temp }}/conan2
@@ -90,7 +105,16 @@ jobs:
         run: rm -rf build_ci
 
       - name: Conan install
-        run: conan install . -pr:a=ci/conan/linux-gcc13.profile --build=missing --output-folder=build_ci
+        run: |
+          # Bounded retry: remote package downloads are the only network step
+          # left here; a transient CDN/remote hiccup must not red a good PR.
+          for attempt in 1 2 3; do
+            if conan install . -pr:a=ci/conan/linux-gcc13.profile --build=missing --output-folder=build_ci; then exit 0; fi
+            echo "conan install failed (attempt $attempt/3); retrying in $((attempt * 15))s"
+            sleep $((attempt * 15))
+          done
+          echo "conan install failed after 3 attempts" >&2
+          exit 1
 
       - name: Configure
         run: cmake -S . -B build_ci -DCMAKE_TOOLCHAIN_FILE=build_ci/conan_toolchain.cmake -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_BUILD_TYPE=Release -DCOIN_DGB=ON
@@ -194,7 +218,21 @@ jobs:
       # torn-cache reds). Export via $GITHUB_ENV -- a job-level `env: runner.temp`
       # fails to parse.
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Install system dependencies
         if: runner.environment == 'github-hosted'
@@ -220,7 +258,8 @@ jobs:
           conan profile detect --force
           sed -i 's/compiler.cppstd=.*/compiler.cppstd=20/' "$(conan profile path default)"
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         uses: actions/cache@v5
         with:
           path: ${{ runner.temp }}/conan2
@@ -236,7 +275,16 @@ jobs:
         run: rm -rf build_asan
 
       - name: Conan install
-        run: conan install . --build=missing --output-folder=build_asan --settings=build_type=Release
+        run: |
+          # Bounded retry: remote package downloads are the only network step
+          # left here; a transient CDN/remote hiccup must not red a good PR.
+          for attempt in 1 2 3; do
+            if conan install . --build=missing --output-folder=build_asan --settings=build_type=Release; then exit 0; fi
+            echo "conan install failed (attempt $attempt/3); retrying in $((attempt * 15))s"
+            sleep $((attempt * 15))
+          done
+          echo "conan install failed after 3 attempts" >&2
+          exit 1
 
       - name: Configure (Release + AsAN + UBSan)
         # NOTE 1: must match the build_type passed to `conan install` (Release).
@@ -325,7 +373,21 @@ jobs:
       # torn-cache reds). Export via $GITHUB_ENV -- a job-level `env: runner.temp`
       # fails to parse.
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Install system dependencies
         if: runner.environment == 'github-hosted'
@@ -351,7 +413,8 @@ jobs:
           conan profile detect --force
           sed -i 's/compiler.cppstd=.*/compiler.cppstd=20/' "$(conan profile path default)"
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         uses: actions/cache@v5
         with:
           path: ${{ runner.temp }}/conan2
@@ -367,7 +430,16 @@ jobs:
         run: rm -rf build_bch
 
       - name: Conan install
-        run: conan install . --build=missing --output-folder=build_bch --settings=build_type=Release
+        run: |
+          # Bounded retry: remote package downloads are the only network step
+          # left here; a transient CDN/remote hiccup must not red a good PR.
+          for attempt in 1 2 3; do
+            if conan install . --build=missing --output-folder=build_bch --settings=build_type=Release; then exit 0; fi
+            echo "conan install failed (attempt $attempt/3); retrying in $((attempt * 15))s"
+            sleep $((attempt * 15))
+          done
+          echo "conan install failed after 3 attempts" >&2
+          exit 1
 
       - name: Configure (-DCOIN_BCH=ON)
         run: cmake -S . -B build_bch -DCMAKE_TOOLCHAIN_FILE=build_bch/conan_toolchain.cmake -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_BUILD_TYPE=Release -DCOIN_BCH=ON
@@ -421,7 +493,21 @@ jobs:
       # torn-cache reds). Export via $GITHUB_ENV -- a job-level `env: runner.temp`
       # fails to parse.
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Install system dependencies
         if: runner.environment == 'github-hosted'
@@ -447,7 +533,8 @@ jobs:
           conan profile detect --force
           sed -i 's/compiler.cppstd=.*/compiler.cppstd=20/' "$(conan profile path default)"
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         uses: actions/cache@v5
         with:
           path: ${{ runner.temp }}/conan2
@@ -463,7 +550,16 @@ jobs:
         run: rm -rf build_bch_asan
 
       - name: Conan install
-        run: conan install . --build=missing --output-folder=build_bch_asan --settings=build_type=Release
+        run: |
+          # Bounded retry: remote package downloads are the only network step
+          # left here; a transient CDN/remote hiccup must not red a good PR.
+          for attempt in 1 2 3; do
+            if conan install . --build=missing --output-folder=build_bch_asan --settings=build_type=Release; then exit 0; fi
+            echo "conan install failed (attempt $attempt/3); retrying in $((attempt * 15))s"
+            sleep $((attempt * 15))
+          done
+          echo "conan install failed after 3 attempts" >&2
+          exit 1
 
       - name: Configure (Release + AsAN + UBSan, -DCOIN_BCH=ON)
         run: |
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
@@ -56,13 +56,13 @@ jobs:
           # the flake ("Failed to restore: The operation cannot be completed in
           # timeout"), and on a miss it forced a from-source boost rebuild too.
           # A local persistent home takes the network off the hot path.
-          # Dedicated path (NOT the shared ~/.conan2) preserves the torn-cache
-          # isolation from #704/#710; this workflow's concurrency group
-          # serialises CodeQL per ref, so nothing else writes this home.
+          # Keyed on the runner instance (NOT the shared ~/.conan2): a runner runs
+          # one job at a time, so this home is never contended even across refs,
+          # preserving the torn-cache isolation of #704/#710.
           if [ "${{ runner.environment }}" = "github-hosted" ]; then
             echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
           else
-            echo "CONAN_HOME=$HOME/.conan2-codeql" >> "$GITHUB_ENV"
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
           fi
 
       # ── C++ build (manual mode only) ─────────────────────────────────────
diff --git a/.github/workflows/coin-matrix.yml b/.github/workflows/coin-matrix.yml
@@ -38,7 +38,21 @@ jobs:
       - uses: actions/checkout@v6
 
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Check source presence
         id: presence
@@ -76,7 +90,8 @@ jobs:
         if: steps.presence.outputs.exists == '1'
         run: conan profile show -pr:a=ci/conan/linux-gcc13.profile
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         if: steps.presence.outputs.exists == '1'
         uses: actions/cache@v5
         with:
@@ -93,21 +108,31 @@ jobs:
       - name: Clear orphaned Conan download tempfiles (self-hosted cache is reused)
         if: steps.presence.outputs.exists == '1' && runner.environment != 'github-hosted'
         # An interrupted or racing conan download can leave a partial conan_package.tgz
-        # in ~/.conan2 that makes the next `conan install` abort with
+        # in $CONAN_HOME that makes the next `conan install` abort with
         # "the file to download already exists". Delete these staging archives so a
         # missing binary package can be re-fetched cleanly; extracted packages are untouched.
         run: |
-          find ~/.conan2/p -type f \( -name 'conan_package.tgz' -o -name 'conan_sources.tgz' \) -delete 2>/dev/null || true
+          find "$CONAN_HOME/p" -type f \( -name 'conan_package.tgz' -o -name 'conan_sources.tgz' \) -delete 2>/dev/null || true
 
       - name: Conan install
         if: steps.presence.outputs.exists == '1'
         run: |
-          conan install . \
-            -pr:a=ci/conan/linux-gcc13.profile \
-            --lockfile=conan.lock \
-            --build=missing \
-            --output-folder=build_${{ matrix.coin }} \
-            --settings=build_type=Release
+          # Bounded retry: remote package downloads are the only network step
+          # left here; a transient CDN/remote hiccup must not red a good PR.
+          for attempt in 1 2 3; do
+            if \
+            conan install . \
+              -pr:a=ci/conan/linux-gcc13.profile \
+              --lockfile=conan.lock \
+              --build=missing \
+              --output-folder=build_${{ matrix.coin }} \
+              --settings=build_type=Release
+            then exit 0; fi
+            echo "conan install failed (attempt $attempt/3); retrying in $((attempt * 15))s"
+            sleep $((attempt * 15))
+          done
+          echo "conan install failed after 3 attempts" >&2
+          exit 1
 
       - name: Configure
         if: steps.presence.outputs.exists == '1'
diff --git a/.github/workflows/dash-gate-g1-byte-parity.yml b/.github/workflows/dash-gate-g1-byte-parity.yml
@@ -32,7 +32,21 @@ jobs:
       - uses: actions/checkout@v6
 
       - name: Set per-job Conan home
-        run: echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+        run: |
+          # github-hosted: ephemeral per-job home, warmed by actions/cache below.
+          # self-hosted:   PERSISTENT home keyed on the runner instance. A runner
+          # executes one job at a time, so this home is never contended (no
+          # sqlite "failed to acquire database lock in 20s") while still keeping
+          # the per-job isolation of #704/#710 -- no two concurrent jobs share it.
+          # Being persistent, it also takes the GH cache-service restore off the
+          # hot path: that restore times out on VM905 ("Failed to restore: The
+          # operation cannot be completed in timeout") and its miss cascaded into
+          # a cold from-source Boost rebuild on all 8 runners at once.
+          if [ "${{ runner.environment }}" = "github-hosted" ]; then
+            echo "CONAN_HOME=$RUNNER_TEMP/conan2" >> "$GITHUB_ENV"
+          else
+            echo "CONAN_HOME=$HOME/.conan2-ci/$RUNNER_NAME" >> "$GITHUB_ENV"
+          fi
 
       - name: Check gate entrypoint presence
         id: presence
@@ -70,7 +84,8 @@ jobs:
         if: steps.presence.outputs.exists == '1'
         run: conan profile show -pr:a=ci/conan/linux-gcc13.profile
 
-      - name: Restore Conan cache
+      - name: Restore Conan cache (github-hosted only)
+        if: runner.environment == 'github-hosted'
         if: steps.presence.outputs.exists == '1'
         uses: actions/cache@v5
         with:
diff --git a/.github/workflows/dash-gate-g3a-regtest-block-production.yml b/.github/workflows/dash-gate-g3a-regtest-block-production.yml
diff --git a/.github/workflows/dgb-phase-b-smoke.yml b/.github/workflows/dgb-phase-b-smoke.yml
