Make cachix builds from CI/CD match CLI/flake inputs#2739
Merged
automergerpr-permission-manager[bot] merged 3 commits intoApr 10, 2025
Conversation
anvacaru
approved these changes
Apr 10, 2025
This was referenced Apr 16, 2025
automergerpr-permission-manager Bot
pushed a commit
to runtimeverification/haskell-backend
that referenced
this pull request
Apr 24, 2025
Following up on #4096, [evm-semantics#2739](runtimeverification/evm-semantics#2739), [evm-semantics#2745](runtimeverification/evm-semantics#2745), and [kontrol#1013](runtimeverification/kontrol#1013), I measured the install time of `kontrol` with `kup` on a fresh Ubuntu virtual machine. Compared to a previous measurement, the install time has been reduced to 10 minutes from previously 20 minutes. During the latest measurement I noticed that Haskell dependencies are still being downloaded, even though the import-from-derivation anti-pattern has already been removed in #4096. I investigated and noticed that the Haskell dependencies are also considered runtime dependencies. This is caused by the `withZ3` wrapper that links to binaries in Haskell derivations. These Haskell derivations also contain references to Haskell dependencies, namely `kore` and `hs-backend-booster`. This pull request changes the `withZ3` wrapper to copy the respective binary prior to wrapping. E.g., for `kore-rpc-booster`, this reduces the total size to download by more than 5GB, when installing from nix cache. In addition, these downloads were spread among many smaller derivations to download. <details><summary>Output of `nix-tree` before the change: </summary> <p> <pre> ┌─────────────────────────────────────────────────────┬────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐ │kore-rpc-booster 5.15 GiB (5.15 GiB)│hs-backend-booster-0.1.0 5.12 GiB (5.08 GiB)│kore-0.1.0 4.89 GiB (1.31 GiB)│ │ │z3-4.13.4 72.03 MiB (32.24 MiB)│kore-rpc-types-0.1.0 3.34 GiB (80.65 MiB)│ │ │bash-5.2p26 32.51 MiB (1.54 MiB)│json-rpc-1.0.4 3.21 GiB (15.82 MiB)│ │ │ │aeson-pretty-0.8.10 3.04 GiB (21.2 MiB)│ │ │ │deriving-aeson-0.2.9 3.02 GiB (605.28 KiB)│ │ │ │aeson-2.1.2.1 3.02 GiB (52.64 MiB)│ │ │ │monad-logger-0.3.40 2.98 GiB (4.6 MiB)│ │ │ │stm-conduit-4.0.1 2.97 GiB (2.49 MiB)│ │ │ │conduit-extra-1.3.6 2.95 GiB (5.0 MiB)│ │ │ │cereal-conduit-0.8.0 2.88 GiB (441.2 KiB)│ │ │ │conduit-1.3.5 2.88 GiB (13.4 MiB)│ │ │ │semialign-1.3 2.86 GiB (3.73 MiB)│ │ │ │mono-traversable-1.0.17.0 2.85 GiB (22.06 MiB)│ │ │ │recursion-schemes-5.2.2.5 2.82 GiB (6.79 MiB)│ │ │ │cryptonite-0.30 2.81 GiB (54.26 MiB)│ │ │ │free-5.2 2.81 GiB (22.79 MiB)│ │ │ │vector-algorithms-0.9.0.1 2.81 GiB (25.24 MiB)│ │ │ │witherable-0.4.2 2.8 GiB (4.6 MiB)│ │ │ │indexed-traversable-instances- 2.79 GiB (358.65 KiB)│ │ │ │bitvec-1.1.5.0 2.78 GiB (10.49 MiB)│ │ │ │vector-0.13.1.0 2.77 GiB (49.75 MiB)│ │ │ │semigroupoids-6.0.1 2.77 GiB (12.16 MiB)│ │ │ │language-c-0.9.3 2.76 GiB (63.64 MiB)│ │ │ │profunctors-5.6.2 2.75 GiB (10.68 MiB)│ │ │ │memory-0.18.0 2.74 GiB (5.69 MiB)│ │ │ │basement-0.0.16 2.74 GiB (28.09 MiB)│ │ │ │bifunctors-5.6.2 2.73 GiB (16.04 MiB)│ │ │ │optparse-applicative-0.18.1.0 2.72 GiB (8.72 MiB)│ │ │ │streaming-commons-0.2.2.6 2.72 GiB (4.91 MiB)│ │ │ │attoparsec-0.14.4 2.72 GiB (17.16 MiB)│ │ │ │QuickCheck-2.14.3 2.72 GiB (19.19 MiB)│ │ │ │network-run-0.2.8 2.72 GiB (1.29 MiB)│ │ │ │hpp-0.6.5 2.71 GiB (13.35 MiB)│ └─────────────────────────────────────────────────────┴────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘ /nix/store/y51pfqn8fcmnimhkswfc1fda2pan3kmj-hs-backend-booster-0.1.0 NAR Size: 108.81 MiB | Closure Size: 5.12 GiB | Added Size: 5.08 GiB Immediate Parents (1): kore-rpc-booster </pre> </p> </details> <details><summary>Output of `nix-tree` after the change:</summary> <p> <pre> ┌─────────────────────────────────────────────────────┬────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐ │kore-rpc-booster 113.87 MiB (113.87 MiB)│z3-4.13.4 72.03 MiB (32.24 MiB)│gcc-13.2.0-lib 39.78 MiB (8.81 MiB)│ │ │elfutils-0.191 58.27 MiB (16.82 MiB)│glibc-2.39-52 30.96 MiB (28.88 MiB)│ │ │gmp-with-cxx-6.3.0 40.51 MiB (741.04 KiB)│ │ │ │ncurses-6.4 34.51 MiB (3.54 MiB)│ │ │ │bash-5.2p26 32.51 MiB (1.54 MiB)│ │ │ │secp256k1-0.4.1 32.22 MiB (1.25 MiB)│ │ │ │zlib-1.3.1 31.09 MiB (125.16 KiB)│ │ │ │libffi-3.4.6 31.03 MiB (71.87 KiB)│ │ │ │glibc-2.39-52 30.96 MiB (28.88 MiB)│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────┴────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘ /nix/store/ih51sgk8g57fnkbd5r82ddi8k5vln8cl-z3-4.13.4 NAR Size: 32.24 MiB | Closure Size: 72.03 MiB | Added Size: 32.24 MiB Immediate Parents (1): kore-rpc-booster </pre> </p> </details>
automergerpr-permission-manager Bot
pushed a commit
to runtimeverification/k
that referenced
this pull request
Apr 30, 2025
This pull request fixes the same kind of issue that I already observed and fixed in `evm-semantics` and `kontrol`, see [evm-semantics#2739](runtimeverification/evm-semantics#2739), [evm-semantics#2745](runtimeverification/evm-semantics#2745), and [kontrol#1013](runtimeverification/kontrol#1013). The CI/CD pipeline for k releases builds and publishes kontrol to the [nix build cache k-framework-binary](https://app.cachix.org/cache/k-framework-binary). In the respective GitHub action job, the nix derivation is built by cloning the repository and building the derivation locally. When building the nix project locally with a cloned git repository, nix does not include and consider empty submodule directories included by git for building the project. On the other hand, when the repository is referenced in a flake input or built by specifying a github URL in a nix CLI command, the empty submodule directories are included. This causes the package requested by kup, nix CLI, or nix flake inputs to not match the package built by CI/CD that is pushed to the build cache. As a consequence, k is built on the local machine instead of fetched from the binary cache. Though typically the requested version/hash is still available in the cache, as it is pushed to the cache by cache pushing jobs of other runtime verification repositories. This pull request fixes this issue such that the same version/hash of the derivation is always built for the cache and requested by flake inputs/CLI/kup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CI/CD pipeline for
evm-semanticsreleases builds and publishes emv-semantics to the nix build cache k-framework. In the respective GitHub action job, the nix derivation is built by cloning the repository and building the derivation locally. Using git to clone causes empty submodule directories to be included in the source considered for the build. On the other hand, when the repository is referenced in a flake input or built by specifying a github URL in a nix CLI command, the empty submodule directories are omitted. This causes the package requested by CLI or nix flake inputs to not match the package built by CI/CD that is pushed to the build cache. As the respective kontrol build cache jobs were failing for some while, see here, installing kontrol required to build kontrol. As kontrol depends on evm-semantics, an evm-semantics build had to be downloaded. But due to the previously described mismatch, the build available in the build cache was ignored and instead emv-semantics was also built by source.This pull request fixes this mismatch by specifically not including the empty submodule directories in the final build.