Make cachix builds from CI/CD match CLI/flake inputs - Part 2, considering kevm-pyk as well#2745
Merged
automergerpr-permission-manager[bot] merged 2 commits intoApr 22, 2025
Conversation
juliankuners
force-pushed
the
cachix-build-matching-kevm-pyk
branch
from
31ae7fe to
3880e26
Compare
anvacaru
previously approved these changes
Apr 22, 2025
anvacaru
reviewed
Apr 22, 2025
juliankuners
force-pushed
the
cachix-build-matching-kevm-pyk
branch
2 times, most recently
from
e1496a2 to
0f48e7e
Compare
anvacaru
approved these changes
Apr 22, 2025
automergerpr-permission-manager
Bot
deleted the
cachix-build-matching-kevm-pyk
branch
Contributor
Author
|
I can confirm now that evm-semantics is successfully being downloaded from the cache! Also replicable with kup. |
automergerpr-permission-manager Bot
pushed a commit
to runtimeverification/haskell-backend
that referenced
this pull request
Apr 24, 2025
Following up on #4096, [evm-semantics#2739](runtimeverification/evm-semantics#2739), [evm-semantics#2745](runtimeverification/evm-semantics#2745), and [kontrol#1013](runtimeverification/kontrol#1013), I measured the install time of `kontrol` with `kup` on a fresh Ubuntu virtual machine. Compared to a previous measurement, the install time has been reduced to 10 minutes from previously 20 minutes. During the latest measurement I noticed that Haskell dependencies are still being downloaded, even though the import-from-derivation anti-pattern has already been removed in #4096. I investigated and noticed that the Haskell dependencies are also considered runtime dependencies. This is caused by the `withZ3` wrapper that links to binaries in Haskell derivations. These Haskell derivations also contain references to Haskell dependencies, namely `kore` and `hs-backend-booster`. This pull request changes the `withZ3` wrapper to copy the respective binary prior to wrapping. E.g., for `kore-rpc-booster`, this reduces the total size to download by more than 5GB, when installing from nix cache. In addition, these downloads were spread among many smaller derivations to download. <details><summary>Output of `nix-tree` before the change: </summary> <p> <pre> ┌─────────────────────────────────────────────────────┬────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐ │kore-rpc-booster 5.15 GiB (5.15 GiB)│hs-backend-booster-0.1.0 5.12 GiB (5.08 GiB)│kore-0.1.0 4.89 GiB (1.31 GiB)│ │ │z3-4.13.4 72.03 MiB (32.24 MiB)│kore-rpc-types-0.1.0 3.34 GiB (80.65 MiB)│ │ │bash-5.2p26 32.51 MiB (1.54 MiB)│json-rpc-1.0.4 3.21 GiB (15.82 MiB)│ │ │ │aeson-pretty-0.8.10 3.04 GiB (21.2 MiB)│ │ │ │deriving-aeson-0.2.9 3.02 GiB (605.28 KiB)│ │ │ │aeson-2.1.2.1 3.02 GiB (52.64 MiB)│ │ │ │monad-logger-0.3.40 2.98 GiB (4.6 MiB)│ │ │ │stm-conduit-4.0.1 2.97 GiB (2.49 MiB)│ │ │ │conduit-extra-1.3.6 2.95 GiB (5.0 MiB)│ │ │ │cereal-conduit-0.8.0 2.88 GiB (441.2 KiB)│ │ │ │conduit-1.3.5 2.88 GiB (13.4 MiB)│ │ │ │semialign-1.3 2.86 GiB (3.73 MiB)│ │ │ │mono-traversable-1.0.17.0 2.85 GiB (22.06 MiB)│ │ │ │recursion-schemes-5.2.2.5 2.82 GiB (6.79 MiB)│ │ │ │cryptonite-0.30 2.81 GiB (54.26 MiB)│ │ │ │free-5.2 2.81 GiB (22.79 MiB)│ │ │ │vector-algorithms-0.9.0.1 2.81 GiB (25.24 MiB)│ │ │ │witherable-0.4.2 2.8 GiB (4.6 MiB)│ │ │ │indexed-traversable-instances- 2.79 GiB (358.65 KiB)│ │ │ │bitvec-1.1.5.0 2.78 GiB (10.49 MiB)│ │ │ │vector-0.13.1.0 2.77 GiB (49.75 MiB)│ │ │ │semigroupoids-6.0.1 2.77 GiB (12.16 MiB)│ │ │ │language-c-0.9.3 2.76 GiB (63.64 MiB)│ │ │ │profunctors-5.6.2 2.75 GiB (10.68 MiB)│ │ │ │memory-0.18.0 2.74 GiB (5.69 MiB)│ │ │ │basement-0.0.16 2.74 GiB (28.09 MiB)│ │ │ │bifunctors-5.6.2 2.73 GiB (16.04 MiB)│ │ │ │optparse-applicative-0.18.1.0 2.72 GiB (8.72 MiB)│ │ │ │streaming-commons-0.2.2.6 2.72 GiB (4.91 MiB)│ │ │ │attoparsec-0.14.4 2.72 GiB (17.16 MiB)│ │ │ │QuickCheck-2.14.3 2.72 GiB (19.19 MiB)│ │ │ │network-run-0.2.8 2.72 GiB (1.29 MiB)│ │ │ │hpp-0.6.5 2.71 GiB (13.35 MiB)│ └─────────────────────────────────────────────────────┴────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘ /nix/store/y51pfqn8fcmnimhkswfc1fda2pan3kmj-hs-backend-booster-0.1.0 NAR Size: 108.81 MiB | Closure Size: 5.12 GiB | Added Size: 5.08 GiB Immediate Parents (1): kore-rpc-booster </pre> </p> </details> <details><summary>Output of `nix-tree` after the change:</summary> <p> <pre> ┌─────────────────────────────────────────────────────┬────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐ │kore-rpc-booster 113.87 MiB (113.87 MiB)│z3-4.13.4 72.03 MiB (32.24 MiB)│gcc-13.2.0-lib 39.78 MiB (8.81 MiB)│ │ │elfutils-0.191 58.27 MiB (16.82 MiB)│glibc-2.39-52 30.96 MiB (28.88 MiB)│ │ │gmp-with-cxx-6.3.0 40.51 MiB (741.04 KiB)│ │ │ │ncurses-6.4 34.51 MiB (3.54 MiB)│ │ │ │bash-5.2p26 32.51 MiB (1.54 MiB)│ │ │ │secp256k1-0.4.1 32.22 MiB (1.25 MiB)│ │ │ │zlib-1.3.1 31.09 MiB (125.16 KiB)│ │ │ │libffi-3.4.6 31.03 MiB (71.87 KiB)│ │ │ │glibc-2.39-52 30.96 MiB (28.88 MiB)│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────┴────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘ /nix/store/ih51sgk8g57fnkbd5r82ddi8k5vln8cl-z3-4.13.4 NAR Size: 32.24 MiB | Closure Size: 72.03 MiB | Added Size: 32.24 MiB Immediate Parents (1): kore-rpc-booster </pre> </p> </details>
automergerpr-permission-manager Bot
pushed a commit
to runtimeverification/k
that referenced
this pull request
Apr 30, 2025
This pull request fixes the same kind of issue that I already observed and fixed in `evm-semantics` and `kontrol`, see [evm-semantics#2739](runtimeverification/evm-semantics#2739), [evm-semantics#2745](runtimeverification/evm-semantics#2745), and [kontrol#1013](runtimeverification/kontrol#1013). The CI/CD pipeline for k releases builds and publishes kontrol to the [nix build cache k-framework-binary](https://app.cachix.org/cache/k-framework-binary). In the respective GitHub action job, the nix derivation is built by cloning the repository and building the derivation locally. When building the nix project locally with a cloned git repository, nix does not include and consider empty submodule directories included by git for building the project. On the other hand, when the repository is referenced in a flake input or built by specifying a github URL in a nix CLI command, the empty submodule directories are included. This causes the package requested by kup, nix CLI, or nix flake inputs to not match the package built by CI/CD that is pushed to the build cache. As a consequence, k is built on the local machine instead of fetched from the binary cache. Though typically the requested version/hash is still available in the cache, as it is pushed to the cache by cache pushing jobs of other runtime verification repositories. This pull request fixes this issue such that the same version/hash of the derivation is always built for the cache and requested by flake inputs/CLI/kup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a follow-up pull request for Make cachix builds from CI/CD match CLI/flake inputs. Even after the pull request, I observed the same behaviour, where CI/CD builds and caches a build that does not match the hash requested by nix using the CLI.
During investigation, I found out that another empty git submodule directory causes this match, namely
/kevm-pyk/src/kevm_pyk/kproj/pluginin thekevm-pykderivation. This left me wondering as to why I didn't catch this, when building and matching the hash prior to publishing the last pull request in evm-semantics. Apparently, I mistakenly observed that the git submodule disappears, when built by nix CLI or as a nix flake input. In reality, the git submodule disappears on CI/CD. This is why my previous test built two matching hashes, while the hash still mismatched on CI/CD.This pull request fixes the remaining git submodule cachix issues. I specifically took a look at all places, where sources are parsed in nix. Though, I cannot reliably test whether versions will actually match using local builds instead of building using the respective CI/CD job.