Skip to content

apollo_compile_to_casm,apollo_compile_to_native: switch to script-based compiler installation#13675

Open
avi-starkware wants to merge 1 commit into
avi/compiler-bootstrap-orderfrom
avi/compiler-install-switch
Open

apollo_compile_to_casm,apollo_compile_to_native: switch to script-based compiler installation#13675
avi-starkware wants to merge 1 commit into
avi/compiler-bootstrap-orderfrom
avi/compiler-install-switch

Conversation

@avi-starkware
Copy link
Copy Markdown
Collaborator

@avi-starkware avi-starkware commented Apr 6, 2026

Remove build.rs from both compiler crates — compiler binaries are now
installed by scripts/install_compiler_binaries.sh (called from
install_cargo_tools.sh) instead of as a cargo build side effect.

  • Delete build.rs and [build-dependencies] from both crates
  • Remove OUT_DIR/RUNTIME_ACCESSIBLE_OUT_DIR env var hack
  • Simplify paths.rs to PATH-based binary lookup
  • Remove old install_compiler_binary(), keep verify_compiler_binary()
  • Wire verify_compiler_binary into compiler constructors (skip for
    custom paths in SierraToNativeCompiler)
  • install_cargo_tools.sh calls install_compiler_binaries.sh
  • Dockerfiles use cargo install with versions from .txt files
  • Update workflow artifact path and BUILD file

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com


Note

Medium Risk
Touches build/packaging and runtime compiler-binary resolution across CI and multiple Docker images; misconfigured PATH or missing installs could break compilation at runtime or artifact publishing.

Overview
Switches Sierra compiler binary provisioning from build.rs side effects (writing into target/.../shared_executables via OUT_DIR) to an explicit scripts/install_compiler_binaries.sh flow that installs versioned starknet-sierra-compile/starknet-native-compile and exposes them via PATH, with runtime constructors now calling verify_compiler_binary to enforce required versions.

Updates Dockerfiles and CI/artifact upload to install/copy these binaries into /usr/local/bin (and to upload starknet-native-compile by resolved PATH), adjusts Bazel BUILD exports accordingly, and tweaks install_cargo_tools.sh version detection to handle prerelease versions before invoking the new installer.

Reviewed by Cursor Bugbot for commit c698fc8. Bugbot is set up for automated code reviews on this repo. Configure here.

Copy link
Copy Markdown
Collaborator Author

avi-starkware commented Apr 6, 2026

@reviewable-StarkWare
Copy link
Copy Markdown

This change is Reviewable

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 6, 2026

Artifacts upload workflows:

Comment thread crates/apollo_compilation_utils/src/build_utils.rs
Comment thread scripts/build_native_blockifier.sh Outdated
Copy link
Copy Markdown
Collaborator

@dorimedini-starkware dorimedini-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dorimedini-starkware reviewed 14 files and all commit messages, and made 7 comments.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on avi-starkware).


crates/blockifier_reexecution/replay/Dockerfile line 91 at r1 (raw file):

# =============================================================================
# Stage 4: Final runtime image

Suggestion:

COPY rust-toolchain.toml rust-toolchain.toml
COPY scripts/install_compiler_binaries.sh scripts/install_compiler_binaries.sh
# .. copy other files required... version.txt etc.
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release -p blockifier_reexecution --features cairo_native --recipe-path recipe.json

COPY . .
RUN cargo build --release -p blockifier_reexecution --features cairo_native

# Install compiler binaries used for Sierra compilation at runtime.
RUN scripts/install_compiler_binaries.sh

# =============================================================================
# Stage 4: Final runtime image

crates/blockifier_reexecution/replay/Dockerfile line 109 at r1 (raw file):

COPY --from=builder /app/target/release/blockifier_reexecution ./target/release/blockifier_reexecution
COPY --from=builder /var/tmp/rust/bin/starknet-sierra-compile /usr/local/bin/starknet-sierra-compile
COPY --from=builder /var/tmp/rust/bin/starknet-native-compile /usr/local/bin/starknet-native-compile

how do you know this is where the binaries were installed to...?
maybe make the install_compiler_binaries.sh script return the absolute paths to the installed packages? then you'd never have to guess

Code quote:

COPY --from=builder /var/tmp/rust/bin/starknet-sierra-compile /usr/local/bin/starknet-sierra-compile
COPY --from=builder /var/tmp/rust/bin/starknet-native-compile /usr/local/bin/starknet-native-compile

crates/starknet_transaction_prover/Dockerfile line 140 at r1 (raw file):

# Install compiler binary used for Sierra to CASM compilation at runtime.
# Version is read from a plain text file (the single source of truth).
RUN cargo install --locked starknet-sierra-compile --version "$(cat crates/apollo_infra_utils/src/cairo_compiler_version.txt)"

see above: you are duplicating logic from you installation script.
here specifically you only need the sierra compiler, but I'm sure you can add an optional argument to your bash script to skip native installation

Code quote:

# Install compiler binary used for Sierra to CASM compilation at runtime.
# Version is read from a plain text file (the single source of truth).
RUN cargo install --locked starknet-sierra-compile --version "$(cat crates/apollo_infra_utils/src/cairo_compiler_version.txt)"

crates/starknet_transaction_prover/Dockerfile line 165 at r1 (raw file):

# Copy starknet-sierra-compile binary required for Sierra to Casm compilation.
COPY --from=builder /var/tmp/rust/bin/starknet-sierra-compile /usr/local/bin/starknet-sierra-compile

ditto - consider taking the output path from the script output

Code quote:

COPY --from=builder /var/tmp/rust/bin/starknet-sierra-compile /usr/local/bin/starknet-sierra-compile

deployments/images/base/Dockerfile line 50 at r1 (raw file):

# Cleanup.
RUN rm -f scripts/install_build_tools.sh scripts/install_cargo_tools.sh scripts/install_compiler_binaries.sh scripts/compiler_versions.sh scripts/cargo_tool_utils.sh scripts/dependencies.sh scripts/apt_utils.sh scripts/requirements.txt && \
    rm -rf crates/apollo_infra_utils crates/apollo_compile_to_native

I get that you save ink using -rf but please keep on being explicit here :P

Code quote:

RUN rm -f scripts/install_build_tools.sh scripts/install_cargo_tools.sh scripts/install_compiler_binaries.sh scripts/compiler_versions.sh scripts/cargo_tool_utils.sh scripts/dependencies.sh scripts/apt_utils.sh scripts/requirements.txt && \
    rm -rf crates/apollo_infra_utils crates/apollo_compile_to_native

deployments/images/sequencer/Dockerfile line 0 at r1 (raw file):
same remarks as above (reuse script + get output paths as output)


scripts/build_native_blockifier.sh line 20 at r1 (raw file):

    # Install starknet-native-compile for artifact upload.
    source "$(dirname "${BASH_SOURCE[0]}")/compiler_versions.sh"
    cargo install --locked starknet-native-compile --version "$NATIVE_COMPILE_VERSION" || ret=$?

I prefer you pass an argument to the original script indicating if you want to install sierra, native or both, and use it..

Code quote:

    # Install starknet-native-compile for artifact upload.
    source "$(dirname "${BASH_SOURCE[0]}")/compiler_versions.sh"
    cargo install --locked starknet-native-compile --version "$NATIVE_COMPILE_VERSION" || ret=$?

@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 92e7bf5 to 9c686ef Compare April 6, 2026 13:55
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from 7c87ee5 to 5814201 Compare April 6, 2026 13:55
Comment thread BUILD Outdated
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 9c686ef to ac92fec Compare April 6, 2026 16:27
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from 5814201 to 0f636e6 Compare April 6, 2026 16:27
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from ac92fec to 36d086b Compare April 6, 2026 17:10
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from 0f636e6 to 1c8c980 Compare April 6, 2026 17:10
Comment thread scripts/build_native_blockifier.sh Outdated
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 36d086b to e332679 Compare April 6, 2026 20:26
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from 1c8c980 to 6b06cdf Compare April 6, 2026 20:26
Comment thread .github/workflows/upload_artifacts_workflow.yml Outdated
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from e332679 to 7cc069b Compare April 7, 2026 05:35
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from 6b06cdf to a529b9e Compare April 7, 2026 05:35
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 7cc069b to 1efda68 Compare April 7, 2026 06:35
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from a529b9e to c698fc8 Compare April 7, 2026 06:35
Copy link
Copy Markdown
Collaborator Author

@avi-starkware avi-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@avi-starkware made 7 comments and resolved 1 discussion.
Reviewable status: 8 of 16 files reviewed, 6 unresolved discussions (waiting on dorimedini-starkware).


crates/blockifier_reexecution/replay/Dockerfile line 109 at r1 (raw file):

Previously, dorimedini-starkware wrote…

how do you know this is where the binaries were installed to...?
maybe make the install_compiler_binaries.sh script return the absolute paths to the installed packages? then you'd never have to guess

Done. Added a --dest flag to install the binaries to a given directory.


crates/starknet_transaction_prover/Dockerfile line 140 at r1 (raw file):

Previously, dorimedini-starkware wrote…

see above: you are duplicating logic from you installation script.
here specifically you only need the sierra compiler, but I'm sure you can add an optional argument to your bash script to skip native installation

Done.


crates/starknet_transaction_prover/Dockerfile line 165 at r1 (raw file):

Previously, dorimedini-starkware wrote…

ditto - consider taking the output path from the script output

Done. (Same approach as above)


deployments/images/base/Dockerfile line 50 at r1 (raw file):

Previously, dorimedini-starkware wrote…

I get that you save ink using -rf but please keep on being explicit here :P

Done.


deployments/images/sequencer/Dockerfile line at r1 (raw file):

Previously, dorimedini-starkware wrote…

same remarks as above (reuse script + get output paths as output)

Done.


scripts/build_native_blockifier.sh line 20 at r1 (raw file):

Previously, dorimedini-starkware wrote…

I prefer you pass an argument to the original script indicating if you want to install sierra, native or both, and use it..

Done.


crates/blockifier_reexecution/replay/Dockerfile line 91 at r1 (raw file):

# =============================================================================
# Stage 4: Final runtime image

Done. All three Dockerfiles now call scripts/install_compiler_binaries.sh

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c698fc8. Configure here.

Comment thread .github/workflows/upload_artifacts_workflow.yml Outdated
Copy link
Copy Markdown
Collaborator

@dorimedini-starkware dorimedini-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dorimedini-starkware reviewed 8 files and all commit messages, made 2 comments, and resolved 6 discussions.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on avi-starkware).


deployments/images/base/Dockerfile line 50 at r5 (raw file):

# Cleanup.
RUN rm -f scripts/install_build_tools.sh scripts/install_cargo_tools.sh scripts/install_compiler_binaries.sh scripts/compiler_versions.sh scripts/cargo_tool_utils.sh scripts/dependencies.sh scripts/apt_utils.sh scripts/requirements.txt && \
    rm -f crates/apollo_infra_utils/src/cairo_compiler_version.txt crates/apollo_compile_to_native/src/native_compiler_version.txt && \

non-blocking

Suggestion:

RUN rm -f scripts/install_build_tools.sh scripts/install_cargo_tools.sh scripts/install_compiler_binaries.sh scripts/compiler_versions.sh scripts/cargo_tool_utils.sh scripts/dependencies.sh scripts/apt_utils.sh scripts/requirements.txt crates/apollo_infra_utils/src/cairo_compiler_version.txt crates/apollo_compile_to_native/src/native_compiler_version.txt && \

Comment thread .github/workflows/upload_artifacts_workflow.yml Outdated
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 8, 2026

There hasn't been any activity on this pull request recently, and in order to prioritize active work, it has been marked as stale.
This PR will be closed and locked in 7 days if no further activity occurs.
Thank you for your contributions!

@github-actions github-actions Bot added the stale label May 8, 2026
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 1efda68 to 4af9d22 Compare May 10, 2026 07:40
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from c698fc8 to fd0d72f Compare May 10, 2026 07:40
@cursor
Copy link
Copy Markdown

cursor Bot commented May 10, 2026

PR Summary

Medium Risk
Changes how Sierra compiler binaries are installed and located (build-time build.rs -> external scripts and $PATH), which can break builds or runtime compilation if images/CI environments don’t install the expected versions.

Overview
Removes build.rs-driven installation of starknet-sierra-compile/starknet-native-compile from apollo_compile_to_casm and apollo_compile_to_native, dropping the OUT_DIR/RUNTIME_ACCESSIBLE_OUT_DIR path hack and related build-dependencies.

Switches runtime resolution to a $PATH-based binary_path() (with legacy_binary_path() kept temporarily for old callers) and adds upfront verify_compiler_binary() checks in compiler constructors to enforce expected versions.

Updates build tooling and images to install/copy compiler binaries via scripts/install_compiler_binaries.sh (invoked from install_cargo_tools.sh and Dockerfiles), including per-version install roots and stable copy destinations (e.g. /usr/local/bin / target/.../shared_executables).

Reviewed by Cursor Bugbot for commit 7a58f78. Bugbot is set up for automated code reviews on this repo. Configure here.

@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 4af9d22 to 7e79823 Compare May 10, 2026 11:21
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from fd0d72f to 5ee741b Compare May 10, 2026 11:21
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 7e79823 to 5b12ec8 Compare May 14, 2026 11:37
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from 5ee741b to c0b3e12 Compare May 14, 2026 11:37
…ed compiler installation

Remove build.rs from both compiler crates — compiler binaries are now
installed by scripts/install_compiler_binaries.sh (called from
install_cargo_tools.sh) instead of as a cargo build side effect.

- Delete build.rs and [build-dependencies] from both crates
- Compiler constructors use verify_compiler_binary + PATH-based lookup
  (skip version check for custom paths in SierraToNativeCompiler)
- install_cargo_tools.sh calls install_compiler_binaries.sh
- Dockerfiles use cargo install with versions from .txt files
- Base Dockerfile copies new scripts and version files

Old install_compiler_binary() and legacy path functions are left for
removal in a follow-up cleanup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@avi-starkware avi-starkware force-pushed the avi/compiler-bootstrap-order branch from 5b12ec8 to eb8a28f Compare May 14, 2026 11:50
@avi-starkware avi-starkware force-pushed the avi/compiler-install-switch branch from c0b3e12 to 7a58f78 Compare May 14, 2026 11:50
Copy link
Copy Markdown
Collaborator Author

@avi-starkware avi-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@avi-starkware made 1 comment and resolved 1 discussion.
Reviewable status: 7 of 16 files reviewed, all discussions resolved (waiting on dorimedini-starkware).


deployments/images/base/Dockerfile line 50 at r5 (raw file):

Previously, dorimedini-starkware wrote…

non-blocking

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants