buck2: init at unstable-2023-05-17

[nickgerace]

Description of changes

This PR adds buck2 with the ability to auto update. A big thanks to @thoughtpolice for the inspiration on where to start as well as from the folks at @DeterminateSystems on advice on how to handle the difficult bits here.

"A large-scale build tool. The successor to Buck."
-- https://buck2.build

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Additional checks

• Ran nix-build -A buck2 to produce the artifacts tested
• Ran pkgs/development/tools/build-managers/buck2/update.sh manually with varying changes to default.nix upon each run to simulate update scenarios

Additional notes

• Fixes Package request: buck2 #226677
• Related Issue Building rustChannelOf({...}).rust-src from rust-overlay.nix errors. mozilla/nixpkgs-mozilla#304
• Related PR rust-overlay: restore SYSROOT detection for Darwin mozilla/nixpkgs-mozilla#309

Potential Issues

• Needed to pass --allow ifd for nixpkgs-review pr to be successful (EDIT: fixed)
• Did not add anything to the release notes as buck2 is not a service

[figsoda]

Using rust nightly from external sources is not allowed due to the dependency on import-from-derivation

Instead, you can try adding RUSTC_BOOTSTRAP = true;, that can sometimes get around the nightly requirement using stable rust.

Consider using nix-update-script for passthru.updateScript, it will also handle updating outputHashes.

The bugs in nixpkgs-mozilla might be fixed in fenix, so if you wan to package buck2 out of tree, this can be an option.

[nickgerace]

Thank you @figsoda for the notes! I have questions and will go one by one.

Using rust nightly from external sources is not allowed due to the dependency on import-from-derivation

Ah that explains why I needed --allow ifd for nixpkgs-review pr.

Instead, you can try adding RUSTC_BOOTSTRAP = true;, that can sometimes get around the nightly requirement using stable rust.

I'll try it, but am a bit concerned since upstream buck2 authors use a specific nightly toolchain pinned to the date. Is there an alternative to that, or is it impossible?

Consider using nix-update-script for passthru.updateScript, it will also handle updating outputHashes.

Noted! That's great! Since there are no docs for it, is there a method you recommend for how to use and configure it?

The bugs in nixpkgs-mozilla might be fixed in fenix, so if you wan to package buck2 out of tree, this can be an option.The bugs in nixpkgs-mozilla might be fixed in fenix, so if you wan to package buck2 out of tree, this can be an option.

I'm a bit confused. I should only pursue using fenix if I want to package buck2 outside of tree? Does "tree" refer to nixpkgs? If so, I already have a flake that builds buck2 and could just use cacheix, but I'd like to specifically include it in nixpkgs, if possible.

[figsoda]

but am a bit concerned since upstream buck2 authors use a specific nightly toolchain pinned to the date

This is probably because there can potentially be breaking changes between nightly versions, not that it only works with one specific date. Though, do note that RUSTC_BOOTSTRAP is very hacky and only works sometimes.

Is there an alternative to that, or is it impossible?

The only alternative is to patch buck2 to be compatible with stable rust, which is probably not the easiest job.

is there a method you recommend for how to use and configure it?

You can add this right above meta

passthru.updateScript = nix-update-script { };

I should only pursue using fenix if I want to package buck2 outside of tree? Does "tree" refer to nixpkgs?

Yes & Yes. Fenix is an alternative to mozilla's rust overlay and will also require IFD (which isn't allowed) to be used in nixpkgs.

[thoughtpolice]

The use of RUSTC_BOOTSTRAP has always made me uneasy frankly. I don't like using it. Also, buck2 has surfaced some ICEs in various nightly versions, so the version chosen in the rust-toolchain file is not for no reason, I think. That's not a hard rejection out of hand, but we should also open an issue upstream about the possibility of targeting a stable compiler release first. If things start breaking because it's fragile or a new stable release introduces some change that causes a failure, that would be unfortunate.

What might be better in the very short term is to instead just contribute a package to the upstream flake.nix file. Right now, it only provides a devShell so you can compile a buck2 binary. But it should also provide a buck2 package itself. Then, users of flakes can simply add it as a dependency for their own projects.

Eventually we'll have to bite the bullet on this package though, because if something else that we want to package starts requiring buck2 itself, there's no way around it.

[thoughtpolice]

As of right now, I've found that the core buck2 codebase also rejects any non-sanctioned compiler, during compile time; see the following build.rs file: https://github.com/facebook/buck2/blob/c038ed271cca720bee43374e0c3883c59a625728/app/buck2_core/build.rs#L14-L55

I'm afraid going against the grain here by patching this out seems to clearly go against the explicit wishes of the dev team, and I don't think that's a place we want to be; so for now I'm going to tag this PR as being on hold and requiring changes.

I'll look into adding a package to the upstream flake instead as a short term mitigation for flake users, at least. I have also opened an issue to track using the stable rust compiler, which will alleviate these issues upstream; see facebook/buck2#265

[nickgerace]

Thank you @figsoda. I appreciate the replies and agree @thoughtpolice that using the toolchain that buck2 authors require is a requirement. I'll look into what you've shared.

Speaking of which, hello @thoughtpolice! Thanks again for the work you've done with buck2-nix. I don't want to stray too far off topic because this is nixpkgs PR, so I may follow up in your repository or the facebook/buck2 repository as needed.

Agreed on the use of RUSTC_BOOTSTRAP and (I'll reiterate) the requirement of the toolchain. I think shifting to contributing to the upstream flake is a good idea. The nightly hack is not only undesired, but also untenable. I think we are all aligned here.

I'll follow your tracking issue and can also help with the flake too. Thanks for posting that!

As an aside, reindeer is built on stable Rust, so I will likely post a similar PR to this one here. I believe we will not run into the same issue there.

As a result of all of this discussion, I am going to close this PR. I think a draft would drift quickly due to both buck2 and nix/nixpkgs itself moving quickly, and since closed PRs are indexable, we should be good should/once the opportunity arises to add buck2. Thank you both!