[CFP] LLVM toolchains in Nixpkgs

[Ericson2314]

Title

LLVM toolchains in Nixpkgs

Author

John Ericson, John.Ericson@Obsidian.Systems, Obsidian Systems

Distribution

Nixpkgs is best known the package set behind the NixOS Linux distribution, but also runs on macOS, and arbitrary cross compilation. LLVM is used on macOS by default, and may be used for any platform; recently we've added a pkgsLLVM to encourage LLVM use in more contexts.

Abstract (optional)

About 5 years ago, we in the Nix community set out to be one of the few distros/package sets to support whole-system cross compilation, meaning not just providing cross toolchains, but also building arbitrary packages. Overhauling our infrastructure to do this, we found that the packaging of compilers was the keystone upon which many of our out decisions rested. To cut down complexity, we decided to strive for a few related polices:

• Make cross compilation the default, native compilation should "just" be cross compilation such that "build = host".
• Packages shouldn't manage their own bootstrapping, the distro should.
• Compilers and run-times should be built separately, so every package installs artifacts just for one platform
• Runtime libraries should be "unspecial" --- like any other library --- at least one building.

LLVM has easily been the compiler most amendable to re-package according to these principles. We want to highlight what has gone well, the current state of the LLVM packaging in Nixpkgs, and what we are trying to get upstreamed into LLVM next.

What's unique about the environment you package LLVM for?

Very open-world goals --- Nixpkgs aims to be the "universal" package set, running or cross compiling everywhere possible. All the other cross compilation targets being discussed --- we're listening and want to support your platform too, no matter how exotic!

What makes your distribution of LLVM unique? (optional)

Very fine-grained, with all runtime libraries and tools being built separately, and installed in separate prefixes.

What might others learn from your experience? (optional)

Techniques to manage many configurations / bootstraps without drowning in complexity.

(obligatory plug :)) Of course, you may find like we do that Nixpkgs a great foundation with which to develop toolchains and push them to their limits, and want to just use Nixpkgs rather than borrow our techniques.

What could be improved in upstream LLVM to make working with it easier as a downstream packager? (optional)

• Better support for "GNU install dirs" --- how we separate regular and "dev" packages instead of DESTDIR or manual separation.
• Perhaps giving the -B/-resource-dir division of labor a fresh review.
• Clarify compiler-rt <-> libc dependencies.
• Use of "target" in source code interfaces D44753

---

@sternenseemann just let me know about this conference, which I am quite grateful for as I've long hoped for a venue like this to discuss our work, and may have more information to provide about pkgsLLVM. I imagine our experience chasing after stray build failures in arbitrary packages using LLVM might also be of interest. If I had figured everything out earlier, I would have written the above knowing the answer of whether he had information to add, but as the deadlines looms I figured I better get something up ASAP. Sorry!

[Ericson2314]

@mstorsjo I saw in #14 (comment) you asked about macOS toolchains.

We do have reproducible macOS ones for x86_64, and, just recently, also Aarch64, and also iOS (Aarch64). A few years back I was asking you about your work since we have GCC MinGW, but would like to have LLVM MinGW and LLVM with official MSVC runtimes as cross compilation toolchains too. I do hope to someday finish the PR I started based on your work (NixOS/nixpkgs#72366).

I figured it was better to point that out here than derail that issue.

[mstorsjo]

@mstorsjo I saw in #14 (comment) you asked about macOS toolchains.

We do have reproducible macOS ones for x86_64, and, just recently, also Aarch64, and also iOS (Aarch64).

That's nice! Witout trying it out for myself - where do you get the macOS/iOS SDKs for it? My understanding has been that the SDKs aren't freely redistributable (but I haven't studied the licenses of them in detail either).

A few years back I was asking you about your work since we have GCC MinGW, but would like to have LLVM MinGW and LLVM with official MSVC runtimes as cross compilation toolchains too. I do hope to someday finish the PR I started based on your work (NixOS/nixpkgs#72366).

Btw I don't remember, was it from your hint where I learnt where/how to download MSVC packages without actually running the installer on Windows? I refined and generalized that a fair bit in https://github.com/mstorsjo/msvc-wine/blob/master/vsdownload.py into a pretty generic downloader/installer.

[Ericson2314]

That's nice! Witout trying it out for myself - where do you get the macOS/iOS SDKs for it? My understanding has been that the SDKs aren't freely redistributable (but I haven't studied the licenses of them in detail either).

Historically for macOS, we only used Apple's open source releases, not he SDK. For iOS we do use xcode, and the user is instructed to just do the download part manually https://github.com/NixOS/nixpkgs/blob/3bd130c09f844ddd7942ea177bc4f5af7566d244/pkgs/os-specific/darwin/xcode/default.nix#L23-L32, with everyone else proceeding automatically. I am not sure what is done for the newer macOS stuff that might use some SDK pieces or similar.

Btw I don't remember, was it from your hint where I learnt where/how to download MSVC packages without actually running the installer on Windows? I

Yes :)

I refined and generalized that a fair bit in https://github.com/mstorsjo/msvc-wine/blob/master/vsdownload.py into a pretty generic downloader/installer.

Awesome! I will take a look.

[nickdesaulniers]

Thanks for taking the time to write up a CFP; we'd be overjoyed to have you present at LLVM Distributors Conf 2021! If you still plan on presenting, this is a reminder to get started on your slides for next week. Once they're done, we will contact you about submitting a PDF of your slides as either a pull request to this repository or via email to the organizer. We hope to have a schedule finalized by EOW; we may iterate on the schedule based on whether presenters have conflicts. Please keep this issue open for attendees to ask questions, or close this issue if you no longer plan on attending. Reminder to keep your talk concise (15 minutes); we wont be doing time for questions in order to fit as much content as possible. Attendees should ask questions here in this github issue.

This is a very interesting topic, and I'm sure it will be eye opening for a lot of people. (It was runner up for keynote)

[tstellar]

@Ericson2314 Do you have more information about the compiler-rt ->libc dependency. I was researching why compiler-rt uses the -fno-stack-protector flag and I came across this commit which implies that at some point there was no libc dependency: https://github.com/llvm/llvm-project/search?q=5829a98c48cf9009e4887e89e3075b3058909e8e&type=commits

[mstorsjo]

On that cyclic dependency - you pretty much need to build compiler-rt/lib/builtins first (which only builds static libraries and should have few to no dependencies on anything else), then your libc on top of that, then you can build the whole compiler-rt including sanitizers (possibly after building libc++).

[Ericson2314]

@tstellar I don't --- I am very much in the dark myself. I see https://reviews.llvm.org/D755 and the adjacent ones are somewhat related too. Not sure what happened to that stuff. There is still a RTSanitizerCommonNoLibc in tree, maybe git log -G RTSanitizerCommonNoLibc compiler-rt might help shed some more light.

In any event, I am not opposed to all cyclic dependencies --- they may be impossible to avoid, I just want the situation to be better planned and not just emergent or a historical accident.