Consider build system revisions

[brghena]

The current Make build system has become unwieldy.

Some particular reasons for this that I can list off the top of my head:

• Some of it is due to the need to check things about or within files. We use a bunch of tools, not just Tockloader and elf2tab, but also perl and test. These can add hidden dependencies and variation between dev environments (Linux vs MacOS).
• We have many flags and configurations for GCC and other tooling. Many of those can be set to defaults, but occasionally we have change those defaults. So variables have to be made available to configure these.
• The biggest issue is support for multiple architectures. Each architecture will have its own configurations and its own build rules. But the build rules share more than they differ, so we created a system to generate build rules for each architecture for a set of standard build rules. This is quite hard to follow (at one point we get four $ variable escapes deep!). This has been exacerbated by the need for hard-coded application addresses on RISC-V. Currently libtock-c builds for 17 targets, 4 ARM and 13 RISC-V (many of which differ only in memory addresses).

What do we do about this? It's not clear that there's a way to do better with Make. So, we've discussed ideas of changing to a new build system.

Some general things that I personally think we want in any build system:

• Standard tools as much as possible (or at least standard interfaces for users). Being able to "just type make" feels nice.
• Creating a new application that does normal things should be simple. (e.g., just copy-paste an existing build file)

So, this issue is for tracking the problem and suggestions on paths forward.

[brghena]

From @reynoldsbd (via slack):

---

I'd like to float the idea of using CMake for the usermode build system. No build system is without its quirks or learning curve, but I think CMake could bring some pretty substantial benefits:

• First class support for out-of-tree apps and libraries without any special cases or non-standard build logic
• Seamless compatibility with a broad ecosystem of CMake libraries, including many things available via vcpkg
• An explicit model for layering toolchain support and other build rules, allowing downstream projects like ours to drop in our own toolchain and libc from out of tree
• First-class Windows support
• Somewhat subjective, but the layering also help ease the cognitive complexity of the build rules in ways that are very difficult to achieve with Makefiles alone

The official docs are pretty good, and they have a healthy mix of terse reference and explanatory prose. https://cmake.org/cmake/help/latest/index.html

I've also heard good things about "Mastering CMake", although I've never read it cover to cover myself. https://cmake.org/cmake/help/book/mastering-cmake/

I was actually able to get an example working end-to-end last night 🙂 I just didn't want to spend too much time going in a particular direction before starting a conversation. Work-in-progress branch (which may change in the future): master...reynoldsbd:libtock-c:cmake

[ppannuto]

First off, this post is not at all to say no, just a bit of screaming into the void of "I wanted the cmake out-of-box experience to be better/easier", and some off-the-cuff impressions from trying cmake naïvely / quickly this afternoon:

---

cmake forces you to make more decisions to do anything than make :/

• i.e., I opened a terminal, I type cmake, and it doesn't "just work"

I did try cmake . after that, which also failed, but that might be a symptom of grabbing a random state of a WIP branch

Looking at the "User Interaction Guide" from the linked documentation, it seems like step 0 is make decisions (how?) about where build artifacts should go, and then remember to invoke all your cmake commands from that new directory; plus here's a random PREFIX variable for you to think about:

I'm not sure if there's a means with a 'more complete' cmake implementation to alleviate some of this, but the guiding light for tock and libtock-c especially has been "ease of first-use experience".

I would advocate strongly that we need to find some means of setting defaults such that we can get to an equivalent user experience of "checkout repo, install cross-compiler if you need, and type BUILD_CMD"

A smaller(?) thing on CMake "generators"?

I also noticed when running cmake --help that it looks like the default install of cmake on my machine wouldn't work, because I don't have the ninja backend available?

Generators

The following generators are available on this platform (* marks default):
* Unix Makefiles               = Generates standard UNIX makefiles.
  Ninja                        = Generates build.ninja files.
  Ninja Multi-Config           = Generates build-.ninja files.
  Watcom WMake                 = Generates Watcom WMake makefiles.
  Xcode                        = Generate Xcode project files.
  CodeBlocks - Ninja           = Generates CodeBlocks project files
                                 (deprecated).
  CodeBlocks - Unix Makefiles  = Generates CodeBlocks project files
                                 (deprecated).
  CodeLite - Ninja             = Generates CodeLite project files
                                 (deprecated).
  CodeLite - Unix Makefiles    = Generates CodeLite project files
                                 (deprecated).
  Eclipse CDT4 - Ninja         = Generates Eclipse CDT 4.0 project files
                                 (deprecated).
  Eclipse CDT4 - Unix Makefiles= Generates Eclipse CDT 4.0 project files
                                 (deprecated).
  Kate - Ninja                 = Generates Kate project files (deprecated).
  Kate - Ninja Multi-Config    = Generates Kate project files (deprecated).
  Kate - Unix Makefiles        = Generates Kate project files (deprecated).
  Sublime Text 2 - Ninja       = Generates Sublime Text 2 project files
                                 (deprecated).
  Sublime Text 2 - Unix Makefiles
                               = Generates Sublime Text 2 project files
                                 (deprecated).

It's another layer of explanation to first-time users to not only "install cmake" but install it with some suite of optional add-ons?

---

A more concerning thing in this screenshot is the sentence I highlighted — the libtock-c userspace build system works very hard to make sure that the default behavior is to build for all permutations of supported architectures. Would cmake preserve this, or is the expectation that you have have, build/cortexm0/ build/cortexm3/ (etc) folders and you run cmake separately in each of them?

Doing a "single-arch build" by default would be a pretty fundamental shift. The mutli-platform build is really where 90% of the complexity of our Makefile infrastructure comes from.

If we want to revisit the multi-arch build decision, then I suspect we could be happy with "vanilla" Makefile again, as all of the various complex build rules and $$$$ issues stem from generating all the required permutations (without heavy copy-pasting).

---

tl;dr—Do we need a new build system, or to revisit the goals of the out of the box build system?

[ppannuto]

Circling back to the top-level comments more substantially for a moment:

Some of it is due to the need to check things about or within files. We use a bunch of tools, not just Tockloader and elf2tab, but also perl and test. These can add hidden dependencies and variation between dev environments (Linux vs MacOS).

perl is there because it is cross-platform (and sed was not). I don't think this has anything to do with the build system choice; this is really a "multi-arch" problem, and the decision of how to manage linker scripts.

We have many flags and configurations for GCC and other tooling. Many of those can be set to defaults, but occasionally we have change those defaults. So variables have to be made available to configure these.

Wouldn't every build system have this? What does this have to do with make or others?

The biggest issue is support for multiple architectures. Each architecture will have its own configurations and its own build rules. But the build rules share more than they differ, so we created a system to generate build rules for each architecture for a set of standard build rules. This is quite hard to follow (at one point we get four $ variable escapes deep!). This has been exacerbated by the need for hard-coded application addresses on RISC-V. Currently libtock-c builds for 17 targets, 4 ARM and 13 RISC-V (many of which differ only in memory addresses).

This is really the discussion point I think, and it's the same realization I came to commenting above on playing with cmake for a second.

Currently, libtock-c chooses to build all permutations for all apps by default. Early on, we made this decision to highlight the cross-platform nature of Tock apps and Tock as a true operating system. I think it's helped to ensure that being multi-platform remains a first class concern of upstream userspace Tock.

It is, however, also where pretty much all of the complexity in our build system comes from.

We could switch to a model that explicitly builds one architecture at a time (with some top-level configuration of "which architecture [and where for RISC-V]"). That would eliminate most if not all of the complexity of the Makefiles, without requiring a switch to a "heavier-weight" build system.

Do we still value the multi-arch build?

[bradjc]

Do we still value the multi-arch build?

I do, quite a bit. Otherwise the builds which don't get used as much will just decay. Also, we lose the "just works" property, as users have to know exactly which board and configuration they need, which is much more obvious to us as the core developers than it is to a new user.

[bradjc]

Our build system is complicated because it does a lot. Perhaps there is another tool that could accomplish the same thing, but I think an alternative needs to aim to be feature complete, OR any removed features need to be decided on beforehand and not just accepted once there is a partial port.

• Builds apps for several architectures and arbitrary fixed addresses.
• Automatically fetches the correct version of newlib and libc++.
• Fetches external libraries as needed.
• Compiles external libraries (and our adapter code) with our per-arch build flags which are necessary.
• Automatically updates elf2tab if needed.
• Documents nearly every flag (included warnings).
• Builds our internal libraries (libtock and libtock-sync).
• Allows internal library builds without warnings, and permits warnings for external libraries.
• Enables apps to specify external libraries by setting a single environment variable.
• Per app makefiles are only a couple lines long, minimizing copying between apps, eg:

  TOCK_USERLAND_BASE_DIR = ../../../..
  C_SRCS := $(wildcard *.c)
  include $(TOCK_USERLAND_BASE_DIR)/AppMakefile.mk
  

• Does a great job of not re-building artifacts for multiple example apps.

In my opinion, having a complicated build system that is fine-tuned for exactly our needs and shifts the burden from library users to library developers is worthwhile tradeoff. And just like any code trying to do something complicated that does require having some functions with deep uses of $.

---

Another take on this is that the libtock-c repo itself is somewhat unwieldy, mostly stemming from the singular fact that we need very specific GCC flags. Overtime, new libraries have gotten added and things have grown. However, the core is really just libtock and libtock-sync. We could think about if packaging those as proper libraries and not having to have all Tock C development essentially live in this repo would be a helpful solution.

[brghena]

Some of it is due to the need to check things about or within files. We use a bunch of tools, not just Tockloader and elf2tab, but also perl and test. These can add hidden dependencies and variation between dev environments (Linux vs MacOS).

perl is there because it is cross-platform (and sed was not). I don't think this has anything to do with the build system choice; this is really a "multi-arch" problem, and the decision of how to manage linker scripts.

Well, a better build system would perhaps include tooling to allow interactions with files without relying external tools. Or you could argue that's worse and relying on standard tools is better. Regardless, my point was actually that it's a source of complication.

We have many flags and configurations for GCC and other tooling. Many of those can be set to defaults, but occasionally we have change those defaults. So variables have to be made available to configure these.

Wouldn't every build system have this? What does this have to do with make or others?

Similar: it's a source of complication with the build system as it currently exists. A system that makes that easier to handle would perhaps be beneficial to the problem.

The biggest issue is support for multiple architectures. Each architecture will have its own configurations and its own build rules. But the build rules share more than they differ, so we created a system to generate build rules for each architecture for a set of standard build rules. This is quite hard to follow (at one point we get four $ variable escapes deep!). This has been exacerbated by the need for hard-coded application addresses on RISC-V. Currently libtock-c builds for 17 targets, 4 ARM and 13 RISC-V (many of which differ only in memory addresses).

This is really the discussion point I think, and it's the same realization I came to commenting above on playing with cmake for a second.

Currently, libtock-c chooses to build all permutations for all apps by default. Early on, we made this decision to highlight the cross-platform nature of Tock apps and Tock as a true operating system. I think it's helped to ensure that being multi-platform remains a first class concern of upstream userspace Tock.

It is, however, also where pretty much all of the complexity in our build system comes from.

We could switch to a model that explicitly builds one architecture at a time (with some top-level configuration of "which architecture [and where for RISC-V]"). That would eliminate most if not all of the complexity of the Makefiles, without requiring a switch to a "heavier-weight" build system.

Do we still value the multi-arch build?

I would also say yes. And I would further say that we want a build system that better handles these permutations. We are certainly not the only project that builds for multiple architectures.

---

Generally, I think we're in agreement about these being challenges/complications of the current system. A build system that clearly made these things easier would very much interest me. A build system that addresses none of these feels unlikely to solve our problems, although there are likely other complications that I haven't listed here.