Using and editing `vcbuild.bat` is not as intuitive as `Makefile`

[avivkeller]

After recently switching from a Linux environment to a Windows machine, I’ve found working with vcbuild.bat in this repository to be significantly more difficult than using the Makefile.

The vcbuild.bat script is harder to read and modify, making even simple changes tedious and prone to errors. In contrast, the Makefile is more maintainable, supports parallel builds, and includes conditional logic.

Modernizing vcbuild.bat, or making the Makefile cross-platform, could greatly improve the development workflow on Windows.

---

I've also heard good things from Cross-Platform tools like CMake, but that's a huge undertaking within itself.

[joyeecheung]

It seems to me what would be the path forward is probably, instead of doing anything about vcbuild.bat, try to make our Makefile work on Windows first, and then we can start ditching vcbuild.bat?

[avivkeller]

Sounds good, however, Makefile's aren't easy (at least in my experience) on Windows. All the bash syntax will fail, even the # comments.

---

I'm going to see if I can get a basic CMakeLists.txt setup, but don't wait on me to improve.

[joyeecheung]

There were some past discussions about switching to CMake, although in that context people were discussing about not just replacing the Makefile/vcbuild.bat (which are basically glorified scripts that drive the build system), but replacing the entire build system nodejs/TSC#648 nodejs/TSC#901

[anonrig]

Bazel is a strong candidate with really good caching and parallelization solutions that can help us ditch ccache and ninja.

[avivkeller]

That also sounds nice! Whatever is chosen, I'd be more than happy to dedicate my time to this :-).

Right now, I've made a basic CMakeLists.txt file that can start generating the node exe (but nothing else yet)

To be honest, I'm partial to (at least for now) starting with CMake because it's more widely adopted, and therefore may be simpler for new contributors to grasp, coming from our Make environment

[bnoordhuis]

I have https://github.com/bnoordhuis/v8-cmake that takes a brute force approach to building V8 with cmake. It's definitely not the best way to do it (static library build works but you can't build it as a shared library, for example) but at least it works. Perhaps it can be a helpful starting point.

I remember from my attempt to port node to cmake that ICU and openssl are huge obstacles.

[avivkeller]

(The below is my opinion, so if anything sounds like a statement, it's not.)

After doing some research, I’ve realized that using a build system like Makefile, CMakeLists.txt, or the vcbuild.bat file is more complicated than necessary for our needs.

Currently, our build system is based on GYP. When we run make, it essentially triggers make out/Makefile (or a similar command depending on the platform/compiler). This out/Makefile is generated by GYP, meaning the only real logic in our Makefile is invoking GYP and GYP generated files.

The rest of the Makefile is essentially a collection of script commands, often Python wrappers. For instance, the jstest target just calls tools/test.py with additional arguments.

In contrast, Bun (https://github.com/oven-sh/bun/blob/main/package.json) handles this more efficiently. It keeps the build logic in CMakeLists.txt, while defining all script commands in package.json as part of the Node.js ecosystem.

I believe we should consider a similar approach. Since much of our scripting is already in Python, why maintain a shell-heavy Makefile that ultimately wraps Python? Instead, we could manage the entire process with Python.

For the build step itself, we’d still use Python to invoke GYP and the required platform-specific compilers. But rather than chaining shell scripts, we could opt for a more maintainable, cross-platform solution.

What do you think about a Python Makefile?

I image it like a file like:

@task(
        dependencies=[DEPENDENCIES],
        outputs=[OUTPUTS],
        description=[DESCRIPTION]
)
def TARGET_NAME(CLI_ARGUMENTS):
    pass

[anonrig]

We can simply move to just file and our transition will be a lot easier

[avivkeller]

Oh! I've never heard of that, but that also looks very good for an easy migration! My concern with Justfile is that it doesn't support dynamic functions, which we make use of (i.e. available-node, etc)

[joyeecheung]

The Makefile is just a glorified script, but with dependency checking out of the box, so that we can implement things like "if the test addons have been built and the test files, the headers and the common config etc. have not changed, don't rebuild before we run the addon tests" (addons are built in parallel by a JS script, it may be possible to port that to GYP but we may lose the parallelism, also GYP is harder to configure to drive the addon building logic), or "if the C++ files don't change, don't run the C++ linter, if they do, only run the linter on the changed files", "if the docs don't change, don't rebuild the docs - which are also needed by doc tests". I think that is also missing from the vcbuild.bat as it almost always rebuild a lot of stuff unnecessarily. If we want to switch to a different scripting option, that dependency checking part should also be ported over.

(Personally most of the time I just use configure + ninja locally, and only very occasionally run the full test suite locally. On Windows I've come to realize that it's better to avoid vcbuild.bat and just use visual studio for proper incremental builds. If some cross-platform scripting with proper incremental build & dependency checking support is available I would be happy to use that. But if dependency checking is missing I would still hope that we can just port the tasks to visual studio custom targets)

[avivkeller]

Okay. I'm going to work on seeing what different setups provide the best results for dependency management and cross platform ability.

[avivkeller]

Okay, I've looked into a few options, and here's my assessment.

Tool	Pros	Cons	Examples
CMake	- Cross-Platform	- Requires users to have CMake installed - On Windows, each target is defined in separate files - Not ideal for advanced or custom targets (e.g., $(PKG) target)	Bun's CMakeLists.txt1
Bazel	- Cross-Platform - Fast - Scalable	- Requires Bazel installation - Lacks native support for scripting or custom build steps - Steep learning curve	V8's BUILD.bazel
Python	- Cross-Platform - Extremely flexible; can manage any build logic - Seamless integration with Python-based tools - Rich standard library for various tasks	- Requires boilerplate for dependency management and basic build tasks (e.g., filter-out)	SymCrypt's Pythonic scripts
JavaScript	- Cross-Platform - Familiar for JavaScript developers in the repo - Flexible, can incorporate custom logic	- Requires node2 - Less suitable for non-JavaScript-based tasks	Bun's package.json
Make/VCBuild	- Established and familiar	- Not natively cross-platform (requires porting for Windows) - Lacks modern features or advanced build capabilities	Moby's Makefile
Just	- Can write targets using Python, utilizing Python's full capability - Simple and lightweight - Similar to Makefile	- Cannot define functions/macros outside of targets - Limited conditional logic - Requires Just to be installed	ts-node's justfile

---

After checking out the available options, CMake and Bazel can be excluded right away (IMO). Both have limitations when it comes to handling complex or custom targets, which would necessitate creating separate scripts for each one, essentially defeating the purpose of having a unified build system.

When considering JavaScript, Just, and Python, it gets a bit more complex:

• JavaScript: While flexible, it requires node to build node², and it's fair to say that assuming every system building the project has all the necessary components is unrealistic.

• Just: Although it's lightweight and allows us to write targets in Python, it has significant drawbacks, such as a limited feature set and lack of advanced conditional logic. It also requires users to install the tool itself.

Ultimately, I think Python is the best option. It is cross-platform, highly flexible, and can seamlessly integrate with our other Python-based tools. While it requires more boilerplate compared to specialized build systems, the extra effort offers significantly more control and flexibility. Python’s familiarity with many contributors already makes it an ideal choice for handling our custom build steps, even those involving complex targets like $(PKG).

Footnotes

1. While Bun uses a CMakeLists.txt file for compilation, it uses JavaScript for script execution. ↩

2. Sort of a Circular Dependency (although, you can use an older version) ↩ ↩²

[bnoordhuis]

What exactly are you trying to accomplish? Getting rid of gyp? Or replacing Makefile+vcbuild.bat with something else?

[avivkeller]

Or replacing Makefile+vcbuild.bat with something else?

This. Gyp is cross-platform, Makefile/VCBuild is not