As they observe, Ninja gets to be fast mostly by cheating: it avoids a lot of work by saying many things are just out of scope for Ninja to do, and that means it is a useful a target to race against. (Funny thing: when I wrote Ninja I was misremembering how fast an earlier build system was so I kept trying to make it faster. So don't treat it as a lower bound, I just made it up!)
I comment here to say I find the explanation for 'why' in this post unsatisfying. They mention three design decisions.
The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. I might have misunderstood?
The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely. It's a very small amount of work: the post mentions 300 compiles, so maybe parsing 300 small text files?
The third is that they run the compiler up front an additional time to gather headers, which is strictly more work than Ninja. There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches. They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
Maybe it's just my own curiosity, I think this post would be better if it had a better explanation for the reason. I'm not disputing the result, I just think the result should make you suspicious that something else is going on, and you might learn something from that! You could for example explore whether it's the header dependency thing by profiling the Ninja invocation and seeing if it's waiting for CPU or waiting for tasks to execute.
(If I had to guess without looking at any of the involved code, I would predict it's something about how CMake generates the build, like it introduces serialization in a place where build2 is parallel, or it adds some extra build steps like gathering the current git hash into a header file or something.)
This claim isn't supported by the author's measurements?
Every effect has a cause, and when it comes to performance, it's crucial you understand that cause. If (as in this case) something, from first principles, shouldn't be faster, yet it is, you dig and understand, not just say your shit rules and the other thing sux0rz.
no?
And if they are in their own targets, they are not really an issue (they would serialize everything that depends on them as you'd expect), but if you have them in a library grouped with other files to compile, then the whole library compilation is serialized.
And obviously worse if you also have to build the generator for the generated files, but that's not a big surprise, you can observe that in full builds of Chromium or its libraries too waiting for protoc if you crank the parallelization a lot.
Since they had to rewrite the build file for their program I also assume that something is missing. Didn’t see any mention of verifying that.
I also really didn’t like their denigrating tone. It totally turns me off trying build2, because it seems they don’t understand the point of separating build stages like environment setup (getting dependencies), configure, native build, cross build, packaging. I am a very happy ninja user instead of a batteries-included solution because it does its one job well and can be used very flexibly. I personally detest cmake, so I use nix + own configure script + ninja.
The blog is also wrong about ninja being unable to call configure, but I intentionally don’t want that (I want the build stages to communicate in one direction for sanity).
That's easy to do if you control the whole pipeline and can integrate all the features together, not so much with the CMake model unfortunately. I think it would be nice if CMake had Ninja integrated as a library, it could lead to some nice optimizations later.
Something is wrong here. Which compression algorithm is being used here and how much has it been tuned? A core hypothesis of the likes of zram is that disk access is so slow (even NVME), that you can often beat it with the bit-rate of decompression.
1. Is something slow like gzip being used?
2. Is the compression effort over-tuned for size? Do some space benchmarks and make sure that you aren't saving a few dozen MB on GBs of data.
zstd, with 1-3 effort (you may even find negative is a overall win), and a trained dictionary (your data does all look identical) is probably a good start.
* Because in CMake there are several target properties that may be affected by the "parent" (another target which added this target as dependency), meaning that these properties need to be re-evaluated in the context of each root target dependent on this one. If I remember correctly, exact mechanism is that some properties can have generator expressions, and these expressions can reference "parent" target. Now we imagine an app which has some std lib with 1k targets, and 100 top-level executables dependent on this std lib - suddenly we have 100k target evaluations in generation step.
I recently (a few weeks ago) made a project to batch those type of commands together and parallelize tests as much as possible. You can check a merge request for CMake upstream here: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/1226... and the associated issue for more details.
In general, CMake is fast enough, the only slow bit on the critical path is detecting the toolchain. The rest is quick, but if your build scripts are using slow serialized functionality, you'll be paying that price. But only once as the slow results are cached, so next incremental run should be pretty fast!
In practice, it does show the limitations of the CMake model though. You should not be paying the price to generate the build files for bits you don't care about. You should be able to just load the CMake files and build a target, ignoring the rest.
This is how tools like Bazel function and that allows them to scale to very large monorepos. Build files are only processed when a target from that folder is built directly or used. CMake has to load everything as it does not know about the build tree and the maintainer has to manage exclusion lists or inclusion lists.
When, from first principles, something shouldn't be faster, yet it is, you dig and dig until you understand. You don't just say your shit rules and the other thing sux0rz. If you do, you'll regret it: if you don't understand how you got a win, you don't understand how to keep it.
Everyone's workflow is different. Your's obviously works for you and many others.
But I like local man pages. They get updated at the same time as the software is updated. Sometimes I have an older version of software and the online version is for a newer version, or vice-versa. Local man pages are always available and generally quicker than accessing them online.
If the machines in question are mult user, you should always install local documentation regardless of your personal preferences.
They do care that the code can compile on their (possibly old) machine. They do care about the ability to include some features and not others (including their ability to not have to make decisions on this if they don't want to). And they certainly want documentation that helps them to achieve their goal (which, again, is to build the software).
The newer tools make substantial inroads in making the developer's life easier. If your software is not of the sort where people download it and try to build it (because it only ships as a closed binary, is entirely internal, etc.) then maybe that makes sense. But until all these fancy gadgets start seriously caring about the use case above, I'm going to stick with autotools.
In my experience is it doesn't work. Sure it can be made to work and autotools asks all the right questions. However in the real world I have yet to see an autotools project that works out of the box in my custom environment.
Meanwhile cmake just worked every time when I plugged in the toolchain file that I created for the very first project I needed to build for my environment.
Again, it isn't that autotools cannot do what is claims. However nobody gets the details right and so something doesn't work. That something is different for every project. In the end what works is either what someone takes the time to make work and those things that are "very close". My project is cross compiling linux, which cmake does well (autotools also messed something up, often failing to use the sysroot paramater where needed)
Posted on 5 Aug 2026 by Boris Kolpackov
Some months ago I read an article about a tool for snooping on slow build systems that can be used to find build bottlenecks. While the whole article is quite illuminating, this quote stood out to me:
Ninja is not a 100% fair comparison to other tools, because it benefits from some "baked in" build logic by the tool that created the ninja file, but I think it's a reasonable "speed of light" performance benchmark for build systems.
To clarify, by "baked in" the author means that nobody writes Ninja build files by hand. Rather, a second tool, such as CMake, is invoked to generate them and some steps performed during this generation phase (examples below) should ideally be part of the build phase. Also, Ninja is notoriously minimalist, providing only the bare minimum of functionality, especially on the change tracking side of things (again, examples below). A modern build system would be expected to provide more.
Still, it would be interesting to see how close a modern, native (that is, without the generation step) build system can approach the "speed of light".
Let's take a look at how build2 measures up. While there is a number of substantial projects (such as Boost and Qt) that can be built with both build systems, finding a project of substance that would result in an apples-to-apples comparison is difficult because when we package more complex projects for build2, we invariably have to untangle the "ball of intra-dependencies" structure into something more orderly (for a good example, take a look at the upstream qtbase module versus build2 packages). And this usually results in a slightly different set of intermediate build artifacts, like bootstrap and utility libraries.
So we will have to make do with something simpler, where we can make sure the same set of object files and binaries is produced with more or less identical compile and link options. In the end I've picked Xerces-C++, an XML parser/serializer for C++. It has quite a few features (like XML Schema validation) so it's not exactly tiny, measuring 299 C++ translation units that are linked into a shared library.
We are going to test a full, from-scratch build, the same as in the quoted article. Ninja completes this build on my machine (see Benchmark Details below) in 3.4s:
Time (mean ± σ): 3.429 s ± 0.029 s [User: 48.536s, System: 5.033s] Range (min … max): 3.383 s … 3.464 s 10 runs
Before we measure build2, let's at least acknowledge the elephant in the room: while Ninja builds the project in 3.4s, CMake takes 15.6s to generate the Ninja build files. So if you had Xerces-C++ as a dependency of your project and it was being built from scratch, you would wait 19 seconds, not 3.4, for this build.
With the matching configuration (same C++ compiler, C++ standard, debug build, etc) build2 takes 3.8s, or about 11% slower:
Time (mean ± σ): 3.808 s ± 0.046 s [User: 58.037s, System: 8.012s] Range (min … max): 3.746 s … 3.886 s 10 runs
Pretty close, but not at the speed of light. Let's see if we can get there. Maybe building in vacuum will help?
To try to get closer to Ninja's time we are going to make the comparison more accurately apples-to-apples. As discussed above, Ninja is notoriously minimalist with build2 providing a lot of functionality that Ninja does not. And some of this functionality has measurable cost, performance-wise. So we are going to disable a few features to closer match the amount of work done by Ninja.
The first feature that we will disable is the more precise change tracking for C and C++ source files. Ninja simply checks whether the file's modification time has changed and if so, recompiles it. build2, in contrast, performs an extra step in this case: it tokenizes the (partially-preprocessed) source file and computes the checksum of the resulting tokens. If this checksum hasn't changed since the last time the file was compiled, then it skips recompiling it. This ignores whitespace-only changes (as long as they do not alter the column numbers of the tokens) and is very useful during development (and is critical in some case, like if you want to change your project's version with every commit). But tokenizing all the 299 translation units in the from-scratch build has an upfront cost, even if it may pay off during further incremental builds.
The way to disable this ignorable change detection is to tell build2 that the project is read-only (which is done automatically by the package manager for external dependencies). In this case build2 will fall back to using just the modification time, the same as Ninja. With this change our build time goes down to 3.4s, pretty much the same as Ninja's:
Time (mean ± σ): 3.433 s ± 0.055 s [User: 51.093s, System: 6.701s] Range (min … max): 3.383 s … 3.551 s 10 runs
Let's see if we can go even faster. Next, we disable compression in the file cache. We will discuss the file cache in more detail a bit later but for now let's just say that by disabling compression we trade temporary disk space usage for speed:
Time (mean ± σ): 3.355 s ± 0.067 s [User: 49.987s, System: 6.153s] Range (min … max): 3.281 s … 3.471 s 10 runs
And now we are 2.2% faster than Ninja! While this may not seem like much, it becomes more impressive considering build2 still does a lot more than Ninja. Some of this work is done by CMake and some is just not done at all.
For example, build2 generates the XercesVersion.hpp header from XercesVersion.hpp.in as part of the build while Ninja leaves this to CMake. build2 also makes sure this file is properly change-tracked (while Ninja expects you to re-run CMake manually). This header is included in pretty much every translation unit in Xerces-C++, meaning that no compilation can start until it is generated. As an experiment, I hacked the build2 build file to pretend XercesVersion.hpp is static. That increased the gap to 2.6%.
build2 also has to extract a lot more information from the compiler, something that in the case of Ninja is, again, done by CMake. Things like the compiler id (GCC, Clang, etc) and its version, target platform, list of system header and library search paths, C and C++ standard libraries used, etc. This information is both made available to build files as well as used to implement more precise change tracking. For example, for C and C++, besides tracking changes to the standard inputs such as the source file itself, all the included headers, and the compile options, build2 also tracks the compiler id/version, system header search paths, and environment variables that may affect the compilation.
Currently, the only reliable way to extract this information is to run the compiler. And, unfortunately, to extract all the bits listed above, we have to run it multiple times. For example, in case of GCC, there are 10 invocations in total, 5 for C and 5 for C++ (Xerces-C++ has a few C translation units):
LC_ALL=C gcc-15 -v gcc-15 -g -print-multiarch gcc-15 -g -x c -E - LC_ALL=C gcc-15 -std=c9x -print-search-dirs LC_ALL=C gcc-15 -std=c9x -x c -v -E -
LC_ALL=C g++-15 -v g++-15 -g -print-multiarch g++-15 -g -x c++ -E - LC_ALL=C g++-15 -std=gnu++17 -print-search-dirs LC_ALL=C g++-15 -std=gnu++17 -x c++ -v -E -
Ok, so build2 does quite a bit more and is on par or even faster than Ninja, even when we choose not to see the elephant (CMake). The next natural question to ask is how does it do it? Ninja was designed from the start with performance in mind and then saw decades of heavy use and optimization. There is little chance of any fruit, let alone low-hanging ones, left for us to pick. So what's the secret?
In a nutshell, we have to do things differently, not just better. I can think of three major design decisions that contributed to this. It's not easy to verify empirically since it would be pretty difficult to test alternative designs in isolation, but I think they are the most likely reasons.
The first is more of a what not to do rather than what to do. Again, nothing will illustrate the point better than the above mentioned article:
Here’s a tiny slice of a CMake build from another open source project:
[...]
Here CMake gets Xcode’s path with xcode-select -print-path, the OS version with sw_vers, and then recursively calls cmake/make a few times for good measure, and finally compiles and links a file.
Only the green boxes in that timeline are doing useful work. One could argue that none of what CMake does is "useful work", in the sense that it just builds the thing that actually builds the project. Regardless, let’s just accept that CMake needs to do this weird cmake->make->make->clang dance to figure out the build environment.
Zooming out reveals that the weird dance happens 85 times!
Yikes, no parallelism. It also studiously re-checks the Xcode path and OS version 85 times, just in case the OS version changes mid-build.
Needless to say, you are not going to get very far performance-wise with such an approach. In contrast, in build2 we aggressively cache every piece of discovered information (but not across the build system runs) to make sure we don't redo any of the work unnecessarily.
Ok, I will stop picking on CMake and focus on Ninja. The second biggest difference between Ninja and build2 (the first being it's a native build system) is build2 being multi-threaded. While Ninja executes compilers, linkers, etc., in parallel, it performs its own housekeeping work serially, from a single thread. For example, both Ninja and build2 need to parse the header dependency information received from GCC, which is inconveniently produced as Makefile fragments. build2 does this (and a lot of other things) in parallel from multiple threads. In fact, the only serial phase in build2 is loading of build files (and even for that there are ideas on how to parallelize some parts of it).
The last design decision that I think contributes to build2 matching or exceeding Ninja's speed is a different C/C++ build model when it comes to the extraction of header dependency information (the list of headers included by every translation unit, transitively).
Ninja uses what we can call a byproduct of compilation model: during the from-scratch build, Ninja compiles every translation unit and gets the header dependency information essentially for free. It parses and stores it in its own format on disk to be used during the next build to see if any translation units need to be recompiled because some headers they include may have changed. This is a clever approach but unfortunately it doesn't work well with auto-generated headers (and is the reason why headers like XercesVersion.hpp are generated by CMake).
In build2 we perform explicit header dependency extraction before we compile a translation unit. One can reasonably expect a naive implementation to perform strictly worse than the byproduct approach, where this extraction is free. What we do in build2 is a bit more advanced: because a header dependency extraction is essentially a preprocessor run on the translation unit, we combine the dependency extraction with partial preprocessing of each translation unit. Or, in other words, instead of just extracting header dependencies, we perform partial preprocessing (-fdirective-only for GCC, -frewrite-includes for Clang) of each translation unit and get the header dependency information as a byproduct of that. Then, when the time comes to compile the translation unit, we compile this partially-preprocessed output instead of the original source file, thus saving on re-preprocessing it.
Still, this sounds like a lot more work than what Ninja does. We even have to store the partially-preprocessed files on disk, how can this be faster than doing nothing? Interestingly, there appears to be another aspect at play: The way Ninja does it, preprocessing of translation units is spread out over the entire build, time-wise. There is also substantial demand for RAM since C and especially C++ compilation is memory-intensive. In contrast, with the build2 approach, all the preprocessing is front-loaded, it's all concentrated at the beginning of the build, before memory-intensive compilation starts. And preprocessing of a typical C/C++ translation unit involves including hundreds of header files (many of them the same as for other translation units), all of which need to be read from disk. It turns out, at least on Linux with GCC, the build2 approach performs better, likely due to better temporal locality of file access and lower memory pressure. In other words, with the build2 approach, the included headers are more likely to still sit in the system's file cache (for details, see Separate Preprocess and Compile Performance).
The benchmark was executed on a physical machine with Intel i9-12900K CPU running Debian on Samsung 980 Pro NVMe formatted as ext4. To obtain stable numbers I disabled turbo boost (failed that, the times gradually increased as the CPU got warmer):
echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo
I used the official Xerces-C++ 3.3.0 source archive for Ninja and the libxerces-c-3.3.0+3 build2 package (which is the official source overlayed with build2 support). I had to modify CMakeLists.txt to change the hardcoded C++ standard from 14 to 17 since the system version of ICU on Debian no longer supports C++14.
The CMake command to prepare the Ninja build:
CC=gcc-15 CXX=g++-15 cmake -G Ninja \ -DCMAKE_BUILD_TYPE=Debug \ -Dnetwork:BOOL=OFF \ -Dtranscoder=icu \ -Dmessage-loader=inmemory \ -Dmutex-manager=standard \ -Dxmlch-type=char16_t \ ../xerces-c-3.3.0
Then to run the Ninja benchmark:
hyperfine --style=basic --warmup 1 --runs 10 \ --prepare 'ninja clean' \ 'ninja src/all'
To run the build2 benchmark we don't need to prepare anything. Below are the command lines for the three measurements shown above:
hyperfine --style=basic --warmup 1 --runs 10 \ --prepare 'b clean: xercesc/' \ "b config.c=gcc-15 config.cxx=g++-15 config.cxx.std=gnu++17 \ config.cc.coptions=-g config.bin.lib=shared \ xercesc/"
hyperfine --style=basic --warmup 1 --runs 10 \ --prepare 'b clean: xercesc/' \ "b config.c=gcc-15 config.cxx=g++-15 config.cxx.std=gnu++17 \ config.cc.coptions=-g config.bin.lib=shared \ config.libxerces_c.build.readonly=true xercesc/"
hyperfine --style=basic --warmup 1 --runs 10 \ --prepare 'b clean: xercesc/' \ "b --file-cache=none \ config.c=gcc-15 config.cxx=g++-15 config.cxx.std=gnu++17 \ config.cc.coptions=-g config.bin.lib=shared \ config.libxerces_c.build.readonly=true xercesc/"