It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason about, or in the worst case, terrifying to touch.
We had to fork BuildKit very early in our Depot journey. We've fixed a ton of things in it that we hit for our use case. Some of them we tried to upstream early on, but only for it to die on the vine for one reason or another.
Today, our container builders are our own version of BuildKit, so we maintain 100% compatibility with the ecosystem. But our implementation is greatly simplified. I hope someday we can open-source that implementation to give back and show what is possible with these ideas applied at scale.
But the real hidden power of buildkit is the ability to swap out the Dockerfile parser. If you want to see that in action, look at this Dockerfile (yes, that's yaml) used for one of their hardened images: https://github.com/docker-hardened-images/catalog/blob/main/...
It sounds great in theory, but it JustDoesn'tWork(tm).
Its caching is plain broken, and the overhead of transmitting the entire build state to the remote computer every time is just busywork for most cases. I switched to Podman+buildah as a result, because it uses the previous dead simple Docker layered build system.
If you don't believe me, try to make caching work on Github with multi-stage images. Just have a base image and a couple of other images produced from it and try to use the GHA cache to minimize the amount of pulled data.
BuildKit is very cool tech, but painful to run at volume
Fun gotchya in BuildKit direct versus Dockerfiles, is the map iteration you loaded those ENV vars into consistent? No, that's why your cache keeps getting busted. You can't do this in the linear Dockerfile
The code first options are quite good these days, but you can get so far with make & other legacy tooling. Docker feels like a company looking to sell enterprise software first and foremost, not move the industry standard forward
great article tho!
It's clear a ton of care has gone into the product, and I also appreciated you personally jumping onto some of my support tickets when I was just getting things off the ground.
This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization - by the time they have active outside development, those idiosyncratic concepts are already ossified.
It's really rare to see these sorts of projects decomposed into building blocks even just having code organization that helps a newcomer understand. Despite all the code being out in public, all the important reasoning about why certain things are the way they are is trapped inside a few dev's heads.
Also, there is nothing stopping you from creating a container that has make + tools required to compile your source code, writing a dockerfile that uses those tools to produce the output and leave it on the file system. Why that approach? Less friction for compiling since I find most make users have more pet build servers then cattle or making modifications can have a lot of friction due to conflicts.
Edit: The claim about the hash being the same is incorrect, but an identical Dockerfile can produce different outputs on different machines/days whereas nix will always produce the same output for a given input.
Managers and executives are happy to hear that you made the builds faster or more reliable, so the infra people who care about this kind of thing don't waste time on design docs and instead focus on getting to a minimum prototype that demonstrates those improved metrics. Once you have that, then there's buy-in and the project is made official... but by then the bones have already been set in place, so design documentation ends up focused on the more visible stuff like user interface, storage formats, etc.
OTOH, bazel (as blaze) was a very intentionally designed second system at Google, and buildx/buildkit is similarly a rewrite of the container builder for Docker, so both of them should have been pretty free of accidental engineering in their early phases.
I find that buildah is sort of unbearably slow when using dockerfiles...
Everything runs in its own docker runner. New buildkitd service for every job. Caching only via buildkit native cache export. Output format oci image compressed with zstd. Works pretty great so far, same or faster builds and we now create multi arch images. All on rootless runners by the way
If they didn't take shortcuts. I don't know if it's been fixed, but at one point Vuze in nix pulled in an arbitrary jar file from a URL. I had to dig through it because the jar had been updated at some point but not the nix config and it was failing at an odd place.
It's yet one more incomprehensible Buildkit decision. The original Docker builder had a very simple cache system: it computed the layer hash and then checked the registry for its presence. Simple content-addressable caching.
Buildkit can NOT do this. Instead, it uses a single image as a dumping ground for the caches. If you have two builders using the same image, they'll step on each other's toes. GHA at least side-steps this.
But I tried the registry cache, and it didn't improve anything. So far, I was not able to get caching to work with multi-stage builds at all. There are open issues for that, dating back to 2020.
Had to recently make it so multiple versions can run on the same host, such that as developers change branches, which may be on different IaC'd versions (we launch on demand), we don't break LTS release branches.
Compared to a Dockerfile it's just too hard to follow
The cache key includes the state of the filesystem so I donβt think that would ever be true.
Regardless, the purpose of the tool is to generate [layer] images to be reused, exactly to avoid the pitfalls of reproducible builds, isnβt it? In the context of the article, what makes builds reproducible is the shared cache.
https://tuananh.net/img/buildkit-llb.png
Maybe the page was changed? If you're just talking about the gaps between lines, that's just the line height in whatever source was used to render the image, which doesn't say much about AI either way.
Most people interact with BuildKit every day without realizing it. When you run docker build, BuildKit is the engine behind it. But reducing BuildKit to βthe thing that builds Dockerfilesβ is like calling LLVM βthe thing that compiles C.β It undersells the architecture by an order of magnitude.
BuildKit is a general-purpose, pluggable build framework. It can produce OCI images, yes, but also tarballs, local directories, APK packages, RPMs, or anything else you can describe as a directed acyclic graph of filesystem operations. The Dockerfile is just one frontend. You can write your own.
BuildKitβs design is clean and surprisingly understandable once you see the layers. There are three key concepts.
At the heart of BuildKit is LLB (Low-Level Build definition). Think of it as the LLVM IR of build systems. LLB is a binary protocol (protobuf) that describes a DAG of filesystem operations: run a command, copy files, mount a filesystem. Itβs content-addressable, which means identical operations produce identical hashes, enabling aggressive caching.
When you write a Dockerfile, the Dockerfile frontend parses it and emits LLB. But nothing in BuildKit requires that the input be a Dockerfile. Any program that can produce valid LLB can drive BuildKit.
A frontend is a container image that BuildKit runs to convert your build definition (Dockerfile, YAML, JSON, HCL, whatever) into LLB. The frontend receives the build context and the build file through the BuildKit Gateway API, and returns a serialized LLB graph.
This is the key insight: the build language is not baked into BuildKit. Itβs a pluggable layer. You can write a frontend that reads a YAML spec, a TOML config, or a custom DSL, and BuildKit will execute it the same way it executes Dockerfiles.
Youβve actually seen this mechanism before. The # syntax= directive at the top of a Dockerfile tells BuildKit which frontend image to use. # syntax=docker/dockerfile:1 is just the default. You can point it at any image.
The solver takes the LLB graph and executes it. Each vertex in the DAG is content-addressed, so if youβve already built a particular step with the same inputs, BuildKit skips it entirely. This is why BuildKit is fast: it doesnβt just cache layers linearly like the old Docker builder. It caches at the operation level across the entire graph, and it can execute independent branches in parallel.
The cache can be local, inline (embedded in the image), or remote (a registry). This makes BuildKit builds reproducible and shareable across CI runners.
βββββββββββββββ ββββββββββββ ββββββββββ βββββββββββ ββββββββββββββ
β Your spec βββββΆβ Frontend βββββΆβ LLB βββββΆβ Solver βββββΆβ Output β
β (YAML, HCL, β β (Gateway β β (DAG) β β (cache, β β (image, β
β Dockerfile)β β client) β β β β execute)β β tarball, β
βββββββββββββββ ββββββββββββ ββββββββββ βββββββββββ β local dir) β
ββββββββββββββ
BuildKitβs --output flag is where this gets practical. You can tell BuildKit to export the result as:
type=image β push to a registry (the default for docker build)type=local,dest=./out β dump the final filesystem to a local directorytype=tar,dest=./out.tar β export as a tarballtype=oci β export as an OCI image tarballThe type=local output is the most interesting for non-image use cases. Your build can produce compiled binaries, packages, documentation, or anything else, and BuildKit will dump the result to disk. No container image required.
Projects like Earthly , Dagger , and Depot are all built on top of BuildKitβs LLB. Itβs a proven pattern.
To demonstrate this concretely, I built apkbuild : a custom BuildKit frontend that reads a YAML spec and produces Alpine APK packages. No Dockerfile involved. The entire build pipeline β from source compilation to APK packaging β runs inside BuildKit using LLB operations. Think of this like a dummy version of Chainguardβs melange
I chose YAML for familiarity, but the spec could be anything you want (JSON, TOML, a custom DSL) as long as your frontend can parse it.
My package YAML spec looks like this:
name: hello
version: "1.0.0"
epoch: "0"
url: https://example.com/hello
license: MIT
description: Minimal CMake APK demo
sources:
app:
context: {}
build:
source_dir: hello
Thatβs it. No Dockerfile. No shell scripts. BuildKit reads this spec through the custom frontend and produces a .apk file.
Build the frontend image:
docker build -t tuananh/apkbuild -f Dockerfile .
Then use it to build an APK package:
cd example
docker buildx build \
-f spec.yml \
--build-arg BUILDKIT_SYNTAX=tuananh/apkbuild \
--output type=local,dest=./out \
.
You should be able to see the APK package in the out folder like below

BUILDKIT_SYNTAX tells BuildKit to use our custom frontend instead of the default Dockerfile parser. The --output type=local dumps the resulting .apk files to ./out. No image is created. No registry is involved.
BuildKit gives you a content-addressable, parallelized, cached build engine for free. You donβt need to reinvent caching, parallelism, or reproducibility. You write a frontend that translates your spec into LLB, and BuildKit handles the rest.
This is relevant beyond toy demos. Dagger uses LLB as its execution engine for CI/CD pipelines. Earthly compiles Earthfiles into LLB. The pattern is proven at scale.
If youβre building a tool that needs to compile code, produce artifacts, or orchestrate multi-step builds, consider BuildKit as your execution backend. The Dockerfile is just the default frontend. The real power is in the engine underneath.