Arch Linux Now Has a Bit-for-Bit Reproducible Docker Image

It is nice to have this confidence.

I ran Arch Linux for almost a year in WSL 2, it was really good.

Then I ran Arch natively for ~5 months, it's really good.

Now I still run Arch natively, but I also use the Arch Docker image to test my dotfiles[0] with a fresh file system.

Also, for when I want to run end to end tests for my dotfiles that set up a complete desktop environment I run Arch in a VM.

I have 99 problems but running Arch isn't one of them.

[0]: https://github.com/nickjj/dotfiles

All docker containers should have been like that. apt-get update in a docker build step is an anti pattern.

reproducible images are one of those features where the payoff is mostly emotional until the day it isn't. we had an incident where two supposedly identical images on two machines had a three byte delta in a timestamp and it cost us an afternoon to bisect from the wrong end. boring win, but a real one.

A totally unrelated comment; but — there is an animation on that page that moves practically everything on the page about 20 pixels down over the course of 1 second.

I thought that would completely trash the Cumulative Layout Shift core web vital. Because, hey! the layout is shifting in front of my very eyes. But no, the CLS on the page is 0.

Is CLS a misleading metric then?

This is a really interesting accomplishment - I am also working heavily on reproducible builds for my firmware projects, and .. lo and behold .. the package manager key administrivia is the final bone to be broken.

I wonder if Arch leading the way on this will prompt other distro's to attempt the same feat. Reproducible builds are important for certification, security and safety-critical applications .. it'd be great to see Linux distros become more conformant to this method.

and they said compilers are deterministic...

This is a huge accomplishment! But it wouldn't be so huge if compilers were trivially deterministic. It took 5 decades of development for compilers to get here. I'm sure ChatGPT in 2073 is going to be more deterministic than it was in 2023.

It is nice to have this confidence.

I ran Arch Linux for almost a year in WSL 2, it was really good.

Then I ran Arch natively for ~5 months, it's really good.

Now I still run Arch natively, but I also use the Arch Docker image to test my dotfiles[0] with a fresh file system.

Also, for when I want to run end to end tests for my dotfiles that set up a complete desktop environment I run Arch in a VM.

I have 99 problems but running Arch isn't one of them.

[0]: https://github.com/nickjj/dotfiles

and they said compilers are deterministic...

All docker containers should have been like that. apt-get update in a docker build step is an anti pattern.

I disagree with that as a hard rule and with the opinion that it's an anti-pattern. Reproducible containers are fine, but not always necessary. There's enough times when I do want to run apt-get in a container and don't care about reproducibility.

You are screwed either way. If you don't update your container has a ton of known security issues, if you do the container is not reproducable. reproducable is neat with some useful security benefits, but it is something a non goal if the container is more than a month old - day might even be a better max age.

I know it's an anti-pattern, but what is the alternative if you need to install some software? Pulling its tagged source code, gcc and compile everything?

This has been a solved problem for over two decades now with Nix but people can't be asked

This is to solve such issues that I am using and running StableBuild.

It is a managed service that keeps a cached copy of your dependencies at a specific time. You can pin your dependencies within a Dockerfile and have reproducible docker images.

Debian already has an ongoing project for this: https://wiki.debian.org/ReproducibleBuilds.

A totally unrelated comment; but — there is an animation on that page that moves practically everything on the page about 20 pixels down over the course of 1 second.

I thought that would completely trash the Cumulative Layout Shift core web vital. Because, hey! the layout is shifting in front of my very eyes. But no, the CLS on the page is 0.

Is CLS a misleading metric then?

It's happening as a result of a deliberate animation. The CLS metric relates to initial render. So yes, there is layout shift, but it's not CLS per se.

The layout isn't shifting, so it's not a layout shift.

And it's not unexpected, because it comes from a css transition.

How did a differing timestamp cause an incident in the first place? Curious.

Debian already has an ongoing project for this: https://wiki.debian.org/ReproducibleBuilds.

I update my docker containers regularly but doing it in a reproducible, auditable, predictable way

I know it's an anti-pattern, but what is the alternative if you need to install some software? Pulling its tagged source code, gcc and compile everything?

Run “nix flake update”. Commit the lockfile. Build a docker image from that; the software you need is almost certainly there, and there’s a handy docker helper.

Copying from another image is an under appreciated feature

FROM ubuntu:24.04

COPY --from=ghcr.io/owner/image:latest /usr/local/bin/somebinary /usr/local/bin/somebinary

CMD ["somebinary"]

Not as simple when you need shared dependencies

Both Debian and Ubuntu provide snapshot mirrors where you can specify a date to get the package lists as they looked at that time.

pretend you don't do it and add your extra software to the layer above

base image

software component image

both should be version pinned for auditing

With a binary cache that is not so bad, see for example what nix does.

This has been a solved problem for over two decades now with Nix but people can't be asked

This is to solve such issues that I am using and running StableBuild.

It is a managed service that keeps a cached copy of your dependencies at a specific time. You can pin your dependencies within a Dockerfile and have reproducible docker images.

It has been solved even without Nix for a long time, just laziness is probably why we are not doing it

I don't wanna be that guy but...

NIX FIXES THIS.

It's happening as a result of a deliberate animation. The CLS metric relates to initial render. So yes, there is layout shift, but it's not CLS per se.

> The CLS metric relates to initial render.

The CLS measures the total sum of layout shifts over the entire lifespan of a page, not just during initial render.

The layout isn't shifting, so it's not a layout shift.

And it's not unexpected, because it comes from a css transition.

Sure.

It's just that the spirit of Google's core web vitals has been to measure the properties of a web page that have the most impact on users. How quickly content appears on a page, how visually stable the content is, and how long it takes the page to respond to an interaction.

In the case of this page, I don't think it can be considered visually stable at all in the first second after it's loaded.

And yet, core web vitals cannot demonstrate this.

How did a differing timestamp cause an incident in the first place? Curious.

My guess is it was the only obvious evidence of an attack.

Copying from another image is an under appreciated feature

FROM ubuntu:24.04

COPY --from=ghcr.io/owner/image:latest /usr/local/bin/somebinary /usr/local/bin/somebinary

CMD ["somebinary"]

Not as simple when you need shared dependencies

base image

software component image

both should be version pinned for auditing

Run “nix flake update”. Commit the lockfile. Build a docker image from that; the software you need is almost certainly there, and there’s a handy docker helper.

Recently I’ve been noticing that Nix software has been falling behind. So “the software you need is almost certainly there” is less true these days. Recently = April 2026.

oh, great, adding more dependency, and one that just had serious security problem

I update my docker containers regularly but doing it in a reproducible, auditable, predictable way

Both Debian and Ubuntu provide snapshot mirrors where you can specify a date to get the package lists as they looked at that time.

With a binary cache that is not so bad, see for example what nix does.

pretend you don't do it and add your extra software to the layer above

Could you explain how you achieve this?

Which is only useful for historical invesigation - the old snapshot has security holes attackers know how to exploit.

I don't really see how that's different from a normal binary install of a reproducible package. Especially with the lacking quality of a lot of Nix packages.

It has been solved even without Nix for a long time, just laziness is probably why we are not doing it

Sure.

In the case of this page, I don't think it can be considered visually stable at all in the first second after it's loaded.

And yet, core web vitals cannot demonstrate this.

> The CLS metric relates to initial render.

The CLS measures the total sum of layout shifts over the entire lifespan of a page, not just during initial render.

My guess is it was the only obvious evidence of an attack.

I don't wanna be that guy but...

NIX FIXES THIS.

Recently I’ve been noticing that Nix software has been falling behind. So “the software you need is almost certainly there” is less true these days. Recently = April 2026.

Are you referring to how the nixpkgs-unstable branch hasn't been updated in the past five days? Or do you have some specific software in mind? (not arguing, just curious)

oh, great, adding more dependency, and one that just had serious security problem

as if other sandboxing software is perfect

I don't really see how that's different from a normal binary install of a reproducible package. Especially with the lacking quality of a lot of Nix packages.

If you're in a situation where you want reproducibility you're using nix to build your own packages anyways, not relying on their packages

It's not if you can pin the package. It gives you reproducable docker containers without having to rebuild the world. Wasn't that the entire question?

Could you explain how you achieve this?

Which is only useful for historical invesigation - the old snapshot has security holes attackers know how to exploit.

Chainguard, Docker Inc’s DHI etc. There’s a whole industry for this.

> the old snapshot has security holes attackers know how to exploit.

So is running `docker build` and the `RUN apt update` line doing a cache hit, except the latter is silent.

The problem solved by pinning to the snapshot is not to magically be secure, it's knowing what a given image is made of so you can trivially assert which ones are safe and which ones aren't.

In both cases you have to rebuild an image anyway so updating the snapshot is just a step that makes it explicit in code instead of implicit.

Are you referring to how the nixpkgs-unstable branch hasn't been updated in the past five days? Or do you have some specific software in mind? (not arguing, just curious)

as if other sandboxing software is perfect

If you're in a situation where you want reproducibility you're using nix to build your own packages anyways, not relying on their packages

It's not if you can pin the package. It gives you reproducable docker containers without having to rebuild the world. Wasn't that the entire question?

> the old snapshot has security holes attackers know how to exploit.

So is running `docker build` and the `RUN apt update` line doing a cache hit, except the latter is silent.

The problem solved by pinning to the snapshot is not to magically be secure, it's knowing what a given image is made of so you can trivially assert which ones are safe and which ones aren't.

In both cases you have to rebuild an image anyway so updating the snapshot is just a step that makes it explicit in code instead of implicit.

Chainguard, Docker Inc’s DHI etc. There’s a whole industry for this.

Tue, Apr 21, 2026
3-minute read

As a follow-up to the similar milestone reached for our WSL image a few months ago, I’m happy to share that Arch Linux now has a bit-for-bit reproducible Docker image!

This bit-for-bit reproducible image is distributed under a new “repro” tag.
The reason for this is due to one noticeable caveat: to ensure reproducibility, the pacman keys have to be stripped from the image, meaning that pacman is not usable out of the box in this image. While waiting to find a suitable solution to this technical constraint, we are therefore providing this reproducible image under a dedicated tag as a first milestone.

In practice, that means that users will need to (re)generate the pacman keyring in the container before being able to install and update packages via pacman, by running: pacman-key --init && pacman-key --populate archlinux (whether interactively at first start or from a RUN statement in a Dockerfile if using this image as base).
Distrobox users can run this as a pre-init hook: distrobox create -n arch-repro -i docker.io/archlinux/archlinux:repro --pre-init-hooks "pacman-key --init && pacman-key --populate archlinux"

The bit-for-bit reproducibility of the image is confirmed by digest equality across builds (via podman inspect --format '{{.Digest}}' <image>) and by using diffoci to compare builds.
Documentation to reproduce this Docker image is available here.

Building the base rootFS for the Docker image in a deterministic way was the main challenge, but it reuses the same process as for our WSL image (as both share the same rootFS build system).

The main Docker-specific adjustments include (see also the related diffoci reports):

Set SOURCE_DATE_EPOCH and honor it in the org.opencontainers.image.created LABEL in the Dockerfile

TYPE NAME INPUT-0 INPUT-1 Cfg ctx:/config/config ? ?
Remove the ldconfig auxiliary cache file (which introduces non-determinism) from the built image in the Dockerfile:

TYPE NAME INPUT-0 INPUT-1 File var/cache/ldconfig/aux-cache 656b08db599dbbd9eb0ec663172392023285ed6598f74a55326a3d95cdd5f5d0 ffee92304701425a85c2aff3ade5668e64bf0cc381cfe0a5cd3c0f4935114195
Normalize timestamps during docker build / podman build using the --source-date-epoch=$SOURCE_DATE_EPOCH and --rewrite-timestamp options:

TYPE NAME INPUT-0 INPUT-1 File etc/ 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC File etc/ld.so.cache 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC File etc/os-release 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC File sys/ 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC File var/cache/ 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC File var/cache/ldconfig/ 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC File proc/ 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC File dev/ 2026-03-31 07:57:46 +0000 UTC 2026-03-31 07:59:21 +0000 UTC

You can check the related change set in our archlinux-docker repository for more details.
Thanks to Mark for his help on that front!

This represents yet another meaningful achievement regarding our general “reproducible builds” efforts and I’m already looking forward to the next step! 🤗

For what it’s worth, I’m eventually considering setting up a rebuilder for this Docker image (as well as for the WSL image and future eventual reproducible images) on my server in order to periodically / automatically rebuild the latest image available, verify it’s reproducibility status and share build logs / results publicly somewhere (if I find the time to get to it 👼).

Hacker Times

Hacker Times

Arch Linux Now Has a Bit-for-Bit Reproducible Docker Image

Discussion

Discussion