It started with me tidying up, which is how a fair few of these things start. I was waiting on a pipeline and idly poking about in the container registry for one of the CI images, and there were forty tags in it, nearly all of them sha- something, one per commit to main, all of which should have been identical to each other and none of which anything was actually using.

So, a bit of a cleanup then. And while I was in there, a thought I’d had before and never chased: if every commit to main builds and pushes an image, and a tag pipeline then builds the same tree again, why not just retag the one we already had? That’s only safe if two builds of one commit come out the same, and I had a vague memory that kaniko had a --reproducible flag for more or less this.

I asked a model, and then I read the README

I asked Gemini what it took to get a consistent digest out of kaniko and it came back with a perfectly sensible list… timestamps, layer caching, the usual. It read like advice about a tool that was alive and well.

Then I went to the kaniko repository to check a flag name and the banner across the top said the project had been archived in June 2025. Not deprecated, not “in maintenance mode”… archived, code kept for historic purposes, more than a year before I was sat there reading it.

And all eight of my image repositories were pinned to v1.23.2, which was cut in July 2024. Fourteen months old, one release behind a tool that had no more releases to give, and nothing was ever going to fix a CVE in it.

Well then.

I’d love to tell you I felt clever for checking. Mostly I felt a bit like a man who’d been asking a very confident stranger for directions to a shop that shut down last year. The model wasn’t wrong about anything it said, it just had no way of telling me the one thing I most needed to know, because a summary of an archived tool reads just like a summary of a live one. I’ve a standing rule about checking that whatever I’m reading is current, and I’d applied it to the docs and the flags and not, that afternoon, to the tool itself (silly old me).

So the question kinda changed under me. Reproducibility could wait a bit, and the builder had to go first.

Choosing a replacement without inviting a daemon in

The shortlist was BuildKit, Buildah, apko and ko, and the constraint I cared about most was the one kaniko had been chosen for in the first place: no Docker daemon. I’d rather not run docker:dind as a service on every image build, and I’d rather not depend on a mounted host socket either, on a runner fleet that may or may not be in AWS next month.

ko went first (it builds a single Go application, and these images are a toolchain, so it’s the wrong category, not a compromise). apko was the tantalising one: Chainguard’s tool, reproducible by design, and ci-base already derives from a Wolfi image, so the ecosystem matched. But apko has no RUN. It composes apk packages and nothing else, and every one of these images curls a release asset, or compiles govulncheck at build time, or unpacks a vendor tarball. Moving to it would have meant packaging Go, golangci-lint, goreleaser, syft and the ONNX runtime myself with melange, and then looking after those packages for ever. Lovely in principle, but not this month.

BuildKit is maintained, it’s already on the runner, and it reads the existing Dockerfiles unchanged, but docker buildx wants a daemon, and the daemonless rootless flavour is a different thing again that I didn’t get as far as trying. Buildah, on the other hand, runs as the job image with no daemon at all, which is just the shape kaniko had. And the objection you usually hear (that Buildah inside a container falls back to the vfs storage driver and crawls) didn’t happen: given /dev/fuse and two security_opt relaxations it used overlay and built the biggest image unprivileged in about a hundred seconds on my workstation, within noise of kaniko.

Buildah it was, then.

And, because I’d had my nose in the registry all afternoon, I had one more question. go-tools was 1.8 GB. How on earth did it get so big! (I said something a good deal ruder than that at the time.)

The builder was never the problem

Now, before switching anything I wanted to know what reproducibility would actually buy me, so I measured it, and this turned out to be the interesting bit.

I built go-tools twice with kaniko and its --reproducible flag: fifteen of twenty layers came out identical. Then twice with BuildKit, SOURCE_DATE_EPOCH pinned and timestamps rewritten, no cache: sixteen of twenty-one. And the layers that differed were the same five under both builders. Two tools with different implementations and different timestamp handling, producing the identical set of non-reproducible layers.

Whatever was going on, it wasn’t the builder, and that left the Dockerfile.

So I went looking in the layers.

Each of those five layers ran a tool that left something behind. Go’s telemetry writes counter files with the build date in the filename, so even the final sanity-check RUN that only prints version numbers changed the filesystem. ldconfig keeps a cache keyed on library mtimes. And the last one took diffing a layer file by file to find: one entry out of 21,288, a 188-byte file under $GOPATH/pkg/sumdb holding the checksum database’s signed tree head, which advances as the transparency log grows, so every build fetched a slightly different one. go clean -modcache doesn’t touch it. GOTELEMETRY=off doesn’t work either, though go telemetry off does, and I’m not sure I’d have believed that if I hadn’t watched it.

Two Dockerfile lines and three flags, and two uncached builds minutes apart came out with the same manifest digest, byte for byte. The images were already deterministic in what they shipped.

What needed removing was litter.

Seven kinds of litter

That was one image, though. There are eight, and rolling the same scrub out across the other seven is where the tidy story got its corners knocked off. The spec on the wiki of the cicd project records four things the spike got wrong, and I’ve written them down because each one cost me a rebuild to rediscover.

The first is that three causes on go-tools turned out to be seven across the set, and hardly any of them carried over from one image to the next. Python’s .pyc files embed the source mtime pip set at install time (536 of them in docs-tools). Node keeps a V8 compile cache under /tmp. tflint caches a version check, sigstore caches TUF metadata, playwright’s Chromium leaves fontconfig caches behind. All of them were a tool caching its own state next to the content, and release-tools needed no scrub at all, which is the useful counter-example: applying one image’s scrub to another would have been cargo-culting.

The second is that a digest comparison can lie to you in two ways, and both cost me a rebuild. Chromium leaves a dotfile in /tmp, and rm -rf /tmp/* does not match dotfiles, so the scrub that worked everywhere else sailed straight past it. And rust-tools had a 634 MB layer differ with zero differing file contents across nine thousand entries… one leftover mktemp directory with a random name, which changed the order of the tar entries and nothing else. A content-level diff reports nothing and looks like agreement, so it’s tar entries you want to be diffing, not just files.

The third is the trap, and it’s a good one. Buildah defaults to the OCI image format, and OCI has no SHELL instruction. All eight Dockerfiles set SHELL ["/bin/bash", "-o", "pipefail", "-c"], and under the default format Buildah throws it away with a warning and carries on. A piped RUN silently stops failing. I measured it on a two-line Dockerfile, RUN false | true: exit 0 under oci, exit 1 under docker. That shipped in ci-base v0.2.0 before anyone noticed, because all the checks I had looked at the build’s output (digest stability, crane validate, hadolint, the scan handoff) and a discarded SHELL changes none of them. The warning was sat there in plain text in the build log, and nothing read it. --format docker is mandatory now, with a comment that says why.

The fourth is that it’s slower.

The spike had Buildah at a hundred seconds on my workstation against kaniko’s 116 on CI, and I let myself read that as a win. Measured properly on the runner, last kaniko build against first Buildah build, it’s slower on all eight images, by 14% on ci-base and 67% on node-tools, with go-tools going from 111 seconds to 179. The switch still stands on its own argument (an archived builder, a fourteen-month-old pin, no daemon) but it’s a cost, not a saving, and every repo now runs a reproducible job that builds the tree twice and compares digests, so a merge request costs roughly three builds where it used to cost one. I’ve kept that job off the nightly on purpose.

What “reproducible” is allowed to mean

Now I want to be a tad careful with the word, because the claim I can make is narrower than the flag suggests.

Two builds of one commit, minutes apart, on the same runner, are bit-identical. That’s the property I was chasing, because it’s what makes a release the artefact that was scanned a few minutes earlier, and not a fresh build that merely resembles it. Two builds a month apart may still differ, because apk add build-base resolves against a rolling Wolfi repository and I’ve left it that way on purpose; pinning it matters for reproducibility across time and not at all for the thing I’m asserting. And anything keyed on the calendar date, not the clock, passes the gate in both builds and is invisible to it.

So run-to-run, then, and not across the years. I suspect that’s still enough to turn the retag idea I started with into a decision about saving six to eleven minutes per release instead of a correctness question, and that’s a much nicer kind of decision to have.

The forty sha- tags are on a retention rule now. I never did get back to the tidying.