The problem it exists for
Every CI job runs inside a container image, and the easy thing is one big image with every tool in it. The estate had two. The development image was 977 MB to pull and 2.7 GB unpacked, carrying 2,582 scanner findings. So a job that only needed curl and jq to post a Discord message pulled 2.36 GB to do it, and a Rust repository inherited 66 serious findings, none of which came from Rust (spec 0074).
How it works
One image per kind of job
Everything starts from ci-base. It’s built on Wolfi, a minimal Linux distribution made for containers, pinned to an exact image digest, and carries little more than bash, git, curl, jq, Python and sigillum, so any job can sign what it builds. On top of that sit one image per track:
| Image | For | Carries |
|---|---|---|
| go-tools | Go lint, tests, security and releases | Go, golangci-lint, goreleaser, govulncheck, syft, and the ONNX Runtime library |
| rust-tools | Rust lint, tests, docs and security | rustup with clippy and rustfmt, cargo-nextest, llvm-cov, deny, audit, semver-checks, release-plz |
| node-tools | Svelte builds and tests | Node and npm |
| playwright-tools | browser tests | node-tools plus a headless Chromium |
| tofu-tools | OpenTofu modules | tofu, tflint with the AWS rules baked in, terraform-docs |
| docs-tools | the docs sites | zensical |
| release-tools | releases | colophon |
A job pulls the image for its track and nothing else. Most run as an ordinary user rather than root. The Go and Rust images run as root because the runner checks code out as root.
Built the same way twice
The images are built with Buildah, with timestamps fixed to the commit’s own time and the build history left out. Then a check builds the image a second time from scratch and fails if the two don’t come out identical, printing exactly what differed. Getting there meant tracking down seven kinds of litter tools leave behind (caches, telemetry files, compiled Python timestamps, stray files in /tmp) rather than blaming the builder (spec 0098).
Scanned by who can fix it
Each image is scanned twice, split by who can act on the result (spec 0099). Findings in the operating system packages fail the build, because a rebuild clears them. Findings inside upstream tools are reported but don’t fail it, because only that tool’s maintainer can fix them. docs-tools fails on both, because its Python packages re-resolve on every build, so a fix lands without anyone acting. Every image is rebuilt and rescanned every night without publishing, so a vulnerability disclosed after a release doesn’t sit unseen until the next one.
Kept current by bots that must release
Every tool version is pinned in the image’s build file with a note telling Renovate where to look for new versions, so adding a pin is all the wiring it needs (spec 0065). Patch and minor bumps merge themselves on a green pipeline after a three-day wait. Major bumps wait for a person. A bump to anything an image ships is committed as a fix, so colophon actually cuts a release for it, because an image that changed without a new version once shipped an old colophon for a day (spec 0081). The new image then reaches the component library’s defaults the same way.
Every download checked
Each tool is fetched with the checksum its upstream publishes and refused if it doesn’t match. Go is checked against go.dev’s own release index. Rust tools come in prebuilt only, through cargo-binstall, because only one of the six publishes a checksum to check against. ONNX Runtime comes from the estate’s own signed artefact channel.
One deliberate exception
ffmpeg-wasi-build is the pinned toolchain behind ffmpeg-wasi’s builds: compilers, the WebAssembly SDK and build tools baked in once, rather than installed by ten build jobs from a Linux mirror. It exists because one day in September that mirror slowed to 50 kB/s, handed back a package index with the wrong checksum and failed the pipeline. It stays on Debian, not Wolfi, because the native driver it builds promises users a minimum C library version (it runs on Ubuntu 22.04, Debian 12 and RHEL 9), and that’s a contract.
Decisions and what they cost
- Wolfi. Docker’s hardened images are built to have no shell or package manager, which a CI job needs, and Alpine’s C library can’t run several prebuilt tools (spec 0074). What it cost: the same package set is 81 MB bigger on Wolfi than on Debian, and the spec says not to sell it on size. The Debian layer alone carried 2,468 findings against Wolfi’s none. Its packages can’t be pinned to versions, because nothing tracks Wolfi’s, so a nightly freshness check stands in.
- One per track. What it cost: a fix to ci-base takes three release hops to reach a job, and nine repositories’ build setups to keep in line.
- No version manager. One was measured on tofu-tools at 23 seconds against 9, with two update bots instead of one and no checksum checks. What it cost: every tool’s checksums arrive in a different shape, handled one by one.
- Playwright in an image of its own (spec 0076). Baking it into node-tools would take that image from about 117 MB to about 780 MB for a job two projects use.
- Reproducible over fast. What it cost: builds 14% to 67% slower, and a merge request costs about three builds. Builds match from one run to the next, not across weeks, because Wolfi’s package index keeps moving.
- Retag, don’t rebuild, on release (spec 0094). Pushing, not building, was most of a release pipeline: 354 of 597 seconds on go-tools. A release now retags the image main already built and tested. Only release-tools does so far.
Proof in use
- 119 of 143 estate repositories use at least one of these images, mostly through the component library: release-tools in 105, go-tools in 102, docs-tools in 72 and rust-tools in 12.
- 55 releases across the nine images in about eight weeks.
- go-tools against the old development image: 484 MB against 958 MB to pull, and 15 serious findings against 73. rust-tools scans clean on its own.
- Compressed sizes today run from 72 MB for docs-tools and 78 MB for ci-base up to 525 MB for go-tools and 830 MB for rust-tools.
Use it when, and when not to
They’re public, so anyone can pull them, but they’re built for the estate’s own jobs and components rather than as general-purpose images, and they’re x86-64 only.
The images themselves aren’t signed and carry no software bill of materials or build provenance, and colophon’s own signature isn’t checked in release-tools, only its checksum. A checksum catches a corrupted download, not a substituted one. ffmpeg-wasi-build lags the rest: it’s still built with kaniko, without the reproducibility check or the nightly scan. The component library’s defaults also run a release or two behind the newest images, and 13 repositories still pin the retired development image directly.
Where it’s going
Retagging on release for the other eight images, with ci-base last (cicd#70), moving the Rust release component onto rust-tools (spec 0104, cicd#71), signed images with provenance (cicd#72), bringing ffmpeg-wasi-build in line with the rest (ffmpeg-wasi-build#1), and moving the last 13 repositories off the retired image (cicd#73).
The images
- docs-tools Zensical on Python, the image that builds every docs microsite in the estate.
- go-tools Go, golangci-lint, goreleaser, syft and govulncheck.
- node-tools Node and npm, used by the Svelte front ends.
- playwright-tools node-tools plus the Chromium runtime libraries, for the browser-driven tests.
- release-tools colophon on ci-base: the image every release in the estate runs on.
- rust-tools The rustup toolchain with clippy, rustfmt and llvm-tools.
- tofu-tools OpenTofu and tflint, with the AWS ruleset pre-baked so a pipeline is not fetching it on every run.
Last reviewed .