Every Go and Rust pipeline I run starts the same way, and for a long time the first thing each job did was waste a minute of my life. Before it could lint or test or scan, it had to go and fetch the tools to do those things: cargo binstall for nextest, cargo-deny, cargo-audit and llvm-cov, a go install for govulncheck, and so on. Every job. Every push. Every branch. Every Renovate bump.

On a hosted runner you’d barely notice. On mine you very much do, because mine is a single self-hosted box in the corner doing all of this with a small concurrency limit. That’s the budget I work within, and it changes the maths. A minute of tool-fetching isn’t a minute, it’s a minute multiplied by every job in every pipeline, all contending for the same runner, all doing the identical download-and-compile that the job before it just did and threw away. It was, comfortably, the single biggest recurring time sink in the whole setup, and none of it was the actual work.

The fix is the obvious one once you stop treating each job as an island: bake the tools into the image. There’s now a dev-tools image with the whole kit already installed, and the Go and Rust components default to it. The per-job installs are gone. A job starts and the tools are simply there. It’s the largest recurring-time win I’ve made on that runner, and it’s not close.

Here’s the part worth slowing down for, though, because it’s where I’d grab your sleeve if you were about to go and copy this: what you can’t bake.

You’d think you’d just throw everything in the image and be done. But there’s a line, and it runs between binaries and toolchain components. A tool like nextest or govulncheck is a standalone binary: it doesn’t care which Rust or Go toolchain the project uses, so baking it in is free and correct. A rustup component, though, like clippy or rustfmt, is tied to a specific toolchain. And a consumer’s repo can pin its own toolchain with a rust-toolchain.toml, which silently overrides whatever toolchain the image was built against. Bake clippy for toolchain A into the image, let a project ask for toolchain B, and you’ve got a clippy that doesn’t match the compiler it’s checking against. So rustup component add deliberately stays at runtime. You bake what’s toolchain-independent and you leave what isn’t. The rule is small but it’s the difference between a cache that helps and a cache that lies.

Which brings me to the bit I got wrong, because I always get one bit wrong.

The first cut of the image, v0.1.0, ran as a non-root user. That’s the textbook call. You don’t run containers as root if you can avoid it, every hardening guide says so, and I’d internalised it well enough to do it without thinking. It promptly broke every Go and Rust job. Permission-denied reading the module cache. Permission-denied on /etc/machine-id. The works.

The cause is one of those collisions between two reasonable things. GitLab’s Docker executor mounts and owns its caches in a particular way, and that ownership assumes the job can write where it expects to. A non-root user in the image doesn’t line up with how those cache volumes are owned, so the “secure” default walks straight into permission errors on the exact paths the build needs. v0.1.1 reverts the image to root.

I don’t love writing that sentence. Reverting to root as a “fix” reads like giving up on a security control. But the reality is that the security posture of a build image and the ownership model of the executor’s caches are two separate systems, and on a self-hosted GitLab runner the second one wins whether you like it or not. The textbook default was textbook for a context that isn’t this one. Better to know exactly why root is required here, and write it down, than to ship a non-root image that fails every pipeline and call it secure.

Bake the shared work in once and you stop paying for it on every job, which claws back more runner time than any other single change I’ve made. Just don’t get carried away and bake in everything, because a toolchain component isn’t a binary, and a hardened default stops being free the moment it fights the executor underneath it. Both of those I found out by breaking my own pipelines, which is, I’m reliably told, the traditional way.