For about a week in July my job was watching a queue.

The estate’s CI ran on one self-hosted runner: a VM with three concurrent slots and 4 GB of memory, on a Proxmox box that was also running, among other things, the desktop I work at. Everything the estate built went through those three slots. Most of the time that was plenty.

The first evening: Rust

It started, as a lot of my July did, with Rust. What I wrote to the session I was working in that evening was that Rust was proving prohibitively expensive to work with on our runners, that it took far too long to do anything, and that my suspicion was the size of the repository (one big monolith) and the fact that we did hardly any caching for Rust at all.

Both of those turned out to be right, but what hurt was more specific than “Rust is slow”. A Rust build is slow and parallel. Cargo will happily use every core it can see, and a test matrix means several of those builds exist at once, so the Rust work took all three slots and held onto them for ages.

Everything else in the estate queued behind it.

The obvious fix was a bigger runner, and that wasn’t on offer. The Proxmox host simply didn’t have the memory to spare with everything else it was running (me included), so a resize went on the list for another day and the job became making do with what there was.

So the first fix was to make Rust slower… deliberately!

A resource_group on the Rust compile jobs means they run one at a time, which leaves two slots free for everything else. I could live with Rust waiting if everything else got unblocked. What I did check before agreeing to it was the blast radius: a resource group is scoped to its project, so a similarly-named group somewhere else wasn’t suddenly going to start holding up jobs that had nothing to do with it.

Later that week, two things at once

Two days later I did two things on the same morning, each sensible on its own.

The first was to ask for a full architectural review of the Go toolkit, fifty-one modules that had recently been extracted out of go-tool-base. It came back with 116 findings, twenty-one of them critical or high, and by the end of the morning those twenty-one had been triaged into nineteen specs and agents were working through them, each fix a branch, a test, a merge request and a pipeline.

The other, a quarter of an hour before that, was proposing that the Rust monolith be split up, because the last round of compilation tests had downloaded more than 170 GB and I was fed up of cleaning it out! That’s a post of its own, but the bit that matters here is that splitting a monolith into separate repositories means each new repository arrives with its own pipeline.

By late morning I was typing that we needed to find a way to speed up these Rust pipelines, because they were clogging the limited runners we had: three slots, 4 GB, not much CPU. By lunchtime I’d settled into the question I’d be asking for the rest of the week: how are we looking?

When code is written at the speed a person thinks, three slots is plenty. The work arrives slowly and the runner sits idle more often than not. Point a set of agents at twenty-one findings and the arrival rate changes completely.

What was in short supply stopped being the ability to produce a change and became the capacity to verify one. The agents’ own reports started reading like queue reports. One, days later, stood down with its merge request still queued behind “the busy shared homelab runner”, everything green locally and nothing to do but wait.

The next morning: seven hours

The dependency updater ran overnight, at three in the morning, and opened its usual pile of merge requests into a queue that was already full. First thing, I asked the session why some of them hadn’t merged, and part of the answer came back as a line I’ve kept: with concurrent = 3, forty-three pipelines were still queued or running about seven hours later.

Seven hours would have been a crisis!

Thankfully it was nothing of the sort. The session hadn’t looked at a clock… it had the time the pipeline started and guessed the rest. It was half past five, the run had started a little after three, and two and a half hours for forty-three pipelines on three slots is about what you’d expect. (I corrected it at the time with a timezone explanation that was out by an hour itself, so neither of us covered ourselves in glory that morning.)

The fixes that came out of that morning were both about giving the queue less to do. The dependency updater moved to midnight, so its backlog had the whole night to clear instead of the last hour of it. And the security scanners, the next slowest thing in the queue after Rust, stopped each downloading their own vulnerability databases on each run: they read them from a shared cache on the runner instead, refreshed on a schedule, and built so that a project using the same CI components without a runner cache still gets a working scan.

And then it ran out

It bought a few days.

The following week the homelab hit its ceiling for good. What I wrote then was blunter: the runner situation was getting to be hard work, and the homelab was straight out of capacity. The overflow went to AWS, in a fleet that scales to zero, and this week’s evidence went straight into it. The workers’ disks were sized from it (a proposed 50 GB was out the moment I pictured a single Rust job running on it), and the caching had to be carried across deliberately, or the fleet would have rebuilt the very bottleneck it was meant to relieve.

That isn’t quite where the story ends, though…

At the start of September the fleet was switched off after a $270 bill, because the cost was prohibitive for what it was doing. So the estate is back on one runner with three slots, and the work since has gone into having fewer pipelines rather than faster ones: capping how many dependency merge requests can be open per repository at once, and trimming what runs on each one, since with fast-forward merges each merge sends the other open requests back for a rebase and another full run.

Three slots, again, and a good deal fewer pipelines to put through them.