The bill for the runner fleet came to $270, which doesn’t sound like a lot of money until it’s money you haven’t got.

Yesterday’s post was about building that fleet, the one that costs nothing when nobody’s pushing. That was true! It’s just that I’d been pushing quite a lot, and by the time the bill landed the fleet was already switched off. It’s still off now, because money’s tight, and the whole estate went back to the one homelab runner with its three slots.

The two easy answers

When a bill like that arrives there are two obvious ways out, and I looked hard at both.

The first is to spend more. GitLab Premium comes with merge trains and a big pile of extra compute minutes, and I spent a morning working out what it would really buy me (beyond a lighter wallet). I resisted the urge to open my wallet any further than I already had. It isn’t off the table for good, but I’m doing my best to avoid it if I can.

The second is to test less. Every pipeline is a stack of jobs, and some of them are big: mutation testing, fuzzing, cross-architecture builds. Pull a few of those off the merge requests, run them later on main or not at all, and the queue gets shorter overnight.

That one I wasn’t having. I would rather pay in waiting time than drop a single test. Quality beats quantity every time, so fast would be nice, but right matters more, and if the pipelines had to get cheaper they’d have to find it somewhere else.

Reading the receipt

So rather than guess at what to cut, I went through the receipt line by line. Three weeks of merge-request pipelines, 3,373 of them, 31,247 jobs, every minute accounted for (it’s all written up in cicd’s spec 0079 if you fancy the full table).

And my hunch about where the money went was wrong. The scary heavy jobs (mutation, fuzz, cross-arch) came to 5.7% of merge-request minutes, which is next to nothing. The ordinary Go gate, test and lint and the vulnerability check, was 40.6%, at about a minute a go, run two thousand-odd times, and that’s a price I’m happy to pay on every push.

The real cost was elsewhere. Renovate’s branches were nearly half of all merge-request minutes, and the single biggest line on the whole receipt was Renovate bumping the version pins on the cicd components, at 23.4% on its own. Release merge requests were another 20.4%, and that one was entirely self-inflicted: I’d put CHANGELOG.md into the test jobs’ change rules back in July, so a release that touched nothing but the changelog woke the whole suite up to test a file of markdown!

None of that is a test I’d miss. The component pins went (everything now tracks the components’ main branch, so there’s nothing to bump), and the changelog came out of the rules, because under colophon a release merge request is main’s tree plus the files colophon writes, and every commit in main’s tree has already been through a merge request that ran the lot.

Paying twice for the same answer

The heavy jobs were where it got interesting, because the tidy move was right there: run mutation, fuzz and cross-arch on main instead of on every merge request. Same tests, fewer runs.

I said no to that, quite firmly. Those jobs are at their most valuable while a change is still reviewable. A mutation test that fails on main tells you something has already merged, which is a bit like a smoke alarm that only goes off once the fire brigade’s left. On main it’s effectively too late.

Now, looking at it that way round showed up where the waste really was, and it was on the other side. These repositories merge fast-forward with the pipeline gate on, so the merge-request pipeline has already run on the exact commit that lands. Then main ran the same heavy jobs again, on the same tree, to produce the same answer. Over those three weeks that rerun cost 1,075 runner minutes, slightly more than the 1,057 the jobs cost on the merge requests in the first place. I was paying twice for every answer, and the second payment bought nothing.

So in the repositories that carry them, mutation and cross-arch now run on merge requests only. Fuzzing is the exception, because it isn’t deterministic: every run explores new inputs, so another run is more fuzzing rather than a repeat. It keeps its merge-request run, and instead of an accidental second helping on main it got a nightly schedule of its own at four times the depth, so a quiet week gets fuzzed just as hard as a busy one. And the coverage number, which I wasn’t going to give up either, stopped needing a test run of its own. It now reads the results the test job already produced, for both Go and Rust.

The preference I nearly gave up

The bill made me reopen something I’d treated as settled for a long time. I merge fast-forward everywhere, and I’m stubborn about it, but every rebase reruns a pipeline, so I asked what switching to ordinary merge commits would save.

About 1%. Once the other changes were in, median pipelines per merge request was still one, and reruns were a percent of the minutes. The churn I’d been blaming fast-forward for had mostly been fixed somewhere else already (which is kinda embarrassing, given how ready I’d been to blame it), and switching would have thrown away the linear history colophon builds its release notes from. So it stayed.

What it came to

On the numbers from that window, the changes add up to something like 40% fewer merge-request minutes, and not one of them is a test I stopped running. The pipelines are all the better for it, and the fleet’s still switched off.

It turns out most of what was on that receipt was things I’d bought twice… which, for a self-confessed cheapskate, stings a bit more than the $270 did.