Featured image of post The move that stopped everything at once

The move that stopped everything at once

I moved my private projects to a new GitLab group in one go while my agents were out of quota. One missing runner stopped the lot, and hid what else had broken.

My Claude sessions ran out of quota on the 5th of November, which left me with a quiet estate and nobody to talk to until it reset. So, naturally, I picked that moment to move house.

All nineteen of my private projects went into a brand new GitLab group in one go. When the sessions came back the next morning, the first thing I typed to them was that the estate had shifted massively in the last 24 hours… “let me explain!”

Why move at all

For years everything I’ve built has lived under one GitLab group, phpboyscout, the open source stuff and the private stuff all mixed in together. It worked well enough, but it was getting a bit untidy, and I’d wanted a cleaner line between the two for a while.

There was a more practical reason too. GitLab runs an open source programme that hands open source projects its top tier for free, and the catch (a perfectly fair one) is that the namespace has to be open source through and through. My open source work could really use that kind of support… and the private projects were stood right in the way of it.

So they got a new top-level group of their own, and the open source estate stayed where it was. All of it on bonfire night, which in hindsight feels about right.

Picking moving day

The timing was opportunistic, and kinda perfect. With the sessions all off the clock no work was half-finished, so no agent was going to have a repo vanish from under it halfway through a merge request. And it was only the private projects moving, so if it all went horribly wrong I was the only one who’d get hurt.

I wasn’t expecting it to go smoothly either. CI and hud (the dashboard I run the sessions from) had both been built on the assumption there’d only ever be one group, so I fully expected them to fall over. That was fine. The plan was to move everything, get the estate back on its feet, and let it sort out its own mess.

Unpacking the boxes

Once I’d done the explaining, the first job I gave the sessions was a read-only audit of everything that had moved… look, but don’t touch. Forty minutes later it landed, and it made grim reading. Nineteen projects: four red, fourteen not green since before the move, one with no CI at all, and not a single green pipeline between them since.

Then I got into the detail, and it was a lot less grim than it looked. All of those failures were the same failure! Each job had sat in the queue for hours until GitLab gave up on it with stuck_pending_no_matching_runners, which is GitLab’s polite way of saying it waited and waited and no runner ever came along to collect it. Shared runners were switched off on all nineteen projects, and my own runners (the ones in my homelab) were registered to the old group. The new group had no runners of its own.

So all the breakages I’d been bracing for (CI variables left behind in the old group, cloud trust policies still pointing at the old paths, tokens that only knew the old address) hadn’t happened yet. They were stood in the queue behind a missing runner, patiently waiting their turn to fail. My gut says that if the runners had come across cleanly I’d have spent that morning on a dozen different fires… instead I had one, with a dozen more lined up behind it.

The box with the kettle in it

Anyone who’s moved house knows to find the kettle first, before the bed, before the curtains, before the box marked “kitchen misc”. Without a brew you’re not unpacking anything. The runners were my kettle.

A runner registers to one group, so serving two means a second runner entry on each machine. We settled on one for all of them, the kids’ Windows PCs included (they’re the estate’s Windows build farm, whether the kids like it or not), and split the capacity between the open source side and the private one. The main runner went first and the kids’ PCs were left for the next day. Once jobs could start, the failures the audit had predicted came out of their boxes right on cue, and we worked down the snag list: credentials recreated for the new group, the infrastructure re-applied with the new paths in its trust policies, and Renovate handed over to the handyman (the session that does the estate’s odd jobs).

The blog was one of the first back on its feet, and it was the daftest mess of the lot. Twelve announcement pipelines had stacked up in its queue, each one primed to fire off social posts the moment a runner appeared… some of them for a post that hadn’t even been deployed yet! We cancelled the lot, ran one deploy and one sweep by hand, and let the next hourly run carry on from there. Green by 07:28.

Broken, and keeping quiet about it

While all that was going on, a few things had been broken for hours without a single pipeline going red to tell me.

My release artifacts are signed, and the index tools use to find them carries an expiry date, so a stale copy gets refused instead of trusted. A pipeline keeps renewing it. No runners, no renewal… and at 06:08 that morning, nine minutes after I’d typed “we are back”, it expired. From then on anything reading it through go/artifacts would refuse it outright. Renovate had gone quiet too, and not only for the private projects: the bot itself lives in a project that moved, so the open source estate lost its dependency updates as well. And hud, bless it, had gone completely blind to everything that moved, since the webhooks it listens for were only ever set up on the old group.

Written down, it sounds a lot scarier than it was. I’m prolific with projects, but as far as I know not many people are depending on them yet (and I’m fine with that), which gives me room for a day like this. It was all put right fairly quickly. I’ll be eyeing an all-green dashboard a little more suspiciously for a while, though.

Get the lights on, tidy later

The whole day was planned around one idea: get the estate running again first, because a running estate can organise its own repairs a lot faster than I can by hand. Anything that could wait, waited. When splitting up some shared credentials came up mid-morning, I told the coordinating session that today was about getting the estate operational again, and it went on a ticket as a future me problem.

It wasn’t all tidy. By mid-afternoon the sessions were throwing questions at me faster than I could read them (which is why I built hud in the first place), so once things were building I paused the lot, closed them down, and brought them all back up under hud with their new scopes and tokens.

Would I do it again?

Maybe. With an agent alongside me it was massively quicker and smoother than doing it by hand would ever have been, and starting with the audit meant the runners got fixed first instead of me chasing nineteen separate failures round the houses. The quota wall made the timing hard to resist as well.

But it was a gamble I could only take because I was the only person exposed. If I ever do anything like it again it gets a proper risk assessment first, so I don’t come a cropper on whatever I didn’t think of. Hopefully it won’t come to that… but if it does, at least I know which box the kettle’s in.

Built with Hugo · Theme Stack designed by Jimmy