It was a Wednesday afternoon, I had a queue of merge requests crawling through the runners, and I’d used the wait to send a session off on a sidebar. GitLab had announced new rate limits for the free tier, and I had a nasty feeling they might be what finally pushed me onto a paid plan.

The report came back with the rate limits in it, as asked. Above them, though, was a heading I hadn’t asked for: two security findings that needed me first. The second one said my release token had been used from machines in AWS London, and that nothing in the estate could explain it.

Five names in the visitors’ book

A bit of context on that token. colophon uses it to cut releases, and it can push tags and create releases on any repository I own. If somebody else had it, they could ship a version of any of my tools with whatever they liked inside it, and the signing and checksums downstream would happily vouch for it, because it would have come from the right place.

GitLab keeps a little record of where each personal access token has been used, and it works a lot like a visitors’ book. Each new address that shows up with the token gets its name written in. This one had five: my homelab runner, which was fine, and four EC2 addresses in eu-west-2 that I didn’t recognise.

My first thought was the runner fleet, because that lives in AWS London and hands each job to a fresh throwaway machine with its own public address. But I’d switched the fleet off at the start of September, and the token had last been used that afternoon. What I typed back was roughly “thats concerning… none of the aws fleet should be running right now”, followed very quickly by the decision to revoke and rotate the lot.

Changing the locks

So that’s what the next hour looked like (nothing focuses the mind quite like the word “unexplained” next to a release token). Four tokens rotated, the other sessions in the estate warned that their runners might fall over for a bit, and the sensitive CI variables deleted and recreated as hidden (GitLab won’t let you hide an existing variable, so it’s delete and recreate or nothing). The old releaser-pleaser token got deleted outright while I was at it, since it shouldn’t have been in use any more.

The infra session checked all five of my AWS accounts, read-only, and came back with zeros: no instances, no autoscaling groups, no network interfaces, and nothing launched since the fleet was stopped on the 3rd of September. Then came the sentence I least wanted to read that day: none of the five addresses is ours.

A sweep for tampering found nothing either. No extra members, no webhooks, no deploy tokens I hadn’t made, and all 115 recent tags matched a pipeline that should have made them. That’s good news, but it doesn’t tell you much when what you’re actually afraid of is someone sitting on a token they haven’t used yet.

There was a brief detour in the middle where I decided to just pay up for Premium, and then spent ten minutes hunting for a promo code because $348 a year “will bite hard” (the cheapskate in me never sleeps, even mid-incident). I didn’t buy it in the end. The thing Premium seemed to offer turned out to be available on Free.

Everything ran as me

That rate-limit report had been saying something more important than the security findings, and in all the excitement I’d skimmed past it. The whole estate was one user. The group had one member, all four tokens belonged to me, and the 2,500 jobs the report sampled on the runner had all been triggered by me. Renovate, colophon, the blog’s own jobs and ten-odd interactive sessions were all drawing on one identity, and one identity’s rate limit… and one identity’s blast radius.

Worse, a lot of it was implicit. Tools like goreleaser and glab read GITLAB_TOKEN from the environment if they find it there, so a variable with that name sitting in CI was being picked up by things I had never knowingly wired to it. My note back to the session was to flip implicit use to explicit wherever possible, and where it wasn’t, to move to job tokens instead.

So that became the plan. GitLab’s CI/CD job token comes first: it exists for the length of a job and is gone afterwards, and these days it can do a surprising amount, including pushing to another project that has allowlisted yours. Where the job token can’t reach (opening merge requests, the issues API, tags that need to start a pipeline of their own) I went for a service account instead, each with one job and one hidden, protected variable, passed explicitly to the single command that needs it. Service accounts are on the free tier, up to a hundred per top-level group, so the $348 stayed in my pocket.

By early evening there were seven of them live and nearly every schedule had moved across. The next morning I turned on two-factor authentication for the group too, which I really should have done years ago!

The book had no dates in it

Not long after five, the session came back and corrected itself.

GitLab’s documentation says the last_used_ips field “lists up to five unique IP addresses that have authenticated with this token”, and that “when the limit is reached, the oldest IP address is removed.” And that’s all it is… names in the book, with no dates beside them. The token had last been used that afternoon from my homelab, and the four EC2 addresses could have signed in at any time at all.

The infra session matched them (along with five more that had appeared on two of my other tokens) against AWS Config’s history, which does keep dates, and all nine were workers from my own fleet, launched between the 29th of August and the 3rd of September. That’s when the fleet was running colophon’s publish jobs, carrying that very token. The incident was closed a little after half five. No compromise.

The session owned the misreading straight away, and fair enough, it had taken a list and read it as a timeline. But I’d taken its “used today” on trust just as quickly. I didn’t ask how it knew the date, and I’m normally the first one to ask that (there’s a version of me who would have typed “wait… how do we know that was today?” and saved everybody two hours).

The locks were worth changing

The next morning I was halfway through rotating the crates.io tokens when it finally dawned on me to ask… if it wasn’t a breach, why was I still rotating anything? The session agreed. Nothing was known to have left my control, so what mattered was that the secrets were hidden and protected now, and they were.

The rotation reflex had kept running a full day after the reason for it had gone, and I suspect that’s kinda how it goes when you’ve had a fright. You keep bolting doors long after you’ve found out the noise downstairs was the cat.

What I don’t regret is the rest of it. The first night on the service accounts, 47 of 48 scheduled pipelines passed, and the failure was a real vulnerability finding doing its job. My only active personal token now is for local development. The secrets are hidden, CI doesn’t run as me any more, and when the rate limits that started all of this came in on the 19th of October, each service account had a budget of its own to draw on (that’s GitLab’s documented behaviour, anyway, and I’m keeping an eye on the graphs).

So a false alarm fixed a real problem, and it took a visitors’ book with no dates in it to make me notice that all the guests had been signing in under my name.