It was mid-afternoon and I had a backfill running in one window (1,284 release files across seven projects, on their way to a new home) and an infrastructure plan in the other, and the day was going well. That morning I’d put a lock on a bucket. By teatime I’d taken it off again, for good, and asked for the bit of code that put it there to be deleted so it couldn’t come back by accident.

The reasoning for the lock was sound, I’d have signed my name to it that morning, and it was still the wrong lock for this bucket. So, in order, here’s how that went.

Why there’s a bucket at all

I’d been asked which old packages to delete. The GitLab package registry has a quota, the estate was 70% of the way to it and climbing, and the sensible-sounding question was which releases could go. My answer was none of them, kinda on principle. These are open-source projects, so if a release linked to a file in 2026 then I want that link to still resolve in 2036, to the same bytes, and a limit set by someone else’s pricing page is a policy they can change, not a promise I get to make.

So the binaries moved to object storage on a domain I own, pkg.phpboyscout.uk, with the promise on the landing page in as plain a set of words as I could manage: once a file is published here, it is never changed and never removed.

Now, a promise like that wants something holding it up, and I thought I had just the thing.

The padlock

R2 will let you declare a bucket write-once, for a period or for ever, and after that nothing can overwrite or delete an object, whatever credentials it turns up with. That’s a bucket lock, and I switched it on the same morning the bucket was built, on an argument I’d written into the infrastructure spec and believed: a leaked write credential’s real danger is tampering, so make tampering impossible at the platform and stop relying on whoever holds the keys being careful. Good argument, as far as it goes.

What I hadn’t done was sit down and draw the failure cases for a bucket that can’t be corrected, and that stings a little, because it’s the sort of thing I’d nag anyone else about. I’d reached for the padlock because it answered the question I’d asked, and I’d only asked the one.

The afternoon it bit

It bit within hours, and it bit on something small. A big upload that dies halfway leaves an abandoned multipart object behind, and under the lock those can’t be removed… only a lifecycle rule gets to sweep them, days later. Frustrating! On its own I’d have lived with it. But it made me get the notepad out and do what I should have done that morning, which was write down what happens to a write-once bucket when something goes wrong, and there were three cases, and each was worse than the one before.

A half-published tag is permanent. If the release job uploads four of eight artefacts and the runner dies, the four that made it refuse a second PUT (measured, it’s a 409 ObjectLockedByBucketPolicy), so that tag can’t be finished, only abandoned and re-cut. Turning the job’s retry off doesn’t help either, because that removes the retry and not the half-published tag.

Then a corrupt artefact is permanent too. The checksum file is generated from the same bytes it certifies, so a copy that’s wrong and self-consistent passes the checks I have, and there’d be no replacing it.

And the one that decided it: an accidentally published secret can’t be removed. A credential swept into a tarball, a token inside an SBOM, and the only remedy under an indefinite lock is to destroy the bucket. All of it, to get rid of one file.

That’s when the promise came apart in my hands. “Published bytes don’t change” is what I’d said to the people downloading these things. “You may not correct anything, for any reason” is what the padlock delivered, and I hadn’t wanted that one at all. Worse, tampering was already the defended risk: colophon signs checksums.txt, and the checksum covers the file and not where it lives, so a swapped binary fails verification on the way in. A leaked secret in a public bucket has no defence at all. The lock stopped the thing I could already catch and guaranteed the thing I couldn’t.

So I dropped it. What I typed at the time, to the agent doing the infra work, was that we’d make the covenant as best we could and provide the guarantees elsewhere, and would they please take the lock rule out of the stack permanently, so it’s the intended state and not drift.

What came back

The infra side pushed back the same afternoon, and the pushback was right.

My “tampering is signature-defended and detectable” argument only ever covered tampering. A deleted object has no checksum to fail, it’s just gone, R2 has no object versioning so a delete can’t be recovered, and the release credential can delete (measured, DELETE comes back 204), because R2’s permission groups don’t separate writing from deleting. I’d measured that myself that morning, written the 204 down in the probe table, and drawn no conclusion from it. Not my finest hour.

So with the padlock off, permanence rests on the custody of one credential and on nothing the platform enforces, and that changed the plan in a way I hadn’t seen coming. The old GitLab copies of each release, the 18 GB I’d been so keen to reclaim (I’m a cheapskate, and it was showing), are staying put for now, because until there’s a second copy of the bucket they are the second copy. A scheduled sync to a second bucket whose keys the release pipeline doesn’t hold went from a nice-to-have to the thing that unblocks deleting anything. And R2 can send an event on a delete, so it does, because one delete is a stronger signal than a daily digest.

Weaker in principle than a padlock, then. In practice I suspect most of the defence had been coming from the signature all along, and what the padlock added was a way of locking my own mistakes in with the things I was protecting.

Guarantees, elsewhere

The “elsewhere” turned out to be three things with one shape between them: read the public URL, hold no credential, shout if something’s wrong.

A tag that’s already published doesn’t get republished. I’d wanted that as a conditional write, and R2 supports one (If-None-Match: *, and a repeat PUT does come back 412), and it sat in the spec for about an hour before I found that goreleaser’s blobs pipe can’t send the header. Eighteen configuration keys and not one of them is a conditional write! So the guarantee moved up a level, and the release job now checks, before it builds anything, whether this tag’s checksums.txt already exists at the public URL, and refuses to run if it does. That file is written last, which gives me the asymmetry I wanted now the lock’s gone: a completed publish is protected, while a tag that died mid-upload can be finished by re-running. A broken release gets fixed by cutting a new tag, not by republishing an old one.

Then there’s a scheduled verifier that re-reads the published files the way you would: fetch checksums.txt and its signature, verify the signature against the vendored keys, fetch each listed file and compare digests, and raise an issue on the owning project when something’s missing or different. It lives in the cicd project as a scheduled pipeline, not in colophon’s release run (a verify pass over the whole back catalogue doesn’t belong on the critical path of cutting a new release) and not in the infra stack (which describes resources and doesn’t run jobs). It holds no bucket credential, and that’s on purpose. A verifier that can write could be the cause of the damage it reports, and one that reads only what the public reads is checking what people get, not what the bucket believes it holds.

And the promise itself is stated where the people relying on it can read it, on the landing page, and not only in a spec.

Which failure would you rather explain

That’s the question a write-once anything comes down to, I think, and it’s the one I hadn’t asked that morning. A padlock is a very good answer to “how do I stop a leaked key rewriting history” and a terrible answer to “what do I do when I’m the one who published the wrong thing”, and being the person holding the keys, the second is the one I’m far more likely to be asked.

The padlock’s off and the covenant’s up, and there’s a job that reads the whole bucket every night, the same way you would, holding nothing that could hurt it.