PHP Boy Scout — CI/CD, and getting things released

CI you include, not copy

Sat, 16 May 2026 00:00:00 +0000

Every infrastructure repo runs the same CI: lint the OpenTofu, scan it, validate it, plan, apply. The first repo, you write that .gitlab-ci.yml by hand. The second, you copy it. By the third, you’ve got three copies of the same pipeline quietly drifting apart, which is the exact problem you’d never tolerate in application code. The cicd repo is the fix, and it’s just the library-first instinct pointed at the pipeline.

The `.gitlab-ci.yml` you keep copying

The infrastructure repos in this series all run the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security issues and secrets, and on the deploy side, plan and apply.

The first repo, you write that .gitlab-ci.yml by hand. The second repo needs the same jobs, so you copy it. The third repo, you copy it again. Now there are three copies of the same pipeline, and they do what copies always do. They drift. A fix you make in one repo’s CI doesn’t reach the other two. A tightened scan rule lands in the repo you were working in and nowhere else. It’s the copy-paste problem, exactly as it shows up in application code, just written in YAML and therefore that bit easier to pretend isn’t code.

GitLab has a feature for exactly this

GitLab CI/CD Components are the answer to that problem. A component is a reusable, versioned piece of pipeline that you publish, and other projects pull in with an include: pinned to a version:

include:
 - component: gitlab.com/phpboyscout/cicd/tofu-lint@v0.5.0

That’s a library import, for pipeline. The component has a defined interface, a version, and a home in GitLab’s CI/CD Catalog. A consuming repo includes it instead of carrying its own copy, and when the component improves, the consumer moves a version pin rather than re-copying YAML.

Why a monorepo of components

The cicd repo holds all of the components together: tofu-lint, tofu-security, tofu-validate, tofu-plan, tofu-apply, and more. One project, not one project per component.

That’s a deliberate call, and the reason is how GitLab versions things. A version is a tag, and a tag belongs to a project. A component’s version is its project’s tag. So a monorepo of components, versioned together as one tag stream, is the natural unit: a consumer pins @v0.5.0 and gets a known-good set of components that were tested together, rather than juggling a separate version for each one.

Authoring discipline

A component is a file under templates/, and it opens with a spec: inputs: block: the typed inputs, their defaults, the component’s public interface.

The discipline that keeps the library usable is that a component must be consumer-agnostic. It never hardcodes a token, and it never names a particular consumer’s variable. Inputs have sensible defaults, and a consuming repo overrides them. A component that reaches out and assumes something about the repo including it is a component that works in one repo and surprises the next. An authoring guide in the repo keeps that consistent across everyone who adds a component.

The self-test you cannot fully write

The cicd repo tests its own components with a self-test pipeline. It’s worth knowing where that self-test stops.

When a repo tests its own components by running them in child pipelines, GitLab masks $CI_PIPELINE_SOURCE as parent_pipeline. A component’s rules:, which often branch on the pipeline source to behave differently for a merge request than for a branch or a tag, therefore can’t be exercised honestly by the self-test: the source they’d branch on has been flattened. The self-test covers what it can, and the component rules: are, in the end, validated by real consumers using them for real. That’s a genuine limit, and naming it is better than pretending the self-test proves more than it does. (It’s also, not coincidentally, the exact rules: quirk that bit me in one of the two bugs I closed the series with.)

The same instinct, again

This blog keeps circling the same instinct. go-tool-base exists because the same CLI scaffolding kept getting rewritten, so it was extracted into a library. cicd is that instinct pointed at the pipeline: the same gate jobs kept getting copied between repos, so they were extracted into a versioned, included library.

Stop copy-pasting. Publish, version, include. It’s true for CLI code, and it turns out to be just as true for the YAML that builds and ships it.

The gist

Every infrastructure repo needs the same CI, and copying the .gitlab-ci.yml between them produces copies that drift apart. GitLab CI/CD Components fix it: reusable, versioned pipeline that a repo include:s and pins, instead of carrying its own copy.

cicd is a monorepo of those components, versioned together as one tag stream, because GitLab tags a project and a component’s version is its project’s tag. Components are authored consumer-agnostic, with typed spec: inputs: and no hardcoded assumptions, and their rules: are validated by real use because the self-test can’t see the pipeline source. It’s the library-first instinct, applied to CI: publish it once, include it everywhere, fix it in one place.

Stop installing the same tools on every pipeline job

Sun, 19 Jul 2026 00:00:00 +0000

Every Go and Rust pipeline I run starts the same way, and for a long time the first thing each job did was waste a minute of my life. Before it could lint or test or scan, it had to go and fetch the tools to do those things: cargo binstall for nextest, cargo-deny, cargo-audit and llvm-cov, a go install for govulncheck, and so on. Every job. Every push. Every branch. Every Renovate bump.

On a hosted runner you’d barely notice. On mine you very much do, because mine is a single self-hosted box in the corner doing all of this with a small concurrency limit. That’s the budget I work within, and it changes the maths. A minute of tool-fetching isn’t a minute, it’s a minute multiplied by every job in every pipeline, all contending for the same runner, all doing the identical download-and-compile that the job before it just did and threw away. It was, comfortably, the single biggest recurring time sink in the whole setup, and none of it was the actual work.

The fix is the obvious one once you stop treating each job as an island: bake the tools into the image. There’s now a dev-tools image with the whole kit already installed, and the Go and Rust components default to it. The per-job installs are gone. A job starts and the tools are simply there. It’s the largest recurring-time win I’ve made on that runner, and it’s not close.

Here’s the part worth slowing down for, though, because it’s where I’d grab your sleeve if you were about to go and copy this: what you can’t bake.

You’d think you’d just throw everything in the image and be done. But there’s a line, and it runs between binaries and toolchain components. A tool like nextest or govulncheck is a standalone binary: it doesn’t care which Rust or Go toolchain the project uses, so baking it in is free and correct. A rustup component, though, like clippy or rustfmt, is tied to a specific toolchain. And a consumer’s repo can pin its own toolchain with a rust-toolchain.toml, which silently overrides whatever toolchain the image was built against. Bake clippy for toolchain A into the image, let a project ask for toolchain B, and you’ve got a clippy that doesn’t match the compiler it’s checking against. So rustup component add deliberately stays at runtime. You bake what’s toolchain-independent and you leave what isn’t. The rule is small but it’s the difference between a cache that helps and a cache that lies.

Which brings me to the bit I got wrong, because I always get one bit wrong.

The first cut of the image, v0.1.0, ran as a non-root user. That’s the textbook call. You don’t run containers as root if you can avoid it, every hardening guide says so, and I’d internalised it well enough to do it without thinking. It promptly broke every Go and Rust job. Permission-denied reading the module cache. Permission-denied on /etc/machine-id. The works.

The cause is one of those collisions between two reasonable things. GitLab’s Docker executor mounts and owns its caches in a particular way, and that ownership assumes the job can write where it expects to. A non-root user in the image doesn’t line up with how those cache volumes are owned, so the “secure” default walks straight into permission errors on the exact paths the build needs. v0.1.1 reverts the image to root.

I don’t love writing that sentence. Reverting to root as a “fix” reads like giving up on a security control. But the reality is that the security posture of a build image and the ownership model of the executor’s caches are two separate systems, and on a self-hosted GitLab runner the second one wins whether you like it or not. The textbook default was textbook for a context that isn’t this one. Better to know exactly why root is required here, and write it down, than to ship a non-root image that fails every pipeline and call it secure.

Bake the shared work in once and you stop paying for it on every job, which claws back more runner time than any other single change I’ve made. Just don’t get carried away and bake in everything, because a toolchain component isn’t a binary, and a hardened default stops being free the moment it fights the executor underneath it. Both of those I found out by breaking my own pipelines, which is, I’m reliably told, the traditional way.

One image for the whole toolchain

Fri, 15 May 2026 00:00:00 +0000

Every CI gate job across the infrastructure repos reaches for the same pile of tools: OpenTofu, tflint, trivy, checkov, gitleaks, terraform-docs, the AWS CLI. Installing that pile per job is both slow and quietly dangerous, because nothing pins it consistently. infra-tools is the obvious fix (one image, one source of truth for versions), but two of its build decisions are less obvious and worth a look: it publishes with crane instead of a second build, and it deliberately lets its own vulnerability scan fail.

The same pile of tools, in every repo

Every infrastructure repo in this series runs the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security problems and secrets, check the docs. Those jobs need a specific set of tools, and it’s the same set in every repo.

Install them per job and you pay twice. You pay in time, because every pipeline downloads and installs the whole set again. And you pay in drift, because unless every repo pins every tool identically, the repos slowly diverge on which version of trivy or tflint they actually run, and a check that passes in one repo fails in another for no reason anyone can see.

One image, one source of truth

infra-tools is the answer: a single Debian-based container image with the whole toolchain baked in. Every CI job in every repo uses it with one image: line.

The real value isn’t the convenience. It’s that the image is the one place tool versions are pinned. The Go-based tools are pinned in a mise.toml. checkov, which has no mise plugin, is pinned in a requirements file installed with pipx. The AWS CLI is pinned by a build argument. Three mechanisms, because the tools come from three kinds of source, but one image, and every pin wired to Renovate so a version bump arrives as a reviewable pull request. There’s exactly one answer to “what version of trivy does the toolchain use”, and it lives here.

Publishing with crane, not a second build

A build-pipeline detail that took a real bug to discover.

The pipeline builds the image with kaniko, which builds images without a privileged Docker daemon, something that matters a great deal on shared CI runners. Then it scans the image, then it publishes it.

The obvious way to write the publish stage is “build the image and push it”. But kaniko has no mode for “just push this tarball I already built”. A second kaniko invocation re-executes the entire Dockerfile from the top, including a second mise install, which makes a fresh round of calls to GitHub’s API to fetch tools. GitHub’s anonymous API limit is low and shared by IP, so on a CI runner that second install reliably trips a 403 rate-limit. (Yes, another 403. They do get everywhere.)

So the publish stage doesn’t rebuild. It uses crane to push the exact image tarball the build stage already produced. The image is built once. And because the published bytes are the same bytes the scan stage scanned, there’s no gap between “the image we checked” and “the image we shipped”.

Soft-failing the scanner on purpose

The decision that looks wrong until you see the reasoning: the pipeline scans the image with trivy, and trivy is allowed to fail without failing the pipeline.

A vulnerability scanner that doesn’t gate the build sounds like a scanner switched off. It isn’t. It’s a scanner pointed at something it can’t helpfully gate.

The tools in the image are prebuilt Go binaries. trivy inspects them, reads the version of the Go runtime each was compiled with, and reports every known CVE in that Go runtime. Those findings are real, but they aren’t mine to fix. The only fix is the upstream tool rebuilding itself against a patched Go. With seven such tools in the image, at any given moment one of them is usually a little behind on its Go version.

A hard gate would mean the image becomes unpublishable whenever any single upstream lags, over a CVE in code I don’t own and can’t patch. That’s not a security control; it’s a way to be unable to ship. So the scan is allow_failure. The findings stay fully visible, and the residual count is genuinely useful as a metric for how far behind upstream the toolchain has drifted. It just doesn’t block shipping an image whose only “vulnerabilities” are other people’s build timelines.

What it comes down to

The infrastructure repos all run the same CI gate jobs, needing the same tools, so infra-tools bakes the whole toolchain into one image and pins every version in one place, wired to Renovate.

Two build choices are worth copying. The publish stage uses crane to push the already-built, already-scanned tarball, because a second kaniko build would re-run mise install and hit GitHub’s anonymous rate limit, and because pushing the scanned bytes means shipping exactly what was checked. And the trivy scan is deliberately allow_failure, because it reports Go-runtime CVEs in prebuilt upstream binaries that no change to this repo can fix, so a hard gate would only make the image unshippable over someone else’s lag.

A release is just another merge request

Fri, 31 Jul 2026 00:00:00 +0000

For a long time my releases happened by accident. Not literally, but close enough. The setup was tag-on-merge, the semantic-release way: you merge to the release branch, the tooling reads your commits, works out the next version, and a tag pops out the other end. A release happens. You didn’t do a release, exactly. You merged a thing, and a release fell out of it.

That feels great right up until you ask an awkward question: where, in that flow, do the tests run?

Because a release that’s a side-effect of a merge rides whatever pipeline that merge triggers, which is the main-branch pipeline. And your quality gates, the lint and the tests and the security scans, are then in an uncomfortable spot. Either they run on that same main-branch pipeline, which means they fire after the merge that already triggered the release, and you’re checking the work after you’ve shipped it. Or they ran back on the merge request and you’re trusting that, hoping nobody merged something red, hoping the MR pipeline and the main pipeline actually agree about what “passing” means. It’s closing the stable door a moment after the horse has bolted, been packaged, tagged, and handed to the public.

I lived with that longer than I’d like, because it mostly works. Most merges are fine. The gates mostly did run on the MR. “Mostly” is doing an awful lot of load-bearing in those sentences, though, and “mostly” is not what you want underneath the thing that decides whether unreviewed code becomes a versioned release.

The fix turned out to be a change of model, not a patch. I moved the estate to releaser-pleaser, which does releases as a Release-PR. Instead of a tag falling out of your feature merge, a bot keeps a pull request open, the one that bumps the version and assembles the changelog, and the release only happens when that PR is merged. Which sounds like a small bit of plumbing rearranged, and that’s the whole of it, because it changes what a release is. The release stops being a consequence of merging your feature and becomes its own explicit merge request.

And here’s the thing I was really after: a merge request runs the gates. My cicd templates default their rules to $CI_PIPELINE_SOURCE == "merge_request_event", so lint, test and security are things that happen to merge requests. The moment the release is a merge request, the release is gated, automatically, by the exact same checks as every other change, with no special handling at all. Nothing becomes a release without going through the door that everything else goes through. I didn’t have to bolt a gate onto the release process. I made the release a merge request and it inherited every gate I already had.

There’s a knock-on that I didn’t expect to care about as much as I do. Once the release is a proper PR off the main branch, the old develop intermediary I’d been carrying had nothing left to do, so it went, and the whole thing collapsed into trunk-based development. That’s a conversion I came to from the other direction entirely, and it has its own story, but the release model and the branching model turned out to be the same decision wearing two hats. A simpler branch structure and a gated release fell out of one change.

I rolled it across everything rather than one repo, because a delivery model that’s different in each project is its own kind of tax. Every terraform-aws-* module, cicd itself, the lot, all flipped to trunk-based with releaser-pleaser, with the reusable pieces pulled into shared cicd components so the gating lives in one place and not in twelve. (The Rust side went to release-plz, a different tool with the same Release-PR idea and a war story all its own.) The wrapping of those components, and the schedule gotcha that came with them, is a tale I’ve already told.

What it comes down to is a question I’d never quite asked out loud: should a release be allowed to skip the queue? With tag-on-merge I’d answered yes, a release was a privileged thing that happened off to the side of my normal review-and-gate flow. The Release-PR model answers no, and I think no is right. A release is a change to your project like any other. It should have to stand in the same line, pass the same checks, and earn its way to main exactly like the smallest bugfix does. Make it a merge request, and it does.

Same config, two answers

Wed, 03 Jun 2026 00:00:00 +0000

Let me confess a small heresy first, because it’s the reason any of this happened. After a career spent as a branching man, gitflow, gitlabflow, a tidy develop branch and a careful dance of merges, I’ve come round to trunk-based development. I resisted it for years. It felt like working without a net.

What changed my mind was working solo with an AI pair. The branch ceremony that earns its keep on a team of eight is just drag when it’s me and a model at two in the morning. So I’ve softened on “main is always deployable” and let the trunk act as the develop branch, with tagged releases as the actual source of truth. For compiled languages, where the artefact you ship is a built, tagged thing and not whatever’s on a server right now, that finally clicks.

I’d already rolled this out on my Go and Terraform projects with releaser-pleaser, a GitLab-native take on release-please: a bot keeps a Release MR open, and merging it cuts the tag. It’s the same model I wrote about when the infra repo moved to plan-on-merge, apply-on-tag. Lovely. Then I came to do the same for rust-tool-base, and Rust, being Rust, had opinions.

Rust brings its own toolchain

releaser-pleaser is happy to tag a repo and write a release. What it does not do is cargo publish seventeen crates to crates.io in dependency order. Rust’s release story isn’t “push a tag and let a runner build a binary”, it’s a whole publishing pipeline with a public registry at the end of it, and that registry has rules of its own. So for the Rust workspace I reached for the tool built for exactly that job: release-plz. Same Release-MR shape, but it understands cargo, versions every crate, and publishes the lot.

That was the right call. Getting it to actually do it was where I spent two days I’d quite like back.

The gauntlet before the gun

Before I got anywhere near the interesting failure, there was a run of CI papercuts, the sort where every fix politely reveals the next one. GitLab checks out a detached HEAD, and release-plz wants to be on a branch (“HEAD does not point to a branch”), so you re-attach. Then the default CI_JOB_TOKEN can’t push to a protected repo, so you point the remote at a real token. Then release-plz assumes you’re on GitHub and errors that the repo “is not hosted in GitHub”, so you tell it --forge gitlab. Then it refuses to run at all because the pages job left a public/ directory lying about and the working tree is “dirty”, so you stop pulling artefacts into the job.

Five merge requests before the thing would even start doing its actual job. You can read the scar tissue in the before_script; every line in it is a fix for something on that list. None of it was hard. It was just death by a thousand cuts, and I was feeling quite smug by the time it finally reached the publish step.

I should not have been.

“Tag v0.5.1 already exists”

My release-plz.toml asked for one tag per release:

git_tag_name = "v{{ version }}"
git_release_name = "v{{ version }}"

That felt obviously right. It matched the repo’s existing v0.5.0 tag, it’s how a single-crate project tags, and the crates all share one workspace version anyway. One version, one tag. What’s to argue with?

release-plz, that’s what. It tags per crate. So it publishes a crate, creates the tag, publishes the next crate, and tries to create the same tag again:

INFO published rtb-assets 0.5.1
ERROR failed to release package
Caused by:
 0: failed to create git tag 'v0.5.1' with ref 'f6de975a75...'
 1: Response body: { "message": "Tag v0.5.1 already exists" }
 2: HTTP status client error (400 Bad Request) ... /repository/tags

The collision is annoying. What makes it a proper trap is the half-second before it: published rtb-assets 0.5.1. That happened. On crates.io. For keeps. A crates.io publish is forever, there is no unpublish, only a yank that still leaves the name and version burned. So every time my flaky pipeline limped one crate further and then fell over on the tag, it left another crate live on the public registry that I could never take back. By the time the dust settled, six of the seventeen were out there: rtb-assets and rtb-config, then on a later retry rtb-credentials and rtb-error, then rtb-app and rtb-redact. Two more permanent crates per failed run.

I assumed the default

My first fix was the clever one, and it deserves to be on display because it’s the whole lesson in miniature. I deleted the git_tag_name line. My reasoning: per-crate tags are release-plz’s native model, so surely its default does the right thing without me spelling it out. I was confident enough to write it into the commit message: “per-crate tags/releases (release-plz defaults).”

The next run collided on v0.5.1, exactly as before.

Because release-plz’s default git_tag_name is not per-crate. It’s the unified v{{ version }}. I had deleted a line that said the wrong thing and replaced it with a default that said the same wrong thing, then congratulated myself for tidiness. If I’d spent thirty seconds on the configuration reference instead of thirty seconds being clever, I’d have read that in black and white.

Same config, two answers

So I read the manual, and set it explicitly:

git_tag_name = "{{ package }}-v{{ version }}"
git_release_name = "{{ package }} v{{ version }}"

On my laptop, a dry run rendered exactly the per-crate tags I wanted. In CI, the very next run published another crate and then created the tag v0.5.1. The unified one. The wrong one. The one I had just, demonstrably, on main, replaced. Same release-plz.toml, two completely different answers depending on who was asking.

That one took me an embarrassingly long time to see. release-plz does not read your config from the working tree when it runs a release. It reads it from the release commit, the commit that introduced the version it’s releasing. My version was still 0.5.1, set days earlier on a commit that still carried the unified template. You can see it in the failure: the tag it tries to create is pinned to ref 'f6de975...', an old commit, not the HEAD that held my fix. Every edit I made at the tip of main was real, committed, and utterly invisible to the release of 0.5.1, because no version bump had created a fresh release commit for it to read. My fix was correct and inert at the same time. The dry run read my working directory and looked perfect; CI read history and did something else.

There is no config change that rescues an in-flight release. The version was already out, half-published, tagged wrong, and pointed at a commit I couldn’t edit without bumping the version, which I couldn’t cleanly do with six crates already live.

Doing it the way I’d have done it a year ago

So I stopped. Three retries deep, each one a seventy-minute CI cycle thrown at an opaque mismatch, six crates already immovable on crates.io, and a tooling problem I now understood well enough to know the tool was never going to dig me out of this particular hole. The question quietly changed from “why is it doing this?” to “am I going to keep grinding, or finish this the way I would have before I had clever tooling?”

I went manual. cargo publish, the remaining eleven, by hand, in dependency order: the leaf crates first and the rust-tool-base umbrella dead last, because it depends on all of them. crates.io rate-limits new crate names, so after a burst it simply made me wait, a roughly half-hour pause in the middle while the registry caught its breath and I caught mine. Then one v0.5.1 tag, cut by hand, and one GitLab release to match the convention. The next CI run came up green, for the gloriously dull reason that there was nothing left to do: every crate published, the tag already there.

Stop being clever and RTFM

The tool was never broken. Every single thing it did was documented behaviour I hadn’t bothered to read: that the default tag template is unified, that the model is per-crate, that a release reads its config from the release commit and not from HEAD. I assumed my way past the manual three times in a row, and each assumption cost me real, permanent state on a public registry that doesn’t take returns.

And that’s the part that actually stung, because I should have known better than most. I wasn’t a beginner here. I knew the Release-MR pattern cold, I’d shipped it half a dozen times with releaser-pleaser on my Go and Terraform repos. That familiarity was the trap. I trusted the pattern and skipped the tool, on the lazy assumption that something I understood well in one tool would behave the same in the next. release-plz carries the same design, but it’s a different tool, with its own defaults and its own idea of where the config lives. The pattern came across fine. The mechanics didn’t, and I never thought to check.

So here’s the lesson, written down in the hope it sticks this time: no matter how familiar I am with a pattern or a design, the moment I switch the tool that implements it, reading the manual is paramount. The familiarity is exactly what tempts you to skip it, and exactly why you can’t. (The narrower, more practical one, while I’m here: a config change that affects how a release behaves has to travel with a version bump, or it sits there looking applied and doing nothing.)

release-plz is genuinely good, and every release since has gone out clean on the first try, the way the rest of the CI now does. I just had to stop being clever long enough to read how it actually works. RTFM. I’ll get it tattooed eventually.

Three traps release-plz sets for a Rust workspace

Fri, 05 Jun 2026 00:00:00 +0000

I wrote up the two days I lost releasing a seventeen-crate workspace to crates.io as a war story, wrong turns and all. This is the other half: the field guide, so you don’t have to lose the same two days.

release-plz is a genuinely good tool, and none of what follows is a bug. It’s three behaviours that are entirely within its design and will still ambush you the moment you point it at a Cargo workspace rather than a single crate. Mildest first, because the third is the one that actually ate my release.

First, what release-plz is doing

In one line: it’s release-please for cargo. It keeps a Release MR open, bumps your versions and per-crate changelogs from your Conventional Commits, and when that MR merges it publishes every crate to crates.io and tags the release. On a workspace where N crates all share one version, “the release” is N publishes and N tag operations. Hold on to that N. It’s hiding behind all three traps.

Trap 1: the default tag template is built for one crate, not a workspace

You will reach for one tag per version, and for me it was more than tidiness. I wanted to ship the whole framework as a single release: one v0.5.1 covering all seventeen crates, because that was the compatibility promise I wanted to make. Use the crates that share a version and they’re guaranteed to work together. A single tag felt like the natural way to say “this is one coherent release of the whole thing” (and it didn’t hurt that the repo already had a v0.5.0 tag from before release-plz, so one unified tag also looked like continuity). So you either set this, or, worse, you leave git_tag_name unset assuming the default does something workspace-aware:

git_tag_name = "v{{ version }}"

Here’s the catch. release-plz’s default git_tag_name is v{{ version }}, and release-plz tags per crate. So the first crate publishes and creates the tag v0.5.1. The second crate publishes and tries to create v0.5.1 again:

ERROR failed to create git tag 'v0.5.1'
 "message": "Tag v0.5.1 already exists"

By the time you read that error, the first crate (and on a retry, the next, and the next) is already live on crates.io, and crates.io publishes are forever. Leaving the line out doesn’t save you, because the default is the same single-crate-shaped template. This is the trap I walked straight into on the release commit.

Trap 2: “one release for the whole workspace” isn’t a setting, it’s a category error

The natural next thought is “fine, I’ll keep one tag but configure release-plz to roll the crates into a single release.” There’s no knob for that, and chasing one is a waste of an afternoon. release-plz’s model is per-crate all the way down: per-crate tags, per-crate GitLab/GitHub releases, per-crate changelogs. “One unified release for the whole workspace” isn’t an option it withholds, it’s a shape it doesn’t have.

So you stop fighting it and set the per-crate templates explicitly:

git_tag_name = "{{ package }}-v{{ version }}"
git_release_name = "{{ package }} v{{ version }}"

Now each crate gets its own tag (rtb-assets-v0.5.1, rtb-config-v0.5.1, and so on) and its own release. It’s more objects per version than you wanted, but it’s the grain the tool works in, and once you accept that the collisions stop.

This is where I had to pull apart two things I’d quietly merged in my head: the version and the tag. The compatibility promise I cared about, that crates sharing a version work together, is carried by the version, and release-plz keeps every crate on the one workspace version no matter how it tags them. The tag is just a label pointing at a commit. I’d wanted a single tag to mean “one coherent framework release”, but the coherence was always in the shared version number, not in the tag. Once that landed, seventeen tags stopped feeling like seventeen releases of seventeen different things and started looking like what they are: seventeen labels on one versioned release. The version is not the tag. If you still want one human-facing narrative for the whole thing, keep a hand-written root CHANGELOG.md alongside the generated per-crate ones, rather than trying to make release-plz aggregate.

Trap 3: a release reads its config from the release commit, not HEAD

This is the small one, and the one that cost me the most, because it makes the fix for Trap 1 look like it isn’t working.

When release-plz runs a release, it does not read release-plz.toml from your working tree. It reads it from the release commit, the commit that first introduced the version it’s releasing. So picture the obvious recovery: you hit the tag collision, you realise your template is wrong, you fix it in a follow-up commit and push to main. Your fix is real. It’s committed. It’s on the default branch. And it is completely ignored, because the version hasn’t changed, so the release commit release-plz reads from is still the old one with the old template.

I didn’t take this on faith. With the corrected per-crate template sitting on HEAD, the CI release job still tried to create the unified tag, pinned to the old commit:

ERROR failed to create git tag 'v0.5.1' with ref 'f6de975...'
 "message": "Tag v0.5.1 already exists"

That ref is the release commit, not the HEAD that held my fix. And the cruel part: release-plz release --dry-run on your laptop reads your working-directory config, so it renders the shiny new per-crate tags and tells you you’re sorted. CI runs the real thing against the release commit and does something else entirely. Same config file, two different answers depending on who’s asking, which is why the war story has the title it does.

The operational rule that falls out of this: any release-plz config change that affects how a release behaves has to ride along with a version bump, or it does not apply. A “fix-up” commit on its own is a no-op.

If you set one thing

If you run release-plz on a multi-crate workspace and you change a single line from the defaults, make it the tag template:

git_tag_name = "{{ package }}-v{{ version }}"

And set it before your first release, not during it, so it’s already in the commit that introduces the version, because that’s the only commit a release will ever read it from. Everything else here follows from two facts: the grain is per-crate, and CI reads history while your laptop reads your working tree. Trust the history.

None of this is release-plz misbehaving. Every bit of it is documented and deliberate. It just isn’t where you’ll think to look until it has published six crates you can’t take back, which is roughly how I came to know it so well.

The version pin you can't lower

Sun, 02 Aug 2026 00:00:00 +0000

A security fix landed in the Go standard library, the kind you don’t get to think very hard about. You bump go.mod to the patched toolchain, 1.26.4, and you move on. Or you would, if your release didn’t immediately fall over the moment you tagged it.

Here’s what greeted me when I cut the next krites release. The before-hooks, the go mod tidy and friends that run ahead of the actual build, died on the spot:

go.mod requires go >= 1.26.4 (running go 1.26.3; GOTOOLCHAIN=local)

The goreleaser image I was releasing from, a perfectly current v2.16.0, bundled Go 1.26.3. One patch version behind. And GOTOOLCHAIN=local, which is the sensible default for a release because it means “use exactly the Go that’s installed, don’t go fetching anything mid-build”, was doing precisely its job: it saw go.mod asking for a toolchain it didn’t have, and refused to pretend otherwise.

Now, the fix that’s quickest to reach for is the one you have to talk yourself out of. The error is a version mismatch, and the fastest way to make a version mismatch go away is to change one of the versions. The image has 1.26.3? Fine, drop go.mod back to 1.26.3 and the red goes green. Except think about what that actually does. The whole reason go.mod says 1.26.4 is that 1.26.4 carries a security fix I need. Lowering the pin to placate the build image doesn’t resolve the tension, it surrenders to it. It un-applies the security update to make a pipeline pass, which is about the worst trade in software, and it’s the one sitting right there with a green tick on it.

So that pin is one you can’t lower. Which leaves making the build find the right toolchain instead of making the requirement smaller.

The quick version is one flag: flip GOTOOLCHAIN from local to auto. Where local means “only ever use what’s installed”, auto means “if go.mod asks for a newer toolchain than you’ve got, go and fetch that one and use it for this build”. So the release runs on the image’s 1.26.3, hits the go.mod directive, fetches 1.26.4 and carries on. The mismatch is resolved by upgrading at build time rather than by downgrading the requirement, which is the right direction for it to resolve in.

The better version, which is where krites ended up, is to not have the gap at all. The release now runs on the dev-tools image I bake my CI tools into, and that image tracks the Go version my code actually targets, 1.26.4. When the toolchain in the image and the toolchain in go.mod are the same thing by construction, there’s nothing to fetch and no flag to set, because there’s no disagreement to paper over.

The toolchain your code requires and the toolchain your build image ships are two different facts, and they drift apart every time a point release with a security fix lands in one a good while before it reaches the other. When they disagree, the instinct to make them agree by lowering the requirement is exactly backwards: the requirement is the thing that’s right, and the environment is the thing that’s behind. The pin only ever moves one way. The build image is what had to catch up to it, not the code learning to want less.

From allow_failure to blocking

Sat, 30 May 2026 00:00:00 +0000

There’s a special kind of CI job that everyone on a team quietly learns to ignore: the one marked allow_failure: true. It runs, it goes red, the pipeline goes green anyway, and after the third time you stop looking at it. I inherited six of those when I moved rust-tool-base’s CI to GitLab. Over a few days I turned three of them into real gates, and the interesting part was never the YAML. It was working out which ones had earned the right to block, and which hadn’t.

What allow_failure actually buys you

allow_failure: true is genuinely useful, and quietly corrosive. It lets a job report a problem without stopping the pipeline, which is exactly right for a check that’s noisy, not yet stable, or guarding against something you can’t fix this minute. The trouble is that a warning nobody is forced to act on is a warning nobody acts on. Leave a job advisory long enough and it becomes scenery: red, ignored, pointless. So an advisory check is really a promise, “I’ll make this blocking once it’s trustworthy”, and a promise you only ever mean to keep is just a lie you haven’t noticed yet.

When I migrated rust-tool-base from GitHub Actions to GitLab CI, the move landed six jobs as allow_failure: true: the macOS and Windows tests, the integration tests, cargo-audit, trivy, and coverage. That wasn’t laziness. A migration is the wrong moment to also be fighting flaky gates. But it left me holding six promises to either keep or admit I wasn’t going to.

A check earns the right to block

Here’s the rule I settled on. A check earns the right to fail your build when two things are true: it’s meaningful (a red result is a real problem, not noise) and it’s reliable (it goes red only when there genuinely is a problem, and it can actually run to completion). Flip a check to blocking before both hold and you haven’t raised the bar, you’ve taught the team to force-merge past red, which is worse than no gate at all, because now the red means nothing.

Three of my six crossed that line within a few days. Three deliberately didn’t. The reasons are the whole story.

trivy: blocked once there was nothing to block on

trivy scans the dependency tree for HIGH and CRITICAL advisories. It went across as advisory for an honest reason: the Cargo.lock at migration time already carried two known HIGH/CRITICAL advisories I hadn’t cleared yet, a path-traversal in gix-validate and a DNS-rebinding issue in rmcp. Make trivy blocking with those sitting there and the pipeline is red from day one, over problems I already knew about and was already fixing. So it stayed advisory until the dependency bumps cleared both, and then the allow_failure line came out. The gate never changed. The tree underneath it got clean enough to stand on.

integration-tests: blocked once it could actually run

The integration tests stand up a real Gitea in a Docker-in-Docker service and talk to it. They were advisory for a different reason: they couldn’t reliably run. dind needs a privileged runner, and the suite was resolving the container host with a hardcoded 127.0.0.1 that didn’t hold everywhere. Blocking a job that fails for infrastructure reasons rather than code reasons is the fastest way to make people distrust the entire pipeline. So the fix wasn’t in the YAML, it was making the thing dependable: privileged set on the runner, and the host resolved through the test library’s own get_host() instead of a hardcoded address. Once it ran the same way every time, it earned the gate.

coverage: blocked once it could run at all, then once it cleared the bar

Coverage is the two-step one, and my favourite, because it nearly didn’t make it for a thoroughly undramatic reason: it ran out of memory. cargo llvm-cov instruments every test binary, and linking hundreds of instrumented object files needs more RAM than the shared medium runner had, so the job bus-errored on the link. I tagged it onto a larger runner, and then the shared SaaS runners were switched off entirely, so the tag matched nothing and the job sat pending forever.

The fix was a self-hosted homelab runner with the RAM the instrumented link actually needs. I moved coverage there but kept it advisory for one run, to confirm the box could finish the build before I trusted it. It did, at 73.22% line coverage, so I set the gate to fail under 70% and made it blocking. Three points of headroom: enough that ordinary churn won’t trip it, tight enough that a real drop will. A coverage gate pinned to the current number is a tripwire that fires on the very next commit; set it a touch below and it catches regressions instead of normal life.

The three I left advisory, on purpose

The point was never “block everything”. Three jobs are still allow_failure in the current pipeline, deliberately. The macOS and Windows tests run on SaaS runners that bill by the minute; they’re worth running, not worth blocking every merge of a Linux-first project over a quota I’m choosing to ration. And cargo-audit stays advisory because cargo-deny already does the blocking advisory check: cargo-audit is a second opinion from a different database, and a second opinion that can veto isn’t a second opinion, it’s a duplicate gate that will eventually disagree with the first and block you on the difference.

That’s the same rule from the other side. Those three haven’t earned the right to block, because blocking them would cost more than it ever caught.

The upshot

allow_failure: true is fine as a waiting room and corrosive as a destination. Every advisory check is a promise to make it blocking once it’s both meaningful and reliable, and the job is to keep the promise or admit you won’t. trivy earned its gate when the advisories cleared, the integration tests when they ran the same way every time, coverage when it had a runner with enough memory and a threshold set just below the current mark. The three I left advisory earned that standing too, by costing more to block than they’d catch. The YAML is one deleted line per job. Knowing which line to delete, and when, is the whole skill.

The build gate these sites never had

Mon, 06 Jul 2026 00:00:00 +0000

I run three Hugo sites now. This blog, my dad’s B&B, and the vanlife site my wife and I are slowly putting together. All three build with Hugo and deploy to GitLab Pages, and until recently all three had the same blind spot: the build only ever ran on main.

Which means that if I pushed something that broke the build, a typo in a shortcode, a cross-link to a page that doesn’t exist, a theme bump that didn’t take, I found out after it landed. The change was already merged, sitting on the default branch, and the deploy was the thing that fell over. No gate, no warning, nothing telling me on the merge request that the site wouldn’t build. Just a red pipeline on main and a site that had just stopped updating.

Three copies of the same job

I’ve written before about pulling shared CI into components you include rather than copy-paste, and this was a textbook case of why. Each of the three sites had its own hand-rolled pages job: the same pinned hugomods/hugo image, the same GIT_DEPTH: 0, the same nightly rebuild, all written out three times and drifting apart a little more with every tweak. I already had a zensical-pages component for the docs framework I use elsewhere, but nothing covered a plain Hugo build.

So I wrote one: a hugo-pages component in my shared cicd project, and pointed all three sites at it.

The gate they were missing

The interesting part isn’t the deduplication, it’s what the component fixes. It splits the work into two jobs, and that split is the entire reason the component exists.

hugo-build builds the site to public/, and it carries an explicit rule so it runs in merge-request pipelines too:

rules:
 - if: '$CI_COMMIT_TAG'
 when: never
 - if: '"$[[ inputs.mr_gate ]]" == "true"'
 when: on_success

That second line is the bit the old jobs never had. Push a broken Hugo build now and it shows up red on the merge request, where you can see it before merging, instead of sailing onto main and leaving the deploy as the only thing left to fall over. It’s interruptible: true, so pushing a fix cancels the in-flight run rather than queuing behind it. The deploy is a separate job, pages, which only runs where it should:

pages:
 variables:
 GIT_STRATEGY: none
 rules:
 - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
 when: never
 - if: '$CI_COMMIT_BRANCH == "$[[ inputs.deploy_branch ]]"'
 - if: '$CI_PIPELINE_SOURCE == "schedule"'

It takes the artifact hugo-build already produced and republishes it, with GIT_STRATEGY: none because there’s nothing to clone. Build everywhere, catch the breakage early; deploy only from main (and one other place, which is where it gets interesting). The component is pinned at v0.13.0.

The one component that wants to run on a schedule

Here’s the bit I like. Every other component in my cicd project carries a leading guard that says never run on a schedule, because a nightly Renovate schedule firing a release job or a deploy is exactly the kind of pointless busywork you don’t want.

hugo-pages does the opposite, on purpose. It deploys on schedule, and the reason is sitting in plain sight in the component itself:

# Schedule note: unlike every other component (which carries a
# `schedule → never` guard ...), hugo-pages DELIBERATELY deploys on
# `schedule`. A nightly pipeline schedule is the mechanism by which
# future-dated, non-draft posts go live once their date arrives (Hugo
# excludes future-dated content from `production` builds until then).
# The schedule IS the feature.

When I write a post and date it next Tuesday, Hugo’s production build simply doesn’t include it yet, because its date is in the future. Nothing publishes it on the day except a build that runs on the day and finds that the date has now arrived. That’s the nightly schedule. It’s not noise to be guarded against here, it’s the publish button. The post you’re reading went live exactly this way: written days early, dated forward, and pushed out by a scheduled pipeline the morning its date came up.

One codebase, two flatly opposite rules about the same trigger, and knowing which site wants which is the actual design.

A gate you can switch off

There was one wrinkle. The new MR build gate is the right default for a docs site with contributors, but for these three blogs it was friction. I run them trunk-based: I work on main, I audit locally, I push. A mandatory merge-request build for a workflow that doesn’t really use merge requests is a job that mostly just sits there.

So the gate is a boolean you can turn off:

mr_gate:
 type: boolean
 default: true

Left at its default, hugo-build runs everywhere and the gate is on, which is the safe posture for anyone adopting the component fresh. Set it to false, as this blog does, and hugo-build falls through to running only where it deploys, matching the classic Pages-only-on-main setup the sites had before, but now without three copies of the YAML to maintain. Safe by default, loose by choice, and the choice is one line.

Where it leaves things

A reusable component instead of three drifting copies, a build gate that catches a broken site before it merges, a schedule that publishes tomorrow’s post on time rather than firing for nothing, and an opt-out for the trunk-based sites that don’t want the gate. None of it is clever. It just means the next time I fat-finger a shortcode, I find out on the merge request and not from my dad asking why the B&B site is showing a stack trace.

Two bugs that taught me the rules

Wed, 20 May 2026 00:00:00 +0000

Some bugs are interesting because they’re subtle. These two were interesting because they were the exact opposite… in each case the tool had a hard rule I simply didn’t know about, and its error message couldn’t be bothered to tell me what that rule was. Both came out of building the infrastructure toolchain, both cost me a good deal more time than they had any right to, and both are the sort of thing that looks blindingly obvious the moment you know it and utterly baffling until you do.

So here they are, written down, partly to save you the bother and partly so I don’t go and forget them myself.

Bug one: the rule-less job that skips your merge requests

The cicd gate components, in their first cut, shipped with no rules: block. They were dead simple jobs: lint, scan, validate. No conditions, because they should just always run. Obviously.

They ran on branch pipelines. On merge requests, they didn’t run at all! The gates that were the entire point of the components were simply absent from the one place you’d most want to see them… the merge request.

The cause is a GitLab CI rule that’s remarkably easy to go years without ever learning: a job with no rules: block runs only on branch and tag pipelines. It does not run on merge-request pipelines. So “no conditions” doesn’t mean “runs everywhere” at all. It means “runs everywhere except a merge request”, which is about the least intuitive default I can think of.

The fix is faintly absurd, and that’s exactly what makes it stick. You add an unconditional rule: rules: [{ when: on_success }]. The content of that rule does precisely nothing. It always matches. What actually matters is that the job now has a rules: block at all, because merely having one is what makes a job eligible for merge-request pipelines. A rule whose content is meaningless, added solely so the block exists. That’s the fix. I’ll admit I stared at it for a moment.

Bug two: the import block that only works at the root

The second one came from terraform-aws-security-baseline. The account-hardening module needed to adopt a resource that already existed in the account, which is exactly what OpenTofu’s import {} block is for. So an import block went into the account-hardening module, right next to the resource it was adopting. The natural home for it, surely.

OpenTofu disagreed, and rejected it outright. The rule: an import block is only allowed in the root module. It can’t live inside a child module. A module that wants one of its own resources imported can’t declare that import itself… the import has to be declared up at the root, and the root caller does the adopting.

The fix was to take the import block out of the module and document caller-side adoption instead. The module describes the resource, and the root configuration that calls the module is where the import actually lives.

Two unrelated bugs, in two completely different tools, and the same shape sitting underneath both of them.

In each case the tool has a hard structural rule. Where a block is allowed to live. What makes a job eligible for a particular kind of pipeline. And in each case the error told me the tool was unhappy without telling me which rule I’d broken, so the obvious next move (debugging my own logic) was the wrong move entirely. There was nothing wrong with the logic. The thing was simply in a place the tool doesn’t allow, or missing a block the tool quietly insists on.

The lasting lesson here isn’t the two specific rules, useful as they are to know. It’s the reflex. When something that should obviously work just doesn’t, and the error is unhelpful, stop debugging your logic and start suspecting a structural rule about where something is allowed to be, or whether a thing is eligible in the first place. GitLab CI and OpenTofu both have a handful of these, and you mostly learn them the hard way, by tripping over them. Knowing the shape of the category at least means the next one costs you an hour instead of a whole afternoon.

Worth remembering

Two bugs from building the toolchain, one shape. A GitLab CI job with no rules: block runs on branches and tags but silently not on merge requests, and the fix is an unconditional rules: block whose content does nothing and whose mere existence is the entire point. An OpenTofu import block gets rejected inside a child module, because imports are only legal at the root, so the caller adopts and the module just describes.

Neither error named the rule it was enforcing, and that’s the category to watch for. When sound logic fails against an unhelpful error, suspect a structural rule about where a thing may live or whether it’s even eligible… not a bug in what you actually wrote. It’ll save you an afternoon. It certainly cost me a couple.

The component that fired on every schedule

Sat, 25 Jul 2026 00:00:00 +0000

I run two release drivers across my estate, because the two language ecosystems have different best-in-class tools and I’d rather use each than force one to do both. The Go projects use releaser-pleaser; the Rust ones use release-plz. That split is a story of its own. This is about something smaller and more annoying that fell out of it: a release driver that kept going off when it had no business going off.

releaser-pleaser ships as a GitLab CI component, and rather than reinvent it I started from Apricote’s upstream one. It does its job. But it decides when to run by gating on the branch: branch == "<default>", which is a perfectly sensible way to say “only manage releases on the main branch.” The trouble is that a Renovate schedule also runs on the default branch. A scheduled pipeline and a push-to-default pipeline both satisfy branch == default, so the component couldn’t tell them apart. Every time Renovate woke up to check for dependency updates, the release driver fired right alongside it. Wasted pipeline minutes at best, and at worst a release driver doing release-shaped things on a schedule nobody asked it to.

The tempting fix, adding a rule to each job to bail out on scheduled runs, just spreads the same workaround everywhere. I could add an override in the consuming repo, a little extra rules: stanza to exclude schedules, and move on. And then do it again in the next repo. And the next. Every project that used the component would carry its own copy of the same patch, and the day Apricote changed something upstream, I’d be reconciling that change against a dozen hand-rolled overrides scattered across the estate. That’s not a fix, it’s a debt with interest.

What actually fixes it is to stop consuming the upstream component directly and wrap it. I now have my own releaser-pleaser component that uses Apricote’s image underneath but bakes in the org’s conventions, the schedule guard included, at the wrapper layer. The repos consume my component. The gating is correct in exactly one place. When upstream changes, there’s one wrapper to reconcile, not a dozen repos. The principle is the one I keep coming back to as the person maintaining shared CI for a whole estate: when you don’t control a dependency’s gating, don’t patch it everywhere it’s used. Wrap it once, put your conventions in the wrapper, and let everything consume that.

There’s a sharp little gotcha buried in this that’s worth pulling out, because it bites everyone building release automation on GitLab eventually. When a job pushes a tag using CI_JOB_TOKEN, that tag will not trigger a downstream pipeline. It’s a deliberate loop-prevention in GitLab: a job’s own token isn’t allowed to kick off more pipelines, or you’d have CI triggering CI triggering CI. Which is fine until your release flow is “the driver pushes a tag, and the tag is supposed to start the release pipeline.” Push that tag with the job token and the release pipeline simply never fires, and you’re left staring at a tag that exists and a release that didn’t happen, with nothing in the logs to explain it. The fix is to push the tag with a token that is allowed to trigger pipelines, but the failure mode is silent, so you have to know it’s there.

This is roughly as thrilling as sorting the recycling: running releases across a handful of repos with a couple of different drivers, on a runner I keep small on purpose. But it goes wider than CI, really. The third time you paste the same workaround into a new place, it stops being a workaround. It’s a feature you haven’t admitted to yet.

Why go-tool-base left GitHub for GitLab

Mon, 11 May 2026 00:00:00 +0000

A botched version bump made me stop and actually look at where go-tool-base lived, and I didn’t much like what I saw. GitHub had spent months quietly falling over, and when Mitchell Hashimoto (GitHub user #1299, no less) publicly walked Ghostty off the platform, it stopped feeling like just my problem. I’ve been a GitLab fan for years, so the move was less a leap and more an overdue nudge. This is the why, not the how.

It started with a wrong number

Every migration has a trigger, and mine was embarrassingly small. A commit landed on main carrying a BREAKING CHANGE: footer it didn’t really deserve. Semantic-release did exactly what it’s told to do with that footer: it cut a major version. go-tool-base lurched from the v1 line straight to v2.0.0, and a chain of things that keyed off the version went sideways with it.

It was fixable. It wasn’t a disaster. But it was the kind of small, stupid breakage that makes you stop and actually look at your setup instead of just patching it and moving on. And when I looked, the version bump wasn’t the thing that bothered me. It was everything around it.

The platform had been quietly failing

I’d been losing time to GitHub for months. Not dramatically. No single outage you’d write home about, just a steady drip of Actions queues that wouldn’t drain, pull requests that wouldn’t merge, the occasional morning where the thing simply wasn’t there. You absorb it. You re-run the job. You make a coffee and try again. You tell yourself it’s a blip.

The trouble with a steady drip is that you stop counting it. It becomes weather.

The canary left the mine

Then, in late April, Mitchell Hashimoto (co-founder of HashiCorp, creator of Vagrant, Terraform and the Ghostty terminal) published Ghostty Is Leaving GitHub, and The Register picked it up a day later under the headline “GitHub ’no longer a place for serious work’”.

This is not a man with a casual relationship to GitHub. He’s, by his own account, user #1299, joined February 2008. He called it “the place that has made me the most happy”. And he still wrote this:

This is no longer a place for serious work if it just blocks you out for hours per day, every day.

The detail that landed hardest for me wasn’t a quote, it was a habit. He’d kept a journal for a month, marking an “X” on every day a GitHub outage had cost him working time. Almost every day had an X. Reading that, I realised I’d been having the same month. I’d just never been disciplined enough to write it down. He’d turned my vague “it’s been flaky lately” into a row of crosses on a calendar.

I want to ship software and it doesn’t want me to ship software.

When the person who’s been on the platform for eighteen years and loves it says that out loud, it stops being your private grumble. It’s the canary, and the canary has stopped singing.

Why GitLab, and not just “somewhere else”

Being annoyed at GitHub is a reason to leave. It is not, on its own, a reason to pick a destination. The destination has to be a positive choice.

For me GitLab was an easy one, because I’ve been a fan for years. Long enough, in fact, to have also been a reliable grumbler about their pricing tiers, which is how you know it’s a real relationship and not a honeymoon. What I’ve always rated is the model: GitLab treats source hosting, CI/CD, the package registry, releases and Pages as one integrated product, not a marketplace of bolted-on parts you assemble yourself.

That integration is the actual prize. On the old setup, “CI” meant a folder of separate GitHub Actions workflow files, each pinned, each its own little world. On GitLab it’s a single .gitlab-ci.yml pipeline with proper stages (lint, test, security, docs, release) and the release stage talks to the built-in package registry and Pages without me wiring up a single external credential. The CI job that builds the project can authenticate to the things the project needs because they’re the same platform.

There’s a second-order benefit too. A migration is a rare licence to fix things you’d never otherwise touch. Moving gave me the cover to reset go-tool-base’s versioning cleanly (back to a sensible v0.x line, the accidental v2.0.0 left behind as a cautionary tale) and to move the module path to its new home in one deliberate change rather than a thousand apologetic ones.

What I’m not going to claim

I’m not going to tell you GitHub is finished, or that GitLab never has a bad day, because it does, everyone does. This isn’t a teardown. GitHub gave go-tool-base a perfectly good home for its first year, and the archived mirror is still sitting there, read-only, pointing anyone who finds it at the new place.

What changed is simpler than a grand verdict. The friction crossed a line, someone I respect said the quiet part loudly enough that I couldn’t keep filing it under “weather”, and the place I’d have moved to anyway was sitting right there with a better model. Sometimes the prudent move and the move you secretly wanted turn out to be the same move, and you just need a wrong version number to give you permission.

Boiling it down

go-tool-base moved from GitHub to GitLab in May 2026. The proximate cause was a self-inflicted version-bump mess; the real cause was months of GitHub unreliability that I’d stopped consciously noticing until Mitchell Hashimoto’s very public departure named it for me. GitLab was a positive pick, not just an escape hatch: its integrated CI/CD, registry, releases and Pages are one product rather than a kit, and that integration is genuinely worth having. The migration also bought a clean versioning restart as a bonus.

If you’ve been absorbing a steady drip of friction and telling yourself it’s normal: try the calendar trick. Mark the X’s for a month. The page will tell you something you already half-know.

No access keys in CI

Fri, 08 May 2026 00:00:00 +0000

A long-lived AWS access key, sitting in a CI system, is just about the single credential I’d most like to be rid of. It’s powerful, it never expires unless someone remembers to rotate it (nobody remembers to rotate it), and it lives in one of the most attractive targets in the whole supply chain. For infrastructure that’s eventually going to hold a release-signing key, it’s exactly the wrong place to start. So the phpboyscout infrastructure has no AWS access key in CI at all. None.

The access key you don’t want

A CI pipeline that runs tofu apply against AWS needs AWS credentials. The traditional way to give it some is an IAM user with an access key pair, pasted into the CI system as a masked variable.

Look at what that key is. It’s long-lived: it works until someone remembers to rotate it, and rotating it is a chore, so mostly nobody does. It’s powerful: it can apply infrastructure, so it can do nearly anything. And it’s sitting in a CI system, which is one of the most attractive targets in your whole supply chain. You’ve taken your highest-value credential and stored a permanent copy of it in a place built for running automated jobs.

For infrastructure that’s going to hold a release-signing key, that’s precisely the wrong starting point. So the phpboyscout infrastructure has no AWS access key in CI at all. Not a well-guarded one. None.

Federation instead of a stored secret

The replacement is OIDC federation, and the shape of it is worth walking through, because it’s genuinely different from “a secret, but better”.

A modern CI platform can mint an OIDC token. GitLab does this with an id_tokens: block: at job time, GitLab issues a short-lived JSON Web Token, signed by GitLab, that asserts a set of facts. This is project X. This is pipeline Y. This is running on ref Z, of this type.

AWS can consume that. The sts:AssumeRoleWithWebIdentity call takes such a token and, if it satisfies an IAM role’s trust policy, returns short-lived AWS credentials for that role. The trust policy is where the control lives: it names GitLab as a trusted token issuer, and it constrains the token’s sub claim so that only the specific project, and the specific refs, you intend can assume the role.

Put it together: the pipeline asks GitLab for a token, hands it to AWS, and gets back credentials that last about an hour and are scoped to one role. Nothing long-lived is stored anywhere. The credential exists only for the job that needs it, and it can’t be stolen from a CI variable store, because it was never in one.

Two halves of one handshake

That handshake is built by two of the repos in this series, each owning one side.

terraform-aws-bootstrap builds the AWS half, in its automation-iam module: it registers GitLab as an OIDC identity provider in the account, and it creates the automation role with the trust policy that decides which pipelines may assume it.

The CI components build the consuming half: the id_tokens: block that asks GitLab for the JWT, and then simply letting the AWS provider’s own credential chain perform the exchange. The pipeline doesn’t call sts by hand. It presents the token; the SDK does the rest.

The gotcha: don’t set a profile

There’s one quiet way to break this, and a stack can look completely correct while doing it.

The AWS SDK finds credentials by walking a chain of sources in order. The web-identity path, the one that uses the OIDC token, is one link in that chain. It triggers off environment variables the CI sets up automatically.

But if the aws provider block has a hardcoded profile = "...", the SDK takes the profile link of the chain instead, and never reaches the web-identity link. A profile line is the sort of thing that ends up in a provider block from someone’s local development setup, where it’s exactly right. Committed and run in CI, it silently short-circuits the federation. The pipeline either fails to find credentials, or finds the wrong ones.

The rule is simple once you know it: the provider block that runs in CI must not name a profile. Leave the chain free to find the web identity. It’s the kind of bug that teaches you to be precise about which link of the credential chain you’re actually relying on.

The bottom line

Giving CI an AWS access key means storing your most powerful, longest-lived credential in one of your most exposed systems. OIDC federation removes it entirely. The CI platform mints a short-lived signed token, AWS exchanges it via AssumeRoleWithWebIdentity for hour-long credentials against a role whose trust policy names the exact pipeline, and nothing permanent is stored.

terraform-aws-bootstrap builds the AWS side, the identity provider and the trust policy; the CI components build the consuming side, the token request. The one trap is a hardcoded profile in the provider block, which short-circuits the SDK’s credential chain before it reaches the web-identity path. Get that right, and a pipeline deploys to AWS as a verifiable, short-lived identity, with no key to steal.

The secret that wasn't on my branch

Sat, 04 Jul 2026 00:00:00 +0000

My CI went red on a secret I’d never committed.

Not a close call, not a near-miss I’d half-forgotten about. gitleaks, the secret scanner, failed a merge request of mine on a private key that was not on my branch, was not in my change, and as far as I could tell had nothing to do with me at all. The job was adamant. I was baffled. Somewhere in between was a lesson about what a secret scanner actually scans.

Prove it isn’t yours

First rule of being accused: don’t get defensive, get evidence. The scanner pointed at a couple of commits carrying a test private key and a PEM block in a spec document. I genuinely didn’t recognise them, but “I don’t recognise it” is a feeling, not a fact, and feelings don’t reopen a pipeline.

Git will tell you the truth if you ask it precisely. The question is: are these flagged commits actually part of my branch?

git merge-base --is-ancestor <flagged-sha> HEAD

That asks “is this commit an ancestor of where I am?”. The answer came back no. The commits the scanner was choking on were not in my history. They weren’t mine.

So whose were they? A bit of digging turned them up on a completely separate, still-unmerged branch, where someone (me, a few days earlier, on a different feature) had committed a throwaway test key and a PEM example in a spec, on purpose, as fixtures. Deliberate, harmless, and nowhere near the branch under review. And yet here they were, failing a merge request that had never touched them.

What gitleaks actually scans

Here’s the bit I’d taken for granted. I assumed gitleaks detect scanned my change. It doesn’t. With no further instruction it scans the whole history reachable in the checkout it’s handed.

And the checkout it’s handed is where the second half of the surprise lives. GitLab runners default to GIT_STRATEGY: fetch, which reuses the runner’s working directory between jobs rather than cloning fresh every time. It’s faster, and most of the time you never notice. But it means a shared runner accumulates refs from every branch it has ever built. My MR’s job happened to run on a runner that had, at some point, built that other branch, so the fixtures were sitting right there in the local object store, fair game for a scanner walking the whole graph.

So gitleaks did exactly what I’d asked it to do, which was “scan everything”, and “everything” turned out to be a great deal more than my change. It walked the lot and dutifully reported fixtures from a branch I wasn’t even proposing to merge. The scanner wasn’t wrong. My idea of what it was looking at was.

Unblock now, fix properly after

Two problems on two timescales. I needed the MR to merge today, and I needed this to never happen again. Those want different fixes.

The immediate one: tell gitleaks those specific fixtures are known and intended. They’re test material, they’re meant to be there, so they go in the allowlist:

paths = [
 # ...
 '''internal/cmd/keys/keys_test\.go''',
 '''docs/development/specs/2026-06-08-keys-mint-command\.md''',
]

That unblocks the merge. It does nothing about the root cause, which is that the scan was looking at the wrong commits in the first place. Allowlisting individual false positives as they crop up is closing the stable door after the horse has bolted, one horse at a time, forever.

The real fix lives in the shared CI component, not in any one repo. Scope the scan to the commits the merge request actually introduces (cicd v0.10.3):

if [ -n "$CI_MERGE_REQUEST_DIFF_BASE_SHA" ]; then
 gitleaks detect --source . --verbose --redact \
 --log-opts="$CI_MERGE_REQUEST_DIFF_BASE_SHA..$CI_COMMIT_SHA"
else
 gitleaks detect --source . --verbose --redact
fi

--log-opts is handed straight through to git log, so that range, base-of-the-MR to tip, is the exact set of commits the merge request adds and nothing else. On a merge request the scan now sees only what you’re proposing to merge. Off a merge request (a plain branch or a tag pipeline) it falls back to the full scan, because there you genuinely do want the lot. The before-and-after in the job log tells the whole story: the entire accumulated history on one side, the handful of commits you actually wrote on the other.

Fixing the fix

There’s a tax on touching CI shell, and I paid it. The change went into the go, rust and tofu security templates. Go and tofu went green. Rust failed.

The rust template had built its optional --config flag the clever way, inline:

... $([ -n "$CONFIG" ] && echo "--config $CONFIG")

As a command argument, that substitution is harmless: its exit status is thrown away, so nobody cares that the test inside returns false when there’s no config. But when I rewrote the block and reached for the same pattern as an assignment, VAR=$(... && ...), it became a different animal. An assignment takes the exit status of the command substitution, and under set -e a non-zero status anywhere aborts the job. So on every run where the config was empty, which was most of them, the test returned false, the assignment inherited that false, and set -e killed the job stone dead. Same $(...), two completely different fates, decided entirely by whether it sat to the right of an = or got handed to a command as an argument. Go and tofu never used the assignment form, so only rust fell down the hole.

The fix was to stop being clever and write the boring if:

GITLEAKS_CONFIG=""
if [ -n "$CONFIG" ]; then
 GITLEAKS_CONFIG="--config $CONFIG"
fi

Boring shell is good shell.

What a scanner is actually looking at

The whole mess came from one unspoken assumption: that a tool called to scan “my change” was scanning my change. It was scanning a checkout, and a checkout on a shared runner is a much bigger, messier thing than the diff in front of you. None of the pieces were broken. gitleaks did its job, GIT_STRATEGY: fetch did its job, my fixtures were exactly where I’d left them. They just added up to a red pipeline that had nothing to do with the code I was trying to ship. I’d spent a good chunk of the day proving my innocence to a scanner that was only ever doing as it was told… and the one thing I’d actually got wrong was being sure I already knew what it was looking at.

PHP Boy Scout — CI/CD, and getting things released

CI you include, not copy

The .gitlab-ci.yml you keep copying

GitLab has a feature for exactly this

Why a monorepo of components

Authoring discipline

The self-test you cannot fully write

The same instinct, again

The gist

Stop installing the same tools on every pipeline job

One image for the whole toolchain

The same pile of tools, in every repo

One image, one source of truth

Publishing with crane, not a second build

Soft-failing the scanner on purpose

What it comes down to

A release is just another merge request

Same config, two answers

Rust brings its own toolchain

The gauntlet before the gun

“Tag v0.5.1 already exists”

I assumed the default

Same config, two answers

Doing it the way I’d have done it a year ago

Stop being clever and RTFM

Three traps release-plz sets for a Rust workspace

First, what release-plz is doing

Trap 1: the default tag template is built for one crate, not a workspace

Trap 2: “one release for the whole workspace” isn’t a setting, it’s a category error

Trap 3: a release reads its config from the release commit, not HEAD

If you set one thing

The version pin you can't lower

From allow_failure to blocking

What allow_failure actually buys you

A check earns the right to block

trivy: blocked once there was nothing to block on

integration-tests: blocked once it could actually run

coverage: blocked once it could run at all, then once it cleared the bar

The three I left advisory, on purpose

The upshot

The build gate these sites never had

Three copies of the same job

The gate they were missing

The one component that wants to run on a schedule

A gate you can switch off

Where it leaves things

Two bugs that taught me the rules

Bug one: the rule-less job that skips your merge requests

Bug two: the import block that only works at the root

The shape they share

Worth remembering

The component that fired on every schedule

Why go-tool-base left GitHub for GitLab

It started with a wrong number

The platform had been quietly failing

The canary left the mine

Why GitLab, and not just “somewhere else”

What I’m not going to claim

Boiling it down

No access keys in CI

The access key you don’t want

Federation instead of a stored secret

Two halves of one handshake

The gotcha: don’t set a profile

The bottom line

The secret that wasn't on my branch

Prove it isn’t yours

What gitleaks actually scans

Unblock now, fix properly after

Fixing the fix

What a scanner is actually looking at

The `.gitlab-ci.yml` you keep copying