CI/CD on PHP Boy Scout

Stop installing the same tools on every pipeline job

Sun, 19 Jul 2026 00:00:00 +0000

Every Go and Rust pipeline I run starts the same way, and for a long time the first thing each job did was waste a minute of my life. Before it could lint or test or scan, it had to go and fetch the tools to do those things: cargo binstall for nextest, cargo-deny, cargo-audit and llvm-cov, a go install for govulncheck, and so on. Every job. Every push. Every branch. Every Renovate bump.

On a hosted runner you’d barely notice. On mine you very much do, because mine is a single self-hosted box in the corner doing all of this with a small concurrency limit. That’s the budget I work within, and it changes the maths. A minute of tool-fetching isn’t a minute, it’s a minute multiplied by every job in every pipeline, all contending for the same runner, all doing the identical download-and-compile that the job before it just did and threw away. It was, comfortably, the single biggest recurring time sink in the whole setup, and none of it was the actual work.

The fix is the obvious one once you stop treating each job as an island: bake the tools into the image. There’s now a dev-tools image with the whole kit already installed, and the Go and Rust components default to it. The per-job installs are gone. A job starts and the tools are simply there. It’s the largest recurring-time win I’ve made on that runner, and it’s not close.

Here’s the part worth slowing down for, though, because it’s where I’d grab your sleeve if you were about to go and copy this: what you can’t bake.

You’d think you’d just throw everything in the image and be done. But there’s a line, and it runs between binaries and toolchain components. A tool like nextest or govulncheck is a standalone binary: it doesn’t care which Rust or Go toolchain the project uses, so baking it in is free and correct. A rustup component, though, like clippy or rustfmt, is tied to a specific toolchain. And a consumer’s repo can pin its own toolchain with a rust-toolchain.toml, which silently overrides whatever toolchain the image was built against. Bake clippy for toolchain A into the image, let a project ask for toolchain B, and you’ve got a clippy that doesn’t match the compiler it’s checking against. So rustup component add deliberately stays at runtime. You bake what’s toolchain-independent and you leave what isn’t. The rule is small but it’s the difference between a cache that helps and a cache that lies.

Which brings me to the bit I got wrong, because I always get one bit wrong.

The first cut of the image, v0.1.0, ran as a non-root user. That’s the textbook call. You don’t run containers as root if you can avoid it, every hardening guide says so, and I’d internalised it well enough to do it without thinking. It promptly broke every Go and Rust job. Permission-denied reading the module cache. Permission-denied on /etc/machine-id. The works.

The cause is one of those collisions between two reasonable things. GitLab’s Docker executor mounts and owns its caches in a particular way, and that ownership assumes the job can write where it expects to. A non-root user in the image doesn’t line up with how those cache volumes are owned, so the “secure” default walks straight into permission errors on the exact paths the build needs. v0.1.1 reverts the image to root.

I don’t love writing that sentence. Reverting to root as a “fix” reads like giving up on a security control. But the reality is that the security posture of a build image and the ownership model of the executor’s caches are two separate systems, and on a self-hosted GitLab runner the second one wins whether you like it or not. The textbook default was textbook for a context that isn’t this one. Better to know exactly why root is required here, and write it down, than to ship a non-root image that fails every pipeline and call it secure.

Bake the shared work in once and you stop paying for it on every job, which claws back more runner time than any other single change I’ve made. Just don’t get carried away and bake in everything, because a toolchain component isn’t a binary, and a hardened default stops being free the moment it fights the executor underneath it. Both of those I found out by breaking my own pipelines, which is, I’m reliably told, the traditional way.

The build gate these sites never had

Mon, 06 Jul 2026 00:00:00 +0000

I run three Hugo sites now. This blog, my dad’s B&B, and the vanlife site my wife and I are slowly putting together. All three build with Hugo and deploy to GitLab Pages, and until recently all three had the same blind spot: the build only ever ran on main.

Which means that if I pushed something that broke the build, a typo in a shortcode, a cross-link to a page that doesn’t exist, a theme bump that didn’t take, I found out after it landed. The change was already merged, sitting on the default branch, and the deploy was the thing that fell over. No gate, no warning, nothing telling me on the merge request that the site wouldn’t build. Just a red pipeline on main and a site that had just stopped updating.

Three copies of the same job

I’ve written before about pulling shared CI into components you include rather than copy-paste, and this was a textbook case of why. Each of the three sites had its own hand-rolled pages job: the same pinned hugomods/hugo image, the same GIT_DEPTH: 0, the same nightly rebuild, all written out three times and drifting apart a little more with every tweak. I already had a zensical-pages component for the docs framework I use elsewhere, but nothing covered a plain Hugo build.

So I wrote one: a hugo-pages component in my shared cicd project, and pointed all three sites at it.

The gate they were missing

The interesting part isn’t the deduplication, it’s what the component fixes. It splits the work into two jobs, and that split is the entire reason the component exists.

hugo-build builds the site to public/, and it carries an explicit rule so it runs in merge-request pipelines too:

rules:
 - if: '$CI_COMMIT_TAG'
 when: never
 - if: '"$[[ inputs.mr_gate ]]" == "true"'
 when: on_success

That second line is the bit the old jobs never had. Push a broken Hugo build now and it shows up red on the merge request, where you can see it before merging, instead of sailing onto main and leaving the deploy as the only thing left to fall over. It’s interruptible: true, so pushing a fix cancels the in-flight run rather than queuing behind it. The deploy is a separate job, pages, which only runs where it should:

pages:
 variables:
 GIT_STRATEGY: none
 rules:
 - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
 when: never
 - if: '$CI_COMMIT_BRANCH == "$[[ inputs.deploy_branch ]]"'
 - if: '$CI_PIPELINE_SOURCE == "schedule"'

It takes the artifact hugo-build already produced and republishes it, with GIT_STRATEGY: none because there’s nothing to clone. Build everywhere, catch the breakage early; deploy only from main (and one other place, which is where it gets interesting). The component is pinned at v0.13.0.

The one component that wants to run on a schedule

Here’s the bit I like. Every other component in my cicd project carries a leading guard that says never run on a schedule, because a nightly Renovate schedule firing a release job or a deploy is exactly the kind of pointless busywork you don’t want.

hugo-pages does the opposite, on purpose. It deploys on schedule, and the reason is sitting in plain sight in the component itself:

# Schedule note: unlike every other component (which carries a
# `schedule → never` guard ...), hugo-pages DELIBERATELY deploys on
# `schedule`. A nightly pipeline schedule is the mechanism by which
# future-dated, non-draft posts go live once their date arrives (Hugo
# excludes future-dated content from `production` builds until then).
# The schedule IS the feature.

When I write a post and date it next Tuesday, Hugo’s production build simply doesn’t include it yet, because its date is in the future. Nothing publishes it on the day except a build that runs on the day and finds that the date has now arrived. That’s the nightly schedule. It’s not noise to be guarded against here, it’s the publish button. The post you’re reading went live exactly this way: written days early, dated forward, and pushed out by a scheduled pipeline the morning its date came up.

One codebase, two flatly opposite rules about the same trigger, and knowing which site wants which is the actual design.

A gate you can switch off

There was one wrinkle. The new MR build gate is the right default for a docs site with contributors, but for these three blogs it was friction. I run them trunk-based: I work on main, I audit locally, I push. A mandatory merge-request build for a workflow that doesn’t really use merge requests is a job that mostly just sits there.

So the gate is a boolean you can turn off:

mr_gate:
 type: boolean
 default: true

Left at its default, hugo-build runs everywhere and the gate is on, which is the safe posture for anyone adopting the component fresh. Set it to false, as this blog does, and hugo-build falls through to running only where it deploys, matching the classic Pages-only-on-main setup the sites had before, but now without three copies of the YAML to maintain. Safe by default, loose by choice, and the choice is one line.

Where it leaves things

A reusable component instead of three drifting copies, a build gate that catches a broken site before it merges, a schedule that publishes tomorrow’s post on time rather than firing for nothing, and an opt-out for the trunk-based sites that don’t want the gate. None of it is clever. It just means the next time I fat-finger a shortcode, I find out on the merge request and not from my dad asking why the B&B site is showing a stack trace.

The secret that wasn't on my branch

Sat, 04 Jul 2026 00:00:00 +0000

My CI went red on a secret I’d never committed.

Not a close call, not a near-miss I’d half-forgotten about. gitleaks, the secret scanner, failed a merge request of mine on a private key that was not on my branch, was not in my change, and as far as I could tell had nothing to do with me at all. The job was adamant. I was baffled. Somewhere in between was a lesson about what a secret scanner actually scans.

Prove it isn’t yours

First rule of being accused: don’t get defensive, get evidence. The scanner pointed at a couple of commits carrying a test private key and a PEM block in a spec document. I genuinely didn’t recognise them, but “I don’t recognise it” is a feeling, not a fact, and feelings don’t reopen a pipeline.

Git will tell you the truth if you ask it precisely. The question is: are these flagged commits actually part of my branch?

git merge-base --is-ancestor <flagged-sha> HEAD

That asks “is this commit an ancestor of where I am?”. The answer came back no. The commits the scanner was choking on were not in my history. They weren’t mine.

So whose were they? A bit of digging turned them up on a completely separate, still-unmerged branch, where someone (me, a few days earlier, on a different feature) had committed a throwaway test key and a PEM example in a spec, on purpose, as fixtures. Deliberate, harmless, and nowhere near the branch under review. And yet here they were, failing a merge request that had never touched them.

What gitleaks actually scans

Here’s the bit I’d taken for granted. I assumed gitleaks detect scanned my change. It doesn’t. With no further instruction it scans the whole history reachable in the checkout it’s handed.

And the checkout it’s handed is where the second half of the surprise lives. GitLab runners default to GIT_STRATEGY: fetch, which reuses the runner’s working directory between jobs rather than cloning fresh every time. It’s faster, and most of the time you never notice. But it means a shared runner accumulates refs from every branch it has ever built. My MR’s job happened to run on a runner that had, at some point, built that other branch, so the fixtures were sitting right there in the local object store, fair game for a scanner walking the whole graph.

So gitleaks did exactly what I’d asked it to do, which was “scan everything”, and “everything” turned out to be a great deal more than my change. It walked the lot and dutifully reported fixtures from a branch I wasn’t even proposing to merge. The scanner wasn’t wrong. My idea of what it was looking at was.

Unblock now, fix properly after

Two problems on two timescales. I needed the MR to merge today, and I needed this to never happen again. Those want different fixes.

The immediate one: tell gitleaks those specific fixtures are known and intended. They’re test material, they’re meant to be there, so they go in the allowlist:

paths = [
 # ...
 '''internal/cmd/keys/keys_test\.go''',
 '''docs/development/specs/2026-06-08-keys-mint-command\.md''',
]

That unblocks the merge. It does nothing about the root cause, which is that the scan was looking at the wrong commits in the first place. Allowlisting individual false positives as they crop up is closing the stable door after the horse has bolted, one horse at a time, forever.

The real fix lives in the shared CI component, not in any one repo. Scope the scan to the commits the merge request actually introduces (cicd v0.10.3):

if [ -n "$CI_MERGE_REQUEST_DIFF_BASE_SHA" ]; then
 gitleaks detect --source . --verbose --redact \
 --log-opts="$CI_MERGE_REQUEST_DIFF_BASE_SHA..$CI_COMMIT_SHA"
else
 gitleaks detect --source . --verbose --redact
fi

--log-opts is handed straight through to git log, so that range, base-of-the-MR to tip, is the exact set of commits the merge request adds and nothing else. On a merge request the scan now sees only what you’re proposing to merge. Off a merge request (a plain branch or a tag pipeline) it falls back to the full scan, because there you genuinely do want the lot. The before-and-after in the job log tells the whole story: the entire accumulated history on one side, the handful of commits you actually wrote on the other.

Fixing the fix

There’s a tax on touching CI shell, and I paid it. The change went into the go, rust and tofu security templates. Go and tofu went green. Rust failed.

The rust template had built its optional --config flag the clever way, inline:

... $([ -n "$CONFIG" ] && echo "--config $CONFIG")

As a command argument, that substitution is harmless: its exit status is thrown away, so nobody cares that the test inside returns false when there’s no config. But when I rewrote the block and reached for the same pattern as an assignment, VAR=$(... && ...), it became a different animal. An assignment takes the exit status of the command substitution, and under set -e a non-zero status anywhere aborts the job. So on every run where the config was empty, which was most of them, the test returned false, the assignment inherited that false, and set -e killed the job stone dead. Same $(...), two completely different fates, decided entirely by whether it sat to the right of an = or got handed to a command as an argument. Go and tofu never used the assignment form, so only rust fell down the hole.

The fix was to stop being clever and write the boring if:

GITLEAKS_CONFIG=""
if [ -n "$CONFIG" ]; then
 GITLEAKS_CONFIG="--config $CONFIG"
fi

Boring shell is good shell.

What a scanner is actually looking at

The whole mess came from one unspoken assumption: that a tool called to scan “my change” was scanning my change. It was scanning a checkout, and a checkout on a shared runner is a much bigger, messier thing than the diff in front of you. None of the pieces were broken. gitleaks did its job, GIT_STRATEGY: fetch did its job, my fixtures were exactly where I’d left them. They just added up to a red pipeline that had nothing to do with the code I was trying to ship. I’d spent a good chunk of the day proving my innocence to a scanner that was only ever doing as it was told… and the one thing I’d actually got wrong was being sure I already knew what it was looking at.

Sign your own binaries with go-tool-base, part 6: sign every release with GoReleaser

Mon, 22 Jun 2026 00:00:00 +0000

By now you’ve got all the pieces lying on the bench. A KMS key that signs but never hands over its private half (Part 2). A CI role you can assume over OIDC with no stored credentials (Part 3). A public key you’ve minted and published (Part 4), and embedded in the binary (Part 5). What you don’t have yet is the bit that makes it routine: a release that signs itself, every time, without you remembering to do anything. That’s this part. We wire signing into the tagged-release pipeline so that pushing a v* tag is the whole ceremony.

This is the part of the signing series where the chain stops being a set of commands you run by hand and becomes something the pipeline does for you. We’re using GoReleaser, which already builds your binaries, writes a checksums file and cuts the release. It needs one extra trick: sign the checksums on the way out, through the KMS key, using credentials that only exist for the length of the job. And you don’t hand-write that trick into your release config. gtb does.

You’ll need a working GoReleaser setup releasing a go-tool-base CLI, plus the KMS key, the signer role and the embedded public key from the earlier parts.

One command wires the signing in

In Part 5 you ran gtb enable signing to turn on the verifying side: embed the key, check every update against it. Now you give that same command the key the release pipeline should sign with, and it wires the producing side too:

gtb enable signing --key-id alias/acme-release-signing-v1

That records the KMS key in your .gtb/manifest.yaml and regenerates .goreleaser.yaml with a signs: block that calls gtb sign. The WKD email and everything else you set in Part 5 stay exactly as they were; you’re adding the key, not starting over. Because the block is generated, you don’t hand-edit the release config any more than you hand-edit the embed wiring. Change the key later (a new region, a rotated alias) and you re-run the command, not the YAML.

The generated block looks like this:

signs:
 - id: checksums
 cmd: gtb
 args:
 - "--ci"
 - "sign"
 - "--backend"
 - "aws-kms"
 - "--kms-region"
 - "eu-west-2"
 - "--key-id"
 - "alias/acme-release-signing-v1"
 - "--public-key"
 - "internal/trustkeys/keys/signing-key-v1.asc"
 - "--output"
 - "${signature}"
 - "${artifact}"
 artifacts: checksum
 signature: "${artifact}.sig"
 output: true

The region defaults to eu-west-2 and the public key to the internal/trustkeys/keys/signing-key-v1.asc you embedded in Part 5; pass --kms-region or --public-key if yours differ. The backend defaults to aws-kms, which is the one that matters in CI.

Why only the checksums

artifacts: checksum is the line doing the load-bearing work. It tells GoReleaser to run the signing command once, over the checksums manifest only, not over every binary and archive in the release.

Here’s why that’s enough. GoReleaser builds your binaries, then writes a checksums.txt listing the SHA-256 of each one. Every artefact in the release is named in that file by its hash. So if you sign the manifest, you’ve transitively vouched for everything it lists: change a single byte of any binary and its hash no longer matches the line in checksums.txt, and the moment you alter checksums.txt to cover for that, the signature over it breaks. One signature, the entire release covered, through the hash chain. The per-binary build stays completely untouched, which keeps reproducible builds reproducible. signature: "${artifact}.sig" names the output checksums.txt.sig, the detached, ASCII-armored OpenPGP signature your tool looks for on every self-update.

No shim, just `gtb`

If you go reading go-tool-base’s own release config, you’ll find its signs: block points at a scripts/sign-release.sh shim rather than calling gtb directly. Yours doesn’t, and the difference is worth understanding.

go-tool-base is signing the very binary it’s in the middle of building, so it can’t use an installed gtb to do it. Its shim runs go run ./cmd/gtb to build a throwaway signer from source. It also reads the key id, public key and region from environment variables, because the one config has to serve every build. Your tool has neither problem. It calls the gtb you already installed, and the generator already knows your key id, region and public-key path because you just told it. So there’s nothing for a shim to abstract: the whole invocation goes straight into args:, where you can read it.

The one thing you do still need is gtb on the release runner’s PATH. Install it in a before_script, or bake it into your CI image, the same way you would any other release tool.

Where the credentials come from

gtb sign asks KMS to sign. KMS will only oblige if the caller has credentials, and the whole point of keyless CI is that there are no stored credentials to leak. So they’re minted on the fly.

On GitLab, the release job declares an id_tokens: block. GitLab injects a short-lived OIDC token (a JWT) into the job, the before_script writes it to a file, and the AWS SDK’s default credential chain picks it up from there. No aws CLI call, no assume-role-with-web-identity you write yourself: set AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE and the SDK does the web-identity exchange the first time gtb sign touches KMS:

goreleaser:
 id_tokens:
 AWS_WEB_IDENTITY_TOKEN:
 aud: sts.amazonaws.com
 variables:
 AWS_REGION: eu-west-2
 AWS_ROLE_ARN: arn:aws:iam::…:role/acme-release-signing-v1-signer
 AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/oidc-token
 before_script:
 - echo "$AWS_WEB_IDENTITY_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"

The key id and region live in the generated signs: block now, not here, so CI only supplies the AWS credentials. AWS_REGION still earns its place: the --kms-region flag points the KMS client at the key, while the SDK uses AWS_REGION for the STS exchange that mints the credentials in the first place. The aud has to match the audience the signer role’s trust policy expects (Part 3 set this up; for the OIDC provider go-tool-base uses, that’s sts.amazonaws.com). On GitHub the moving parts are the same, you just let aws-actions/configure-aws-credentials do the token-to-credentials dance instead of writing the file yourself.

Those credentials are scoped tight. The signer role’s trust policy pins it to this project’s tag pipelines, so even if the role ARN leaked, nothing but a release tag on your repo can assume it.

Don’t sign when there’s nothing to sign with

A local goreleaser release --snapshot, or a CI run that isn’t a release, has no OIDC token and no business reaching for KMS. GoReleaser is told to skip the whole signing step in that case: the release job runs with --skip=sign unless the web-identity token is present. So a non-release build never so much as looks at KMS, which is exactly what you want when you’re iterating on your laptop. The signing only fires on the real thing.

Cut a release

With all of that in place, releasing is one push:

git tag v1.4.0
git push origin v1.4.0

The tag pipeline fires. GoReleaser builds the binaries, writes checksums.txt, calls gtb sign, gtb asks KMS to sign over OIDC, and checksums.txt.sig lands next to the manifest. Both get attached to the release. Nobody typed a signing command and no private key was anywhere near the runner.

A two-person gate, if you want one

There’s a window worth thinking about: a compromised CI runner during a release could, in principle, ride the OIDC credentials to get one malicious thing signed. You can shut that window with an approval gate in front of the signing job. On GitLab, a protected environment with a required approval (or a manual job) makes the release wait for a second pair of eyes; on GitHub, “required reviewers” on the release environment does the same. It’s optional, and it adds friction to every release, so weigh it against how exposed your runners are. For a lot of projects the OIDC scoping alone is enough; for anything where a forged release would be a genuine incident, the gate is cheap insurance.

Where this leaves you

Every release you cut from here on carries a signature made by a key you control and verifiable by anyone, and you got there by pushing a tag. That closes the loop the series has been building toward: the production side now actually produces signed releases.

Which means you can finally pull the trigger on the bit Part 5 left primed. We embedded the public key and left enforcement off, because turning it on before you ship signatures would brick every update. Now that signatures are shipping, Part 7 deals with the part everyone skips and nobody can afford to: rotating the key, and getting yourself out of trouble if one ever goes bad, without locking your users out.

Three traps release-plz sets for a Rust workspace

Fri, 05 Jun 2026 00:00:00 +0000

I wrote up the two days I lost releasing a seventeen-crate workspace to crates.io as a war story, wrong turns and all. This is the other half: the field guide, so you don’t have to lose the same two days.

release-plz is a genuinely good tool, and none of what follows is a bug. It’s three behaviours that are entirely within its design and will still ambush you the moment you point it at a Cargo workspace rather than a single crate. Mildest first, because the third is the one that actually ate my release.

First, what release-plz is doing

In one line: it’s release-please for cargo. It keeps a Release MR open, bumps your versions and per-crate changelogs from your Conventional Commits, and when that MR merges it publishes every crate to crates.io and tags the release. On a workspace where N crates all share one version, “the release” is N publishes and N tag operations. Hold on to that N. It’s hiding behind all three traps.

Trap 1: the default tag template is built for one crate, not a workspace

You will reach for one tag per version, and for me it was more than tidiness. I wanted to ship the whole framework as a single release: one v0.5.1 covering all seventeen crates, because that was the compatibility promise I wanted to make. Use the crates that share a version and they’re guaranteed to work together. A single tag felt like the natural way to say “this is one coherent release of the whole thing” (and it didn’t hurt that the repo already had a v0.5.0 tag from before release-plz, so one unified tag also looked like continuity). So you either set this, or, worse, you leave git_tag_name unset assuming the default does something workspace-aware:

git_tag_name = "v{{ version }}"

Here’s the catch. release-plz’s default git_tag_name is v{{ version }}, and release-plz tags per crate. So the first crate publishes and creates the tag v0.5.1. The second crate publishes and tries to create v0.5.1 again:

ERROR failed to create git tag 'v0.5.1'
 "message": "Tag v0.5.1 already exists"

By the time you read that error, the first crate (and on a retry, the next, and the next) is already live on crates.io, and crates.io publishes are forever. Leaving the line out doesn’t save you, because the default is the same single-crate-shaped template. This is the trap I walked straight into on the release commit.

Trap 2: “one release for the whole workspace” isn’t a setting, it’s a category error

The natural next thought is “fine, I’ll keep one tag but configure release-plz to roll the crates into a single release.” There’s no knob for that, and chasing one is a waste of an afternoon. release-plz’s model is per-crate all the way down: per-crate tags, per-crate GitLab/GitHub releases, per-crate changelogs. “One unified release for the whole workspace” isn’t an option it withholds, it’s a shape it doesn’t have.

So you stop fighting it and set the per-crate templates explicitly:

git_tag_name = "{{ package }}-v{{ version }}"
git_release_name = "{{ package }} v{{ version }}"

Now each crate gets its own tag (rtb-assets-v0.5.1, rtb-config-v0.5.1, and so on) and its own release. It’s more objects per version than you wanted, but it’s the grain the tool works in, and once you accept that the collisions stop.

This is where I had to pull apart two things I’d quietly merged in my head: the version and the tag. The compatibility promise I cared about, that crates sharing a version work together, is carried by the version, and release-plz keeps every crate on the one workspace version no matter how it tags them. The tag is just a label pointing at a commit. I’d wanted a single tag to mean “one coherent framework release”, but the coherence was always in the shared version number, not in the tag. Once that landed, seventeen tags stopped feeling like seventeen releases of seventeen different things and started looking like what they are: seventeen labels on one versioned release. The version is not the tag. If you still want one human-facing narrative for the whole thing, keep a hand-written root CHANGELOG.md alongside the generated per-crate ones, rather than trying to make release-plz aggregate.

Trap 3: a release reads its config from the release commit, not HEAD

This is the small one, and the one that cost me the most, because it makes the fix for Trap 1 look like it isn’t working.

When release-plz runs a release, it does not read release-plz.toml from your working tree. It reads it from the release commit, the commit that first introduced the version it’s releasing. So picture the obvious recovery: you hit the tag collision, you realise your template is wrong, you fix it in a follow-up commit and push to main. Your fix is real. It’s committed. It’s on the default branch. And it is completely ignored, because the version hasn’t changed, so the release commit release-plz reads from is still the old one with the old template.

I didn’t take this on faith. With the corrected per-crate template sitting on HEAD, the CI release job still tried to create the unified tag, pinned to the old commit:

ERROR failed to create git tag 'v0.5.1' with ref 'f6de975...'
 "message": "Tag v0.5.1 already exists"

That ref is the release commit, not the HEAD that held my fix. And the cruel part: release-plz release --dry-run on your laptop reads your working-directory config, so it renders the shiny new per-crate tags and tells you you’re sorted. CI runs the real thing against the release commit and does something else entirely. Same config file, two different answers depending on who’s asking, which is why the war story has the title it does.

The operational rule that falls out of this: any release-plz config change that affects how a release behaves has to ride along with a version bump, or it does not apply. A “fix-up” commit on its own is a no-op.

If you set one thing

If you run release-plz on a multi-crate workspace and you change a single line from the defaults, make it the tag template:

git_tag_name = "{{ package }}-v{{ version }}"

And set it before your first release, not during it, so it’s already in the commit that introduces the version, because that’s the only commit a release will ever read it from. Everything else here follows from two facts: the grain is per-crate, and CI reads history while your laptop reads your working tree. Trust the history.

None of this is release-plz misbehaving. Every bit of it is documented and deliberate. It just isn’t where you’ll think to look until it has published six crates you can’t take back, which is roughly how I came to know it so well.

Same config, two answers

Wed, 03 Jun 2026 00:00:00 +0000

Let me confess a small heresy first, because it’s the reason any of this happened. After a career spent as a branching man, gitflow, gitlabflow, a tidy develop branch and a careful dance of merges, I’ve come round to trunk-based development. I resisted it for years. It felt like working without a net.

What changed my mind was working solo with an AI pair. The branch ceremony that earns its keep on a team of eight is just drag when it’s me and a model at two in the morning. So I’ve softened on “main is always deployable” and let the trunk act as the develop branch, with tagged releases as the actual source of truth. For compiled languages, where the artefact you ship is a built, tagged thing and not whatever’s on a server right now, that finally clicks.

I’d already rolled this out on my Go and Terraform projects with releaser-pleaser, a GitLab-native take on release-please: a bot keeps a Release MR open, and merging it cuts the tag. It’s the same model I wrote about when the infra repo moved to plan-on-merge, apply-on-tag. Lovely. Then I came to do the same for rust-tool-base, and Rust, being Rust, had opinions.

Rust brings its own toolchain

releaser-pleaser is happy to tag a repo and write a release. What it does not do is cargo publish seventeen crates to crates.io in dependency order. Rust’s release story isn’t “push a tag and let a runner build a binary”, it’s a whole publishing pipeline with a public registry at the end of it, and that registry has rules of its own. So for the Rust workspace I reached for the tool built for exactly that job: release-plz. Same Release-MR shape, but it understands cargo, versions every crate, and publishes the lot.

That was the right call. Getting it to actually do it was where I spent two days I’d quite like back.

The gauntlet before the gun

Before I got anywhere near the interesting failure, there was a run of CI papercuts, the sort where every fix politely reveals the next one. GitLab checks out a detached HEAD, and release-plz wants to be on a branch (“HEAD does not point to a branch”), so you re-attach. Then the default CI_JOB_TOKEN can’t push to a protected repo, so you point the remote at a real token. Then release-plz assumes you’re on GitHub and errors that the repo “is not hosted in GitHub”, so you tell it --forge gitlab. Then it refuses to run at all because the pages job left a public/ directory lying about and the working tree is “dirty”, so you stop pulling artefacts into the job.

Five merge requests before the thing would even start doing its actual job. You can read the scar tissue in the before_script; every line in it is a fix for something on that list. None of it was hard. It was just death by a thousand cuts, and I was feeling quite smug by the time it finally reached the publish step.

I should not have been.

“Tag v0.5.1 already exists”

My release-plz.toml asked for one tag per release:

git_tag_name = "v{{ version }}"
git_release_name = "v{{ version }}"

That felt obviously right. It matched the repo’s existing v0.5.0 tag, it’s how a single-crate project tags, and the crates all share one workspace version anyway. One version, one tag. What’s to argue with?

release-plz, that’s what. It tags per crate. So it publishes a crate, creates the tag, publishes the next crate, and tries to create the same tag again:

INFO published rtb-assets 0.5.1
ERROR failed to release package
Caused by:
 0: failed to create git tag 'v0.5.1' with ref 'f6de975a75...'
 1: Response body: { "message": "Tag v0.5.1 already exists" }
 2: HTTP status client error (400 Bad Request) ... /repository/tags

The collision is annoying. What makes it a proper trap is the half-second before it: published rtb-assets 0.5.1. That happened. On crates.io. For keeps. A crates.io publish is forever, there is no unpublish, only a yank that still leaves the name and version burned. So every time my flaky pipeline limped one crate further and then fell over on the tag, it left another crate live on the public registry that I could never take back. By the time the dust settled, six of the seventeen were out there: rtb-assets and rtb-config, then on a later retry rtb-credentials and rtb-error, then rtb-app and rtb-redact. Two more permanent crates per failed run.

I assumed the default

My first fix was the clever one, and it deserves to be on display because it’s the whole lesson in miniature. I deleted the git_tag_name line. My reasoning: per-crate tags are release-plz’s native model, so surely its default does the right thing without me spelling it out. I was confident enough to write it into the commit message: “per-crate tags/releases (release-plz defaults).”

The next run collided on v0.5.1, exactly as before.

Because release-plz’s default git_tag_name is not per-crate. It’s the unified v{{ version }}. I had deleted a line that said the wrong thing and replaced it with a default that said the same wrong thing, then congratulated myself for tidiness. If I’d spent thirty seconds on the configuration reference instead of thirty seconds being clever, I’d have read that in black and white.

Same config, two answers

So I read the manual, and set it explicitly:

git_tag_name = "{{ package }}-v{{ version }}"
git_release_name = "{{ package }} v{{ version }}"

On my laptop, a dry run rendered exactly the per-crate tags I wanted. In CI, the very next run published another crate and then created the tag v0.5.1. The unified one. The wrong one. The one I had just, demonstrably, on main, replaced. Same release-plz.toml, two completely different answers depending on who was asking.

That one took me an embarrassingly long time to see. release-plz does not read your config from the working tree when it runs a release. It reads it from the release commit, the commit that introduced the version it’s releasing. My version was still 0.5.1, set days earlier on a commit that still carried the unified template. You can see it in the failure: the tag it tries to create is pinned to ref 'f6de975...', an old commit, not the HEAD that held my fix. Every edit I made at the tip of main was real, committed, and utterly invisible to the release of 0.5.1, because no version bump had created a fresh release commit for it to read. My fix was correct and inert at the same time. The dry run read my working directory and looked perfect; CI read history and did something else.

There is no config change that rescues an in-flight release. The version was already out, half-published, tagged wrong, and pointed at a commit I couldn’t edit without bumping the version, which I couldn’t cleanly do with six crates already live.

Doing it the way I’d have done it a year ago

So I stopped. Three retries deep, each one a seventy-minute CI cycle thrown at an opaque mismatch, six crates already immovable on crates.io, and a tooling problem I now understood well enough to know the tool was never going to dig me out of this particular hole. The question quietly changed from “why is it doing this?” to “am I going to keep grinding, or finish this the way I would have before I had clever tooling?”

I went manual. cargo publish, the remaining eleven, by hand, in dependency order: the leaf crates first and the rust-tool-base umbrella dead last, because it depends on all of them. crates.io rate-limits new crate names, so after a burst it simply made me wait, a roughly half-hour pause in the middle while the registry caught its breath and I caught mine. Then one v0.5.1 tag, cut by hand, and one GitLab release to match the convention. The next CI run came up green, for the gloriously dull reason that there was nothing left to do: every crate published, the tag already there.

Stop being clever and RTFM

The tool was never broken. Every single thing it did was documented behaviour I hadn’t bothered to read: that the default tag template is unified, that the model is per-crate, that a release reads its config from the release commit and not from HEAD. I assumed my way past the manual three times in a row, and each assumption cost me real, permanent state on a public registry that doesn’t take returns.

And that’s the part that actually stung, because I should have known better than most. I wasn’t a beginner here. I knew the Release-MR pattern cold, I’d shipped it half a dozen times with releaser-pleaser on my Go and Terraform repos. That familiarity was the trap. I trusted the pattern and skipped the tool, on the lazy assumption that something I understood well in one tool would behave the same in the next. release-plz carries the same design, but it’s a different tool, with its own defaults and its own idea of where the config lives. The pattern came across fine. The mechanics didn’t, and I never thought to check.

So here’s the lesson, written down in the hope it sticks this time: no matter how familiar I am with a pattern or a design, the moment I switch the tool that implements it, reading the manual is paramount. The familiarity is exactly what tempts you to skip it, and exactly why you can’t. (The narrower, more practical one, while I’m here: a config change that affects how a release behaves has to travel with a version bump, or it sits there looking applied and doing nothing.)

release-plz is genuinely good, and every release since has gone out clean on the first try, the way the rest of the CI now does. I just had to stop being clever long enough to read how it actually works. RTFM. I’ll get it tattooed eventually.

From allow_failure to blocking

Sat, 30 May 2026 00:00:00 +0000

There’s a special kind of CI job that everyone on a team quietly learns to ignore: the one marked allow_failure: true. It runs, it goes red, the pipeline goes green anyway, and after the third time you stop looking at it. I inherited six of those when I moved rust-tool-base’s CI to GitLab. Over a few days I turned three of them into real gates, and the interesting part was never the YAML. It was working out which ones had earned the right to block, and which hadn’t.

What allow_failure actually buys you

allow_failure: true is genuinely useful, and quietly corrosive. It lets a job report a problem without stopping the pipeline, which is exactly right for a check that’s noisy, not yet stable, or guarding against something you can’t fix this minute. The trouble is that a warning nobody is forced to act on is a warning nobody acts on. Leave a job advisory long enough and it becomes scenery: red, ignored, pointless. So an advisory check is really a promise, “I’ll make this blocking once it’s trustworthy”, and a promise you only ever mean to keep is just a lie you haven’t noticed yet.

When I migrated rust-tool-base from GitHub Actions to GitLab CI, the move landed six jobs as allow_failure: true: the macOS and Windows tests, the integration tests, cargo-audit, trivy, and coverage. That wasn’t laziness. A migration is the wrong moment to also be fighting flaky gates. But it left me holding six promises to either keep or admit I wasn’t going to.

A check earns the right to block

Here’s the rule I settled on. A check earns the right to fail your build when two things are true: it’s meaningful (a red result is a real problem, not noise) and it’s reliable (it goes red only when there genuinely is a problem, and it can actually run to completion). Flip a check to blocking before both hold and you haven’t raised the bar, you’ve taught the team to force-merge past red, which is worse than no gate at all, because now the red means nothing.

Three of my six crossed that line within a few days. Three deliberately didn’t. The reasons are the whole story.

trivy: blocked once there was nothing to block on

trivy scans the dependency tree for HIGH and CRITICAL advisories. It went across as advisory for an honest reason: the Cargo.lock at migration time already carried two known HIGH/CRITICAL advisories I hadn’t cleared yet, a path-traversal in gix-validate and a DNS-rebinding issue in rmcp. Make trivy blocking with those sitting there and the pipeline is red from day one, over problems I already knew about and was already fixing. So it stayed advisory until the dependency bumps cleared both, and then the allow_failure line came out. The gate never changed. The tree underneath it got clean enough to stand on.

integration-tests: blocked once it could actually run

The integration tests stand up a real Gitea in a Docker-in-Docker service and talk to it. They were advisory for a different reason: they couldn’t reliably run. dind needs a privileged runner, and the suite was resolving the container host with a hardcoded 127.0.0.1 that didn’t hold everywhere. Blocking a job that fails for infrastructure reasons rather than code reasons is the fastest way to make people distrust the entire pipeline. So the fix wasn’t in the YAML, it was making the thing dependable: privileged set on the runner, and the host resolved through the test library’s own get_host() instead of a hardcoded address. Once it ran the same way every time, it earned the gate.

coverage: blocked once it could run at all, then once it cleared the bar

Coverage is the two-step one, and my favourite, because it nearly didn’t make it for a thoroughly undramatic reason: it ran out of memory. cargo llvm-cov instruments every test binary, and linking hundreds of instrumented object files needs more RAM than the shared medium runner had, so the job bus-errored on the link. I tagged it onto a larger runner, and then the shared SaaS runners were switched off entirely, so the tag matched nothing and the job sat pending forever.

The fix was a self-hosted homelab runner with the RAM the instrumented link actually needs. I moved coverage there but kept it advisory for one run, to confirm the box could finish the build before I trusted it. It did, at 73.22% line coverage, so I set the gate to fail under 70% and made it blocking. Three points of headroom: enough that ordinary churn won’t trip it, tight enough that a real drop will. A coverage gate pinned to the current number is a tripwire that fires on the very next commit; set it a touch below and it catches regressions instead of normal life.

The three I left advisory, on purpose

The point was never “block everything”. Three jobs are still allow_failure in the current pipeline, deliberately. The macOS and Windows tests run on SaaS runners that bill by the minute; they’re worth running, not worth blocking every merge of a Linux-first project over a quota I’m choosing to ration. And cargo-audit stays advisory because cargo-deny already does the blocking advisory check: cargo-audit is a second opinion from a different database, and a second opinion that can veto isn’t a second opinion, it’s a duplicate gate that will eventually disagree with the first and block you on the difference.

That’s the same rule from the other side. Those three haven’t earned the right to block, because blocking them would cost more than it ever caught.

The upshot

allow_failure: true is fine as a waiting room and corrosive as a destination. Every advisory check is a promise to make it blocking once it’s both meaningful and reliable, and the job is to keep the promise or admit you won’t. trivy earned its gate when the advisories cleared, the integration tests when they ran the same way every time, coverage when it had a runner with enough memory and a threshold set just below the current mark. The three I left advisory earned that standing too, by costing more to block than they’d catch. The YAML is one deleted line per job. Knowing which line to delete, and when, is the whole skill.

Two bugs that taught me the rules

Wed, 20 May 2026 00:00:00 +0000

Some bugs are interesting because they’re subtle. These two were interesting because they were the exact opposite… in each case the tool had a hard rule I simply didn’t know about, and its error message couldn’t be bothered to tell me what that rule was. Both came out of building the infrastructure toolchain, both cost me a good deal more time than they had any right to, and both are the sort of thing that looks blindingly obvious the moment you know it and utterly baffling until you do.

So here they are, written down, partly to save you the bother and partly so I don’t go and forget them myself.

Bug one: the rule-less job that skips your merge requests

The cicd gate components, in their first cut, shipped with no rules: block. They were dead simple jobs: lint, scan, validate. No conditions, because they should just always run. Obviously.

They ran on branch pipelines. On merge requests, they didn’t run at all! The gates that were the entire point of the components were simply absent from the one place you’d most want to see them… the merge request.

The cause is a GitLab CI rule that’s remarkably easy to go years without ever learning: a job with no rules: block runs only on branch and tag pipelines. It does not run on merge-request pipelines. So “no conditions” doesn’t mean “runs everywhere” at all. It means “runs everywhere except a merge request”, which is about the least intuitive default I can think of.

The fix is faintly absurd, and that’s exactly what makes it stick. You add an unconditional rule: rules: [{ when: on_success }]. The content of that rule does precisely nothing. It always matches. What actually matters is that the job now has a rules: block at all, because merely having one is what makes a job eligible for merge-request pipelines. A rule whose content is meaningless, added solely so the block exists. That’s the fix. I’ll admit I stared at it for a moment.

Bug two: the import block that only works at the root

The second one came from terraform-aws-security-baseline. The account-hardening module needed to adopt a resource that already existed in the account, which is exactly what OpenTofu’s import {} block is for. So an import block went into the account-hardening module, right next to the resource it was adopting. The natural home for it, surely.

OpenTofu disagreed, and rejected it outright. The rule: an import block is only allowed in the root module. It can’t live inside a child module. A module that wants one of its own resources imported can’t declare that import itself… the import has to be declared up at the root, and the root caller does the adopting.

The fix was to take the import block out of the module and document caller-side adoption instead. The module describes the resource, and the root configuration that calls the module is where the import actually lives.

Two unrelated bugs, in two completely different tools, and the same shape sitting underneath both of them.

In each case the tool has a hard structural rule. Where a block is allowed to live. What makes a job eligible for a particular kind of pipeline. And in each case the error told me the tool was unhappy without telling me which rule I’d broken, so the obvious next move (debugging my own logic) was the wrong move entirely. There was nothing wrong with the logic. The thing was simply in a place the tool doesn’t allow, or missing a block the tool quietly insists on.

The lasting lesson here isn’t the two specific rules, useful as they are to know. It’s the reflex. When something that should obviously work just doesn’t, and the error is unhelpful, stop debugging your logic and start suspecting a structural rule about where something is allowed to be, or whether a thing is eligible in the first place. GitLab CI and OpenTofu both have a handful of these, and you mostly learn them the hard way, by tripping over them. Knowing the shape of the category at least means the next one costs you an hour instead of a whole afternoon.

Worth remembering

Two bugs from building the toolchain, one shape. A GitLab CI job with no rules: block runs on branches and tags but silently not on merge requests, and the fix is an unconditional rules: block whose content does nothing and whose mere existence is the entire point. An OpenTofu import block gets rejected inside a child module, because imports are only legal at the root, so the caller adopts and the module just describes.

Neither error named the rule it was enforcing, and that’s the category to watch for. When sound logic fails against an unhelpful error, suspect a structural rule about where a thing may live or whether it’s even eligible… not a bug in what you actually wrote. It’ll save you an afternoon. It certainly cost me a couple.

Reviewed, then applied

Mon, 18 May 2026 00:00:00 +0000

The genuinely dangerous moment in infrastructure-as-code isn’t the apply. It’s the gap between the plan a human read and approved, and the change that actually runs a moment later. If those two are different computations (and by default they are) then nobody really reviewed the thing that touched your account. The infra repo closes that gap from both ends.

The gap between “reviewed” and “ran”

Here’s the moment in infrastructure-as-code where things go wrong.

Someone opens a merge request. CI runs tofu plan and the output is there to review: these three resources change, this one is destroyed. A human reads it, decides it’s correct, approves, merges. Then apply runs.

The trap is in what apply actually applies. If apply does its own fresh tofu plan and then applies that, the change that runs is not necessarily the change that was reviewed. State can have moved. A provider can have drifted. Someone else can have applied something in between. The reviewed plan and the applied change are two separate computations done at two different moments, and every difference between those moments is a change nobody looked at.

infra closes that gap from both ends.

Plan as an artifact

The first end is making the reviewed plan and the applied plan the same object.

The tofu-plan component runs the plan and saves it. It writes tfplan.cache, OpenTofu’s binary plan file, as a CI artifact. It also writes tfplan.json, which GitLab renders as a plan widget right in the merge request: the add, change and destroy summary, there to review without leaving the MR.

The tofu-apply component then does not re-plan. It applies that saved tfplan.cache. And OpenTofu itself enforces the safety net: applying a stale plan file, one captured against a state that has since moved, is rejected by the tool. So what reaches the account is provably the plan that was reviewed, or it’s nothing at all. There’s no third option where something unreviewed slips through.

Applying is a human decision

The second end is when apply runs.

infra is trunk-based: it dropped the develop branch and works on main. But a naive trunk setup auto-applies every push to main, which means there’s no human gate at all, just whatever the last merge happened to contain.

So the gate is built explicitly. releaser-pleaser keeps a release merge request open against main. Ordinary merges to main run plans but apply nothing. The apply happens only when a person merges the release MR. Merging it cuts a release tag, and the tag pipeline is what runs tofu-apply, against the plan banked by the latest main pipeline.

The effect is that the act of applying to the account is the deliberate, visible act of merging the release request. Nothing reaches the account because a commit landed. It reaches the account because a person decided a release should go out and merged it. (Which, after the accidental v2.0.0 that kicked off the whole GitLab move, is a discipline I’d freshly relearned the value of.)

The guard on the gate

There’s one more piece, because a gate is only as good as its precondition.

A verify-main-plan job blocks the release MR from being mergeable unless the latest main pipeline is green. You can’t cut a release, and therefore can’t apply, on top of a main whose plan didn’t even succeed. The human gate has its own gate: the thing you’re about to merge has to be standing on a known-good plan before you’re allowed to merge it.

The bottom line

The risk in infrastructure-as-code is the gap between the plan a human reviewed and the change that runs, because a re-plan at apply time is a different computation from the one that was approved.

infra closes it twice over. tofu-plan saves the plan as a tfplan.cache artifact and renders it as a merge-request widget; tofu-apply applies that exact artifact, and OpenTofu rejects it outright if the state has moved underneath it. And applying is gated on a human merging a releaser-pleaser release request, not on a push, with a verify-main-plan check making sure that request can only be merged on top of a green plan. What gets applied is what was reviewed, when a person decided it should be.

CI you include, not copy

Sat, 16 May 2026 00:00:00 +0000

Every infrastructure repo runs the same CI: lint the OpenTofu, scan it, validate it, plan, apply. The first repo, you write that .gitlab-ci.yml by hand. The second, you copy it. By the third, you’ve got three copies of the same pipeline quietly drifting apart, which is the exact problem you’d never tolerate in application code. The cicd repo is the fix, and it’s just the library-first instinct pointed at the pipeline.

The `.gitlab-ci.yml` you keep copying

The infrastructure repos in this series all run the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security issues and secrets, and on the deploy side, plan and apply.

The first repo, you write that .gitlab-ci.yml by hand. The second repo needs the same jobs, so you copy it. The third repo, you copy it again. Now there are three copies of the same pipeline, and they do what copies always do. They drift. A fix you make in one repo’s CI doesn’t reach the other two. A tightened scan rule lands in the repo you were working in and nowhere else. It’s the copy-paste problem, exactly as it shows up in application code, just written in YAML and therefore that bit easier to pretend isn’t code.

GitLab has a feature for exactly this

GitLab CI/CD Components are the answer to that problem. A component is a reusable, versioned piece of pipeline that you publish, and other projects pull in with an include: pinned to a version:

include:
 - component: gitlab.com/phpboyscout/cicd/tofu-lint@v0.5.0

That’s a library import, for pipeline. The component has a defined interface, a version, and a home in GitLab’s CI/CD Catalog. A consuming repo includes it instead of carrying its own copy, and when the component improves, the consumer moves a version pin rather than re-copying YAML.

Why a monorepo of components

The cicd repo holds all of the components together: tofu-lint, tofu-security, tofu-validate, tofu-plan, tofu-apply, and more. One project, not one project per component.

That’s a deliberate call, and the reason is how GitLab versions things. A version is a tag, and a tag belongs to a project. A component’s version is its project’s tag. So a monorepo of components, versioned together as one tag stream, is the natural unit: a consumer pins @v0.5.0 and gets a known-good set of components that were tested together, rather than juggling a separate version for each one.

Authoring discipline

A component is a file under templates/, and it opens with a spec: inputs: block: the typed inputs, their defaults, the component’s public interface.

The discipline that keeps the library usable is that a component must be consumer-agnostic. It never hardcodes a token, and it never names a particular consumer’s variable. Inputs have sensible defaults, and a consuming repo overrides them. A component that reaches out and assumes something about the repo including it is a component that works in one repo and surprises the next. An authoring guide in the repo keeps that consistent across everyone who adds a component.

The self-test you cannot fully write

The cicd repo tests its own components with a self-test pipeline. It’s worth knowing where that self-test stops.

When a repo tests its own components by running them in child pipelines, GitLab masks $CI_PIPELINE_SOURCE as parent_pipeline. A component’s rules:, which often branch on the pipeline source to behave differently for a merge request than for a branch or a tag, therefore can’t be exercised honestly by the self-test: the source they’d branch on has been flattened. The self-test covers what it can, and the component rules: are, in the end, validated by real consumers using them for real. That’s a genuine limit, and naming it is better than pretending the self-test proves more than it does. (It’s also, not coincidentally, the exact rules: quirk that bit me in one of the two bugs I closed the series with.)

The same instinct, again

This blog keeps circling the same instinct. go-tool-base exists because the same CLI scaffolding kept getting rewritten, so it was extracted into a library. cicd is that instinct pointed at the pipeline: the same gate jobs kept getting copied between repos, so they were extracted into a versioned, included library.

Stop copy-pasting. Publish, version, include. It’s true for CLI code, and it turns out to be just as true for the YAML that builds and ships it.

The gist

Every infrastructure repo needs the same CI, and copying the .gitlab-ci.yml between them produces copies that drift apart. GitLab CI/CD Components fix it: reusable, versioned pipeline that a repo include:s and pins, instead of carrying its own copy.

cicd is a monorepo of those components, versioned together as one tag stream, because GitLab tags a project and a component’s version is its project’s tag. Components are authored consumer-agnostic, with typed spec: inputs: and no hardcoded assumptions, and their rules: are validated by real use because the self-test can’t see the pipeline source. It’s the library-first instinct, applied to CI: publish it once, include it everywhere, fix it in one place.

One image for the whole toolchain

Fri, 15 May 2026 00:00:00 +0000

Every CI gate job across the infrastructure repos reaches for the same pile of tools: OpenTofu, tflint, trivy, checkov, gitleaks, terraform-docs, the AWS CLI. Installing that pile per job is both slow and quietly dangerous, because nothing pins it consistently. infra-tools is the obvious fix (one image, one source of truth for versions), but two of its build decisions are less obvious and worth a look: it publishes with crane instead of a second build, and it deliberately lets its own vulnerability scan fail.

The same pile of tools, in every repo

Every infrastructure repo in this series runs the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security problems and secrets, check the docs. Those jobs need a specific set of tools, and it’s the same set in every repo.

Install them per job and you pay twice. You pay in time, because every pipeline downloads and installs the whole set again. And you pay in drift, because unless every repo pins every tool identically, the repos slowly diverge on which version of trivy or tflint they actually run, and a check that passes in one repo fails in another for no reason anyone can see.

One image, one source of truth

infra-tools is the answer: a single Debian-based container image with the whole toolchain baked in. Every CI job in every repo uses it with one image: line.

The real value isn’t the convenience. It’s that the image is the one place tool versions are pinned. The Go-based tools are pinned in a mise.toml. checkov, which has no mise plugin, is pinned in a requirements file installed with pipx. The AWS CLI is pinned by a build argument. Three mechanisms, because the tools come from three kinds of source, but one image, and every pin wired to Renovate so a version bump arrives as a reviewable pull request. There’s exactly one answer to “what version of trivy does the toolchain use”, and it lives here.

Publishing with crane, not a second build

A build-pipeline detail that took a real bug to discover.

The pipeline builds the image with kaniko, which builds images without a privileged Docker daemon, something that matters a great deal on shared CI runners. Then it scans the image, then it publishes it.

The obvious way to write the publish stage is “build the image and push it”. But kaniko has no mode for “just push this tarball I already built”. A second kaniko invocation re-executes the entire Dockerfile from the top, including a second mise install, which makes a fresh round of calls to GitHub’s API to fetch tools. GitHub’s anonymous API limit is low and shared by IP, so on a CI runner that second install reliably trips a 403 rate-limit. (Yes, another 403. They do get everywhere.)

So the publish stage doesn’t rebuild. It uses crane to push the exact image tarball the build stage already produced. The image is built once. And because the published bytes are the same bytes the scan stage scanned, there’s no gap between “the image we checked” and “the image we shipped”.

Soft-failing the scanner on purpose

The decision that looks wrong until you see the reasoning: the pipeline scans the image with trivy, and trivy is allowed to fail without failing the pipeline.

A vulnerability scanner that doesn’t gate the build sounds like a scanner switched off. It isn’t. It’s a scanner pointed at something it can’t helpfully gate.

The tools in the image are prebuilt Go binaries. trivy inspects them, reads the version of the Go runtime each was compiled with, and reports every known CVE in that Go runtime. Those findings are real, but they aren’t mine to fix. The only fix is the upstream tool rebuilding itself against a patched Go. With seven such tools in the image, at any given moment one of them is usually a little behind on its Go version.

A hard gate would mean the image becomes unpublishable whenever any single upstream lags, over a CVE in code I don’t own and can’t patch. That’s not a security control; it’s a way to be unable to ship. So the scan is allow_failure. The findings stay fully visible, and the residual count is genuinely useful as a metric for how far behind upstream the toolchain has drifted. It just doesn’t block shipping an image whose only “vulnerabilities” are other people’s build timelines.

What it comes down to

The infrastructure repos all run the same CI gate jobs, needing the same tools, so infra-tools bakes the whole toolchain into one image and pins every version in one place, wired to Renovate.

Two build choices are worth copying. The publish stage uses crane to push the already-built, already-scanned tarball, because a second kaniko build would re-run mise install and hit GitHub’s anonymous rate limit, and because pushing the scanned bytes means shipping exactly what was checked. And the trivy scan is deliberately allow_failure, because it reports Go-runtime CVEs in prebuilt upstream binaries that no change to this repo can fix, so a hard gate would only make the image unshippable over someone else’s lag.

A 403 you can't fix in IAM

Thu, 14 May 2026 00:00:00 +0000

The OIDC post explained the handshake that lets a GitLab pipeline deploy to AWS with no stored key. This is the story of the first time I got it wrong, and spent an afternoon fixing the wrong thing. The error was a flat 403 from AWS, and the maddening part is that no amount of editing the IAM policy was ever going to fix it.

A 403 on the first real run

The OIDC post covered the handshake: GitLab CI mints a signed token, AWS exchanges it for short-lived credentials against a role whose trust policy names the pipeline. During the GitLab migration I wired exactly that up for the infra repo, including a trust policy condition meant to let merge-request pipelines run a plan.

The first merge request that should have triggered tofu-plan didn’t run it. The job failed, and the error from AWS was a flat AccessDenied. A 403.

The instinct, and why it wastes an afternoon

The instinct on an IAM 403 is immediate and almost always right: the policy’s wrong, so go and edit the policy. Tighten the condition. Loosen the condition. Check the wildcard. Re-read the sub pattern character by character.

All of that was wasted, and it was wasted for a reason that took me far too long to see. The trust policy wasn’t matching the wrong value. It was matching a value that does not exist. No amount of editing a condition makes it match a thing that’s never present.

What is actually in the token

GitLab’s OIDC token has a sub claim that encodes the pipeline’s context, and part of that encoding is a ref_type. I’d assumed ref_type could be branch, tag, or mr, because a pipeline can certainly be a branch pipeline, a tag pipeline, or a merge-request pipeline. So the trust policy, for the plan job, matched a sub containing ref_type:mr.

That assumption was wrong. GitLab’s ref_type is branch or tag. That’s the entire set. There is no mr.

A merge-request pipeline doesn’t run against a merge-request ref. It runs against the source branch. So its token’s sub carries ref_type:branch, like any other branch pipeline. The trust policy condition asked for ref_type:mr, GitLab never puts mr in a token, the condition was therefore never true, and every merge-request pipeline got a 403. Forever, until the policy stopped asking for a claim that isn’t real.

The fix, and the lesson worth more than the fix

The fix is small once it’s visible: match ref_type:branch and narrow it down by branch name or project path instead. An afternoon of policy edits, and the actual change is one word.

The lesson is the part worth keeping. When an OIDC trust fails, the useful question is never “is my policy clever enough”. It’s “what’s actually in the token”. An OIDC trust policy can only ever match the claims the identity provider genuinely asserts, and the gap between what a provider asserts and what you assumed it asserts is precisely where this class of bug lives.

So the move, when an OIDC handshake 403s, is to get hold of a real token and decode it. Look at the actual sub, the actual claims, the actual values. Match what’s there. A 403 that survives every sensible edit to the policy is usually not a policy that’s too loose or too strict. It’s a policy matching a claim that was never going to be in the token.

The habit it left behind

I wired an OIDC trust policy to let merge-request pipelines plan, by matching a sub claim with ref_type:mr. The first real merge request got a 403, and no edit to the policy fixed it, because GitLab’s ref_type is only ever branch or tag. A merge-request pipeline runs on a branch ref, so the mr value the policy demanded was never in any token.

The fix was one word. The habit it left behind is the valuable bit: when an OIDC trust fails, stop editing the policy and go and read a real token. A trust policy can only match what the provider actually asserts, and “what I assumed it asserts” is where the 403 was hiding the whole time. (If this shape of bug feels familiar by the end of the series, that’s not an accident: I come back to it with two more from exactly the same family.)

Why go-tool-base left GitHub for GitLab

Mon, 11 May 2026 00:00:00 +0000

A botched version bump made me stop and actually look at where go-tool-base lived, and I didn’t much like what I saw. GitHub had spent months quietly falling over, and when Mitchell Hashimoto (GitHub user #1299, no less) publicly walked Ghostty off the platform, it stopped feeling like just my problem. I’ve been a GitLab fan for years, so the move was less a leap and more an overdue nudge. This is the why, not the how.

It started with a wrong number

Every migration has a trigger, and mine was embarrassingly small. A commit landed on main carrying a BREAKING CHANGE: footer it didn’t really deserve. Semantic-release did exactly what it’s told to do with that footer: it cut a major version. go-tool-base lurched from the v1 line straight to v2.0.0, and a chain of things that keyed off the version went sideways with it.

It was fixable. It wasn’t a disaster. But it was the kind of small, stupid breakage that makes you stop and actually look at your setup instead of just patching it and moving on. And when I looked, the version bump wasn’t the thing that bothered me. It was everything around it.

The platform had been quietly failing

I’d been losing time to GitHub for months. Not dramatically. No single outage you’d write home about, just a steady drip of Actions queues that wouldn’t drain, pull requests that wouldn’t merge, the occasional morning where the thing simply wasn’t there. You absorb it. You re-run the job. You make a coffee and try again. You tell yourself it’s a blip.

The trouble with a steady drip is that you stop counting it. It becomes weather.

The canary left the mine

Then, in late April, Mitchell Hashimoto (co-founder of HashiCorp, creator of Vagrant, Terraform and the Ghostty terminal) published Ghostty Is Leaving GitHub, and The Register picked it up a day later under the headline “GitHub ’no longer a place for serious work’”.

This is not a man with a casual relationship to GitHub. He’s, by his own account, user #1299, joined February 2008. He called it “the place that has made me the most happy”. And he still wrote this:

This is no longer a place for serious work if it just blocks you out for hours per day, every day.

The detail that landed hardest for me wasn’t a quote, it was a habit. He’d kept a journal for a month, marking an “X” on every day a GitHub outage had cost him working time. Almost every day had an X. Reading that, I realised I’d been having the same month. I’d just never been disciplined enough to write it down. He’d turned my vague “it’s been flaky lately” into a row of crosses on a calendar.

I want to ship software and it doesn’t want me to ship software.

When the person who’s been on the platform for eighteen years and loves it says that out loud, it stops being your private grumble. It’s the canary, and the canary has stopped singing.

Why GitLab, and not just “somewhere else”

Being annoyed at GitHub is a reason to leave. It is not, on its own, a reason to pick a destination. The destination has to be a positive choice.

For me GitLab was an easy one, because I’ve been a fan for years. Long enough, in fact, to have also been a reliable grumbler about their pricing tiers, which is how you know it’s a real relationship and not a honeymoon. What I’ve always rated is the model: GitLab treats source hosting, CI/CD, the package registry, releases and Pages as one integrated product, not a marketplace of bolted-on parts you assemble yourself.

That integration is the actual prize. On the old setup, “CI” meant a folder of separate GitHub Actions workflow files, each pinned, each its own little world. On GitLab it’s a single .gitlab-ci.yml pipeline with proper stages (lint, test, security, docs, release) and the release stage talks to the built-in package registry and Pages without me wiring up a single external credential. The CI job that builds the project can authenticate to the things the project needs because they’re the same platform.

There’s a second-order benefit too. A migration is a rare licence to fix things you’d never otherwise touch. Moving gave me the cover to reset go-tool-base’s versioning cleanly (back to a sensible v0.x line, the accidental v2.0.0 left behind as a cautionary tale) and to move the module path to its new home in one deliberate change rather than a thousand apologetic ones.

What I’m not going to claim

I’m not going to tell you GitHub is finished, or that GitLab never has a bad day, because it does, everyone does. This isn’t a teardown. GitHub gave go-tool-base a perfectly good home for its first year, and the archived mirror is still sitting there, read-only, pointing anyone who finds it at the new place.

What changed is simpler than a grand verdict. The friction crossed a line, someone I respect said the quiet part loudly enough that I couldn’t keep filing it under “weather”, and the place I’d have moved to anyway was sitting right there with a better model. Sometimes the prudent move and the move you secretly wanted turn out to be the same move, and you just need a wrong version number to give you permission.

Boiling it down

go-tool-base moved from GitHub to GitLab in May 2026. The proximate cause was a self-inflicted version-bump mess; the real cause was months of GitHub unreliability that I’d stopped consciously noticing until Mitchell Hashimoto’s very public departure named it for me. GitLab was a positive pick, not just an escape hatch: its integrated CI/CD, registry, releases and Pages are one product rather than a kit, and that integration is genuinely worth having. The migration also bought a clean versioning restart as a bonus.

If you’ve been absorbing a steady drip of friction and telling yourself it’s normal: try the calendar trick. Mark the X’s for a month. The page will tell you something you already half-know.

No access keys in CI

Fri, 08 May 2026 00:00:00 +0000

A long-lived AWS access key, sitting in a CI system, is just about the single credential I’d most like to be rid of. It’s powerful, it never expires unless someone remembers to rotate it (nobody remembers to rotate it), and it lives in one of the most attractive targets in the whole supply chain. For infrastructure that’s eventually going to hold a release-signing key, it’s exactly the wrong place to start. So the phpboyscout infrastructure has no AWS access key in CI at all. None.

The access key you don’t want

A CI pipeline that runs tofu apply against AWS needs AWS credentials. The traditional way to give it some is an IAM user with an access key pair, pasted into the CI system as a masked variable.

Look at what that key is. It’s long-lived: it works until someone remembers to rotate it, and rotating it is a chore, so mostly nobody does. It’s powerful: it can apply infrastructure, so it can do nearly anything. And it’s sitting in a CI system, which is one of the most attractive targets in your whole supply chain. You’ve taken your highest-value credential and stored a permanent copy of it in a place built for running automated jobs.

For infrastructure that’s going to hold a release-signing key, that’s precisely the wrong starting point. So the phpboyscout infrastructure has no AWS access key in CI at all. Not a well-guarded one. None.

Federation instead of a stored secret

The replacement is OIDC federation, and the shape of it is worth walking through, because it’s genuinely different from “a secret, but better”.

A modern CI platform can mint an OIDC token. GitLab does this with an id_tokens: block: at job time, GitLab issues a short-lived JSON Web Token, signed by GitLab, that asserts a set of facts. This is project X. This is pipeline Y. This is running on ref Z, of this type.

AWS can consume that. The sts:AssumeRoleWithWebIdentity call takes such a token and, if it satisfies an IAM role’s trust policy, returns short-lived AWS credentials for that role. The trust policy is where the control lives: it names GitLab as a trusted token issuer, and it constrains the token’s sub claim so that only the specific project, and the specific refs, you intend can assume the role.

Put it together: the pipeline asks GitLab for a token, hands it to AWS, and gets back credentials that last about an hour and are scoped to one role. Nothing long-lived is stored anywhere. The credential exists only for the job that needs it, and it can’t be stolen from a CI variable store, because it was never in one.

Two halves of one handshake

That handshake is built by two of the repos in this series, each owning one side.

terraform-aws-bootstrap builds the AWS half, in its automation-iam module: it registers GitLab as an OIDC identity provider in the account, and it creates the automation role with the trust policy that decides which pipelines may assume it.

The CI components build the consuming half: the id_tokens: block that asks GitLab for the JWT, and then simply letting the AWS provider’s own credential chain perform the exchange. The pipeline doesn’t call sts by hand. It presents the token; the SDK does the rest.

The gotcha: don’t set a profile

There’s one quiet way to break this, and a stack can look completely correct while doing it.

The AWS SDK finds credentials by walking a chain of sources in order. The web-identity path, the one that uses the OIDC token, is one link in that chain. It triggers off environment variables the CI sets up automatically.

But if the aws provider block has a hardcoded profile = "...", the SDK takes the profile link of the chain instead, and never reaches the web-identity link. A profile line is the sort of thing that ends up in a provider block from someone’s local development setup, where it’s exactly right. Committed and run in CI, it silently short-circuits the federation. The pipeline either fails to find credentials, or finds the wrong ones.

The rule is simple once you know it: the provider block that runs in CI must not name a profile. Leave the chain free to find the web identity. It’s the kind of bug that teaches you to be precise about which link of the credential chain you’re actually relying on.

The bottom line

Giving CI an AWS access key means storing your most powerful, longest-lived credential in one of your most exposed systems. OIDC federation removes it entirely. The CI platform mints a short-lived signed token, AWS exchanges it via AssumeRoleWithWebIdentity for hour-long credentials against a role whose trust policy names the exact pipeline, and nothing permanent is stored.

terraform-aws-bootstrap builds the AWS side, the identity provider and the trust policy; the CI components build the consuming side, the token request. The one trap is a hardcoded profile in the provider block, which short-circuits the SDK’s credential chain before it reaches the web-identity path. Get that right, and a pipeline deploys to AWS as a verifiable, short-lived identity, with no key to steal.

OpenSSF Scorecard graded my supply chain

Tue, 14 Apr 2026 00:00:00 +0000

I turned OpenSSF Scorecard on expecting a pat on the head. go-tool-base is a security-minded project, I’m careful, surely the robot would agree. The robot did not agree. It handed back a report card with a fair bit of red ink, and the most pointed finding on it wasn’t about my code at all. It was about me.

A linter for the things you don’t call code

Scorecard is an automated set of checks that grades a repository’s supply-chain hygiene: are your CI dependencies pinned, are your workflow tokens least-privilege, is your branch protected, do commits get reviewed. It’s a linter, but pointed at the part of the project you don’t usually think of as code, the build and release machinery and the practices around them. And like any good linter, its value is mostly in catching the things you’d swear you’d already got right.

Three of its findings were worth the price of admission on their own.

Pin the actions you don’t control

The first was about how go-tool-base’s GitHub Actions referenced other actions. Like nearly everyone, I’d written uses: actions/checkout@v6. Scorecard doesn’t like that, and it’s right not to.

@v6 is a tag, and a tag is mutable. Whoever controls that action can move v6 to point at different code tomorrow, and your CI will pick it up silently on the next run. For an action that runs in a job holding your repository token, that’s a supply-chain hole the width of a barn door: compromise the tag, compromise every pipeline that trusts it. The fix is to pin to an immutable commit SHA, with the human-readable version left as a comment, which is exactly what I changed:

-  - uses: actions/checkout@v6
-  - uses: actions/setup-go@v6
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+ - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6

Now the action is frozen at bytes I reviewed. Dependabot still bumps the SHA when a real new version lands, so I get updates as reviewable pull requests rather than as silent tag movements. The pin doesn’t stop me updating. It stops me updating without noticing.

Give the workflow token the least it can do

The second finding was about permissions. My workflow declared its token permissions at the top, once, for the whole file:

permissions:
 contents: read
 security-events: write
 id-token: write

That reads as careful, and it’s still too broad, because top-level permissions apply to every job in the workflow. A job that only needs to read the repo is now also holding id-token: write and security-events: write, for no reason other than that some other job in the same file needed them. Scorecard rejects exactly this, and the fix is to default the whole workflow to read-only and grant write narrowly, in the job that actually needs it:

permissions: read-all

Write permissions moved down into the single job that uses them. It’s the same least-privilege instinct that runs through everything else in these projects, just applied to a CI token instead of an IAM role: a credential should be able to do the one thing it’s for, and nothing else, no matter how convenient the broad grant looked.

The finding that was about me

The third one stung, because there was no YAML to fix. Scorecard’s Code-Review check scores how consistently changes are reviewed before they land, and mine scored badly for the most embarrassing possible reason: I’d set up branch protection on main, and then, being the solo maintainer in a hurry, I’d been merrily bypassing it to push straight to main whenever it suited me.

So I had a rule, written down and enforced by the platform, that I was personally and routinely ignoring. Scorecard noticed, totted up the unreviewed commits, and graded me on it. There’s something properly humbling about a robot reading your git history and pointing out that the person breaking your security policy most often is you. The fix wasn’t code. It was going through a pull request like everyone else, even when “everyone else” is just me on a different day.

The bottom line

OpenSSF Scorecard is a linter for your supply chain, and like any linter it’s most useful when it tells you something you were sure you’d already handled. It dinged go-tool-base for referencing actions by mutable tag instead of pinned SHA, for granting workflow-token write permissions at the top level where every job inherited them, and for a Code-Review score I’d earned fair and square by bypassing my own branch protection.

The first two were quick, satisfying changes with a clear security story. The third was the one that stuck, because the tool I’d added to grade the project ended up grading the maintainer, and was entirely right to. Turn it on. Brace yourself a little.

CI/CD on PHP Boy Scout

Stop installing the same tools on every pipeline job

The build gate these sites never had

Three copies of the same job

The gate they were missing

The one component that wants to run on a schedule

A gate you can switch off

Where it leaves things

The secret that wasn't on my branch

Prove it isn’t yours

What gitleaks actually scans

Unblock now, fix properly after

Fixing the fix

What a scanner is actually looking at

Sign your own binaries with go-tool-base, part 6: sign every release with GoReleaser

One command wires the signing in

Why only the checksums

No shim, just gtb

Where the credentials come from

Don’t sign when there’s nothing to sign with

Cut a release

A two-person gate, if you want one

Where this leaves you

Three traps release-plz sets for a Rust workspace

First, what release-plz is doing

Trap 1: the default tag template is built for one crate, not a workspace

Trap 2: “one release for the whole workspace” isn’t a setting, it’s a category error

Trap 3: a release reads its config from the release commit, not HEAD

If you set one thing

Same config, two answers

Rust brings its own toolchain

The gauntlet before the gun

“Tag v0.5.1 already exists”

I assumed the default

Same config, two answers

Doing it the way I’d have done it a year ago

Stop being clever and RTFM

From allow_failure to blocking

What allow_failure actually buys you

A check earns the right to block

trivy: blocked once there was nothing to block on

integration-tests: blocked once it could actually run

coverage: blocked once it could run at all, then once it cleared the bar

The three I left advisory, on purpose

The upshot

Two bugs that taught me the rules

Bug one: the rule-less job that skips your merge requests

Bug two: the import block that only works at the root

The shape they share

Worth remembering

Reviewed, then applied

The gap between “reviewed” and “ran”

Plan as an artifact

Applying is a human decision

The guard on the gate

The bottom line

CI you include, not copy

The .gitlab-ci.yml you keep copying

GitLab has a feature for exactly this

Why a monorepo of components

Authoring discipline

The self-test you cannot fully write

The same instinct, again

The gist

One image for the whole toolchain

The same pile of tools, in every repo

One image, one source of truth

Publishing with crane, not a second build

Soft-failing the scanner on purpose

What it comes down to

A 403 you can't fix in IAM

A 403 on the first real run

The instinct, and why it wastes an afternoon

What is actually in the token

The fix, and the lesson worth more than the fix

The habit it left behind

Why go-tool-base left GitHub for GitLab

It started with a wrong number

The platform had been quietly failing

The canary left the mine

No shim, just `gtb`

The `.gitlab-ci.yml` you keep copying