GitLab on PHP Boy Scout

Stop installing the same tools on every pipeline job

Sun, 19 Jul 2026 00:00:00 +0000

Every Go and Rust pipeline I run starts the same way, and for a long time the first thing each job did was waste a minute of my life. Before it could lint or test or scan, it had to go and fetch the tools to do those things: cargo binstall for nextest, cargo-deny, cargo-audit and llvm-cov, a go install for govulncheck, and so on. Every job. Every push. Every branch. Every Renovate bump.

On a hosted runner you’d barely notice. On mine you very much do, because mine is a single self-hosted box in the corner doing all of this with a small concurrency limit. That’s the budget I work within, and it changes the maths. A minute of tool-fetching isn’t a minute, it’s a minute multiplied by every job in every pipeline, all contending for the same runner, all doing the identical download-and-compile that the job before it just did and threw away. It was, comfortably, the single biggest recurring time sink in the whole setup, and none of it was the actual work.

The fix is the obvious one once you stop treating each job as an island: bake the tools into the image. There’s now a dev-tools image with the whole kit already installed, and the Go and Rust components default to it. The per-job installs are gone. A job starts and the tools are simply there. It’s the largest recurring-time win I’ve made on that runner, and it’s not close.

Here’s the part worth slowing down for, though, because it’s where I’d grab your sleeve if you were about to go and copy this: what you can’t bake.

You’d think you’d just throw everything in the image and be done. But there’s a line, and it runs between binaries and toolchain components. A tool like nextest or govulncheck is a standalone binary: it doesn’t care which Rust or Go toolchain the project uses, so baking it in is free and correct. A rustup component, though, like clippy or rustfmt, is tied to a specific toolchain. And a consumer’s repo can pin its own toolchain with a rust-toolchain.toml, which silently overrides whatever toolchain the image was built against. Bake clippy for toolchain A into the image, let a project ask for toolchain B, and you’ve got a clippy that doesn’t match the compiler it’s checking against. So rustup component add deliberately stays at runtime. You bake what’s toolchain-independent and you leave what isn’t. The rule is small but it’s the difference between a cache that helps and a cache that lies.

Which brings me to the bit I got wrong, because I always get one bit wrong.

The first cut of the image, v0.1.0, ran as a non-root user. That’s the textbook call. You don’t run containers as root if you can avoid it, every hardening guide says so, and I’d internalised it well enough to do it without thinking. It promptly broke every Go and Rust job. Permission-denied reading the module cache. Permission-denied on /etc/machine-id. The works.

The cause is one of those collisions between two reasonable things. GitLab’s Docker executor mounts and owns its caches in a particular way, and that ownership assumes the job can write where it expects to. A non-root user in the image doesn’t line up with how those cache volumes are owned, so the “secure” default walks straight into permission errors on the exact paths the build needs. v0.1.1 reverts the image to root.

I don’t love writing that sentence. Reverting to root as a “fix” reads like giving up on a security control. But the reality is that the security posture of a build image and the ownership model of the executor’s caches are two separate systems, and on a self-hosted GitLab runner the second one wins whether you like it or not. The textbook default was textbook for a context that isn’t this one. Better to know exactly why root is required here, and write it down, than to ship a non-root image that fails every pipeline and call it secure.

Bake the shared work in once and you stop paying for it on every job, which claws back more runner time than any other single change I’ve made. Just don’t get carried away and bake in everything, because a toolchain component isn’t a binary, and a hardened default stops being free the moment it fights the executor underneath it. Both of those I found out by breaking my own pipelines, which is, I’m reliably told, the traditional way.

The build gate these sites never had

Mon, 06 Jul 2026 00:00:00 +0000

I run three Hugo sites now. This blog, my dad’s B&B, and the vanlife site my wife and I are slowly putting together. All three build with Hugo and deploy to GitLab Pages, and until recently all three had the same blind spot: the build only ever ran on main.

Which means that if I pushed something that broke the build, a typo in a shortcode, a cross-link to a page that doesn’t exist, a theme bump that didn’t take, I found out after it landed. The change was already merged, sitting on the default branch, and the deploy was the thing that fell over. No gate, no warning, nothing telling me on the merge request that the site wouldn’t build. Just a red pipeline on main and a site that had just stopped updating.

Three copies of the same job

I’ve written before about pulling shared CI into components you include rather than copy-paste, and this was a textbook case of why. Each of the three sites had its own hand-rolled pages job: the same pinned hugomods/hugo image, the same GIT_DEPTH: 0, the same nightly rebuild, all written out three times and drifting apart a little more with every tweak. I already had a zensical-pages component for the docs framework I use elsewhere, but nothing covered a plain Hugo build.

So I wrote one: a hugo-pages component in my shared cicd project, and pointed all three sites at it.

The gate they were missing

The interesting part isn’t the deduplication, it’s what the component fixes. It splits the work into two jobs, and that split is the entire reason the component exists.

hugo-build builds the site to public/, and it carries an explicit rule so it runs in merge-request pipelines too:

rules:
 - if: '$CI_COMMIT_TAG'
 when: never
 - if: '"$[[ inputs.mr_gate ]]" == "true"'
 when: on_success

That second line is the bit the old jobs never had. Push a broken Hugo build now and it shows up red on the merge request, where you can see it before merging, instead of sailing onto main and leaving the deploy as the only thing left to fall over. It’s interruptible: true, so pushing a fix cancels the in-flight run rather than queuing behind it. The deploy is a separate job, pages, which only runs where it should:

pages:
 variables:
 GIT_STRATEGY: none
 rules:
 - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
 when: never
 - if: '$CI_COMMIT_BRANCH == "$[[ inputs.deploy_branch ]]"'
 - if: '$CI_PIPELINE_SOURCE == "schedule"'

It takes the artifact hugo-build already produced and republishes it, with GIT_STRATEGY: none because there’s nothing to clone. Build everywhere, catch the breakage early; deploy only from main (and one other place, which is where it gets interesting). The component is pinned at v0.13.0.

The one component that wants to run on a schedule

Here’s the bit I like. Every other component in my cicd project carries a leading guard that says never run on a schedule, because a nightly Renovate schedule firing a release job or a deploy is exactly the kind of pointless busywork you don’t want.

hugo-pages does the opposite, on purpose. It deploys on schedule, and the reason is sitting in plain sight in the component itself:

# Schedule note: unlike every other component (which carries a
# `schedule → never` guard ...), hugo-pages DELIBERATELY deploys on
# `schedule`. A nightly pipeline schedule is the mechanism by which
# future-dated, non-draft posts go live once their date arrives (Hugo
# excludes future-dated content from `production` builds until then).
# The schedule IS the feature.

When I write a post and date it next Tuesday, Hugo’s production build simply doesn’t include it yet, because its date is in the future. Nothing publishes it on the day except a build that runs on the day and finds that the date has now arrived. That’s the nightly schedule. It’s not noise to be guarded against here, it’s the publish button. The post you’re reading went live exactly this way: written days early, dated forward, and pushed out by a scheduled pipeline the morning its date came up.

One codebase, two flatly opposite rules about the same trigger, and knowing which site wants which is the actual design.

A gate you can switch off

There was one wrinkle. The new MR build gate is the right default for a docs site with contributors, but for these three blogs it was friction. I run them trunk-based: I work on main, I audit locally, I push. A mandatory merge-request build for a workflow that doesn’t really use merge requests is a job that mostly just sits there.

So the gate is a boolean you can turn off:

mr_gate:
 type: boolean
 default: true

Left at its default, hugo-build runs everywhere and the gate is on, which is the safe posture for anyone adopting the component fresh. Set it to false, as this blog does, and hugo-build falls through to running only where it deploys, matching the classic Pages-only-on-main setup the sites had before, but now without three copies of the YAML to maintain. Safe by default, loose by choice, and the choice is one line.

Where it leaves things

A reusable component instead of three drifting copies, a build gate that catches a broken site before it merges, a schedule that publishes tomorrow’s post on time rather than firing for nothing, and an opt-out for the trunk-based sites that don’t want the gate. None of it is clever. It just means the next time I fat-finger a shortcode, I find out on the merge request and not from my dad asking why the B&B site is showing a stack trace.

The secret that wasn't on my branch

Sat, 04 Jul 2026 00:00:00 +0000

My CI went red on a secret I’d never committed.

Not a close call, not a near-miss I’d half-forgotten about. gitleaks, the secret scanner, failed a merge request of mine on a private key that was not on my branch, was not in my change, and as far as I could tell had nothing to do with me at all. The job was adamant. I was baffled. Somewhere in between was a lesson about what a secret scanner actually scans.

Prove it isn’t yours

First rule of being accused: don’t get defensive, get evidence. The scanner pointed at a couple of commits carrying a test private key and a PEM block in a spec document. I genuinely didn’t recognise them, but “I don’t recognise it” is a feeling, not a fact, and feelings don’t reopen a pipeline.

Git will tell you the truth if you ask it precisely. The question is: are these flagged commits actually part of my branch?

git merge-base --is-ancestor <flagged-sha> HEAD

That asks “is this commit an ancestor of where I am?”. The answer came back no. The commits the scanner was choking on were not in my history. They weren’t mine.

So whose were they? A bit of digging turned them up on a completely separate, still-unmerged branch, where someone (me, a few days earlier, on a different feature) had committed a throwaway test key and a PEM example in a spec, on purpose, as fixtures. Deliberate, harmless, and nowhere near the branch under review. And yet here they were, failing a merge request that had never touched them.

What gitleaks actually scans

Here’s the bit I’d taken for granted. I assumed gitleaks detect scanned my change. It doesn’t. With no further instruction it scans the whole history reachable in the checkout it’s handed.

And the checkout it’s handed is where the second half of the surprise lives. GitLab runners default to GIT_STRATEGY: fetch, which reuses the runner’s working directory between jobs rather than cloning fresh every time. It’s faster, and most of the time you never notice. But it means a shared runner accumulates refs from every branch it has ever built. My MR’s job happened to run on a runner that had, at some point, built that other branch, so the fixtures were sitting right there in the local object store, fair game for a scanner walking the whole graph.

So gitleaks did exactly what I’d asked it to do, which was “scan everything”, and “everything” turned out to be a great deal more than my change. It walked the lot and dutifully reported fixtures from a branch I wasn’t even proposing to merge. The scanner wasn’t wrong. My idea of what it was looking at was.

Unblock now, fix properly after

Two problems on two timescales. I needed the MR to merge today, and I needed this to never happen again. Those want different fixes.

The immediate one: tell gitleaks those specific fixtures are known and intended. They’re test material, they’re meant to be there, so they go in the allowlist:

paths = [
 # ...
 '''internal/cmd/keys/keys_test\.go''',
 '''docs/development/specs/2026-06-08-keys-mint-command\.md''',
]

That unblocks the merge. It does nothing about the root cause, which is that the scan was looking at the wrong commits in the first place. Allowlisting individual false positives as they crop up is closing the stable door after the horse has bolted, one horse at a time, forever.

The real fix lives in the shared CI component, not in any one repo. Scope the scan to the commits the merge request actually introduces (cicd v0.10.3):

if [ -n "$CI_MERGE_REQUEST_DIFF_BASE_SHA" ]; then
 gitleaks detect --source . --verbose --redact \
 --log-opts="$CI_MERGE_REQUEST_DIFF_BASE_SHA..$CI_COMMIT_SHA"
else
 gitleaks detect --source . --verbose --redact
fi

--log-opts is handed straight through to git log, so that range, base-of-the-MR to tip, is the exact set of commits the merge request adds and nothing else. On a merge request the scan now sees only what you’re proposing to merge. Off a merge request (a plain branch or a tag pipeline) it falls back to the full scan, because there you genuinely do want the lot. The before-and-after in the job log tells the whole story: the entire accumulated history on one side, the handful of commits you actually wrote on the other.

Fixing the fix

There’s a tax on touching CI shell, and I paid it. The change went into the go, rust and tofu security templates. Go and tofu went green. Rust failed.

The rust template had built its optional --config flag the clever way, inline:

... $([ -n "$CONFIG" ] && echo "--config $CONFIG")

As a command argument, that substitution is harmless: its exit status is thrown away, so nobody cares that the test inside returns false when there’s no config. But when I rewrote the block and reached for the same pattern as an assignment, VAR=$(... && ...), it became a different animal. An assignment takes the exit status of the command substitution, and under set -e a non-zero status anywhere aborts the job. So on every run where the config was empty, which was most of them, the test returned false, the assignment inherited that false, and set -e killed the job stone dead. Same $(...), two completely different fates, decided entirely by whether it sat to the right of an = or got handed to a command as an argument. Go and tofu never used the assignment form, so only rust fell down the hole.

The fix was to stop being clever and write the boring if:

GITLEAKS_CONFIG=""
if [ -n "$CONFIG" ]; then
 GITLEAKS_CONFIG="--config $CONFIG"
fi

Boring shell is good shell.

What a scanner is actually looking at

The whole mess came from one unspoken assumption: that a tool called to scan “my change” was scanning my change. It was scanning a checkout, and a checkout on a shared runner is a much bigger, messier thing than the diff in front of you. None of the pieces were broken. gitleaks did its job, GIT_STRATEGY: fetch did its job, my fixtures were exactly where I’d left them. They just added up to a red pipeline that had nothing to do with the code I was trying to ship. I’d spent a good chunk of the day proving my innocence to a scanner that was only ever doing as it was told… and the one thing I’d actually got wrong was being sure I already knew what it was looking at.

Sign your own binaries with go-tool-base, part 3: keyless CI signing with OIDC

Fri, 19 Jun 2026 00:00:00 +0000

Part 2 left you with a KMS key your release pipeline can sign through and a role (<name>-signer) that’s allowed to call kms:Sign and nothing else. There’s one obvious question left hanging: how does a CI job become that role without an AWS access key stashed in a CI variable? Because a long-lived key sitting in your settings is exactly the kind of credential that ends up in a breach write-up. This part wires CI in with no stored credentials at all, on GitLab and GitHub.

The mechanism is OIDC federation, and the one-sentence version is: your CI platform already proves who it is to AWS, so let it. When a pipeline runs, GitLab or GitHub can hand the job a short-lived signed token (a JWT) that says “this is a tag pipeline on acme/acme-cli”. AWS trusts that token the same way a bouncer trusts a passport: it checks who issued it and what it says, and if the claims match a role’s trust policy, it swaps the token for temporary AWS credentials that live only for the job’s run. No key is stored anywhere; the credentials are minted on the spot and evaporate when the runner stops. The deep-dive on keyless CI covers the why and the threat model; here we just do the wiring.

Two things have to line up for this to work: the IAM OIDC identity provider (the thing in your AWS account that says “I trust tokens from this issuer”), and the signer role’s trust policy (which says “and only from these pipelines”). The terraform-aws-signing-kms module owns the second. The first comes from a sibling module.

Where the identity provider comes from

You don’t register the OIDC provider in the signing module. That’s deliberate: the same provider is shared across every role in the account that federates from CI, so it lives one level up, in terraform-aws-bootstrap. It takes a ci_provider input (github by default, or gitlab), registers the right IAM OIDC identity provider for that forge, and emits its ARN. You feed that ARN into the signing module. If you ran the bootstrap in Part 2 you already have it; if not, stand it up first and grab the oidc_provider_arn output.

That ARN is the only thing the two modules need to agree on. Everything else about which pipelines may sign lives in the signing module’s trust policy, which is what the rest of this part configures.

The GitLab path

Look the provider up by URL so you never hardcode the ARN, then pass it in along with the subject filter that scopes who can assume the role:

data "aws_iam_openid_connect_provider" "gitlab" {
 url = "https://gitlab.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn

 # oidc_issuer_host defaults to "gitlab.com"
 # oidc_audience defaults to "sts.amazonaws.com"

 ci_subject_filters = [
 "project_path:acme/acme-cli:ref_type:tag:ref:v*",
 ]

 key_administrator_arns = [/* ... from Part 2 ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

The interesting line is ci_subject_filters. GitLab stamps each CI token’s sub claim with the project path, the ref type and the ref. The pattern above reads as “tag pipelines on acme/acme-cli, for any ref starting v”. A branch pipeline or a merge-request pipeline carries ref_type:branch instead, so it simply doesn’t match, and the role refuses to be assumed. Your signer can only be driven from a release tag, which is the whole point: a dependency author opening an MR can’t trick CI into minting a signature.

On the pipeline side, the release job declares an id_tokens block so GitLab issues a token with the right audience, writes it to a file, and the AWS SDK picks it up:

goreleaser:
 rules:
 - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
 id_tokens:
 AWS_WEB_IDENTITY_TOKEN:
 aud: sts.amazonaws.com
 variables:
 AWS_REGION: eu-west-2
 AWS_ROLE_ARN: ${SIGNER_ROLE_ARN}
 AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/aws-token
 before_script:
 - echo "$AWS_WEB_IDENTITY_TOKEN" > /tmp/aws-token
 script:
 - aws sts get-caller-identity
 - goreleaser release --clean

AWS_ROLE_ARN plus AWS_WEB_IDENTITY_TOKEN_FILE is the convention the SDK recognises: it sees the two together and calls assume-role-with-web-identity for you, so by the time goreleaser runs it’s already the signer role. The actual signing job is Part 6; the aws sts get-caller-identity line is just a sanity check that federation worked. It should print the signer role’s ARN.

The GitHub path

Same shape, different issuer and a different sub format. GitHub’s sub support landed in module v0.1.2: earlier versions validated ci_subject_filters against GitLab’s format only and would reject a GitHub subject outright, so pin the version.

data "aws_iam_openid_connect_provider" "github" {
 url = "https://token.actions.githubusercontent.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.github.arn
 oidc_issuer_host = "token.actions.githubusercontent.com"

 # oidc_audience still defaults to "sts.amazonaws.com"

 ci_subject_filters = [
 "repo:acme/acme-cli:ref:refs/tags/v*",
 ]

 key_administrator_arns = [/* ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

Two differences from GitLab. The oidc_issuer_host has to change, because it’s the prefix on the trust-policy condition keys (token.actions.githubusercontent.com:sub rather than gitlab.com:sub). And the subject format is GitHub’s own: repo:<owner>/<repo>:ref:refs/tags/v* scopes the same way the GitLab pattern did, to tag refs only. The audience stays sts.amazonaws.com, because that’s the default aws-actions/configure-aws-credentials requests, so there’s nothing to override.

The workflow side is the official AWS action. It needs id-token: write permission to ask GitHub for the token in the first place, and contents: write so GoReleaser can create the release:

permissions:
 id-token: write
 contents: write
jobs:
 release:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: aws-actions/configure-aws-credentials@v4
 with:
 role-to-assume: ${{ secrets.SIGNER_ROLE_ARN }}
 aws-region: eu-west-2
 audience: sts.amazonaws.com
 - run: aws sts get-caller-identity
 - run: goreleaser release --clean

If you forget id-token: write, GitHub never issues a token and the action fails before it reaches AWS. It’s the most common GitHub-side trip-up.

The gotcha that wastes an afternoon: token audience

Here’s the one worth burning into memory, because the error message points you at the wrong thing. An IAM OIDC provider carries a client_id_list, and it rejects any token whose aud claim isn’t on that list before AWS even looks at the role’s trust policy. So if your CI token’s audience and the provider’s client ID don’t match, you get:

InvalidIdentityToken: Incorrect token audience

That reads like a role-trust problem, and you’ll go round in circles editing ci_subject_filters, but the trust policy was never consulted. The fix is to keep the audience aligned to sts.amazonaws.com everywhere: it’s the module’s oidc_audience default, it’s what terraform-aws-bootstrap puts on the provider’s client_id_list, it’s the aud in the GitLab id_tokens block, and it’s the default aws-actions/configure-aws-credentials requests. Leave all four alone and they agree by default. The moment you override one, override all of them, or you’ll meet that error.

Where this leaves you

There’s now no AWS access key anywhere in either forge. The pipeline proves it’s a release tag on your project, AWS hands it the signer role for the length of the run, and the credentials are gone the moment the runner stops. The signing module’s trust policy is the gate; the bootstrap module’s OIDC provider is the lock it hangs on. Both are verifiable in the public modules: terraform-aws-signing-kms for the role and terraform-aws-bootstrap for the provider.

The role can sign, but you still can’t verify anything yet, because nobody has the public half of that KMS key. Part 4 fixes that: gtb keys mint pulls the public key straight out of KMS, and gtb keys wkd publishes it somewhere the release platform can’t touch.

From allow_failure to blocking

Sat, 30 May 2026 00:00:00 +0000

There’s a special kind of CI job that everyone on a team quietly learns to ignore: the one marked allow_failure: true. It runs, it goes red, the pipeline goes green anyway, and after the third time you stop looking at it. I inherited six of those when I moved rust-tool-base’s CI to GitLab. Over a few days I turned three of them into real gates, and the interesting part was never the YAML. It was working out which ones had earned the right to block, and which hadn’t.

What allow_failure actually buys you

allow_failure: true is genuinely useful, and quietly corrosive. It lets a job report a problem without stopping the pipeline, which is exactly right for a check that’s noisy, not yet stable, or guarding against something you can’t fix this minute. The trouble is that a warning nobody is forced to act on is a warning nobody acts on. Leave a job advisory long enough and it becomes scenery: red, ignored, pointless. So an advisory check is really a promise, “I’ll make this blocking once it’s trustworthy”, and a promise you only ever mean to keep is just a lie you haven’t noticed yet.

When I migrated rust-tool-base from GitHub Actions to GitLab CI, the move landed six jobs as allow_failure: true: the macOS and Windows tests, the integration tests, cargo-audit, trivy, and coverage. That wasn’t laziness. A migration is the wrong moment to also be fighting flaky gates. But it left me holding six promises to either keep or admit I wasn’t going to.

A check earns the right to block

Here’s the rule I settled on. A check earns the right to fail your build when two things are true: it’s meaningful (a red result is a real problem, not noise) and it’s reliable (it goes red only when there genuinely is a problem, and it can actually run to completion). Flip a check to blocking before both hold and you haven’t raised the bar, you’ve taught the team to force-merge past red, which is worse than no gate at all, because now the red means nothing.

Three of my six crossed that line within a few days. Three deliberately didn’t. The reasons are the whole story.

trivy: blocked once there was nothing to block on

trivy scans the dependency tree for HIGH and CRITICAL advisories. It went across as advisory for an honest reason: the Cargo.lock at migration time already carried two known HIGH/CRITICAL advisories I hadn’t cleared yet, a path-traversal in gix-validate and a DNS-rebinding issue in rmcp. Make trivy blocking with those sitting there and the pipeline is red from day one, over problems I already knew about and was already fixing. So it stayed advisory until the dependency bumps cleared both, and then the allow_failure line came out. The gate never changed. The tree underneath it got clean enough to stand on.

integration-tests: blocked once it could actually run

The integration tests stand up a real Gitea in a Docker-in-Docker service and talk to it. They were advisory for a different reason: they couldn’t reliably run. dind needs a privileged runner, and the suite was resolving the container host with a hardcoded 127.0.0.1 that didn’t hold everywhere. Blocking a job that fails for infrastructure reasons rather than code reasons is the fastest way to make people distrust the entire pipeline. So the fix wasn’t in the YAML, it was making the thing dependable: privileged set on the runner, and the host resolved through the test library’s own get_host() instead of a hardcoded address. Once it ran the same way every time, it earned the gate.

coverage: blocked once it could run at all, then once it cleared the bar

Coverage is the two-step one, and my favourite, because it nearly didn’t make it for a thoroughly undramatic reason: it ran out of memory. cargo llvm-cov instruments every test binary, and linking hundreds of instrumented object files needs more RAM than the shared medium runner had, so the job bus-errored on the link. I tagged it onto a larger runner, and then the shared SaaS runners were switched off entirely, so the tag matched nothing and the job sat pending forever.

The fix was a self-hosted homelab runner with the RAM the instrumented link actually needs. I moved coverage there but kept it advisory for one run, to confirm the box could finish the build before I trusted it. It did, at 73.22% line coverage, so I set the gate to fail under 70% and made it blocking. Three points of headroom: enough that ordinary churn won’t trip it, tight enough that a real drop will. A coverage gate pinned to the current number is a tripwire that fires on the very next commit; set it a touch below and it catches regressions instead of normal life.

The three I left advisory, on purpose

The point was never “block everything”. Three jobs are still allow_failure in the current pipeline, deliberately. The macOS and Windows tests run on SaaS runners that bill by the minute; they’re worth running, not worth blocking every merge of a Linux-first project over a quota I’m choosing to ration. And cargo-audit stays advisory because cargo-deny already does the blocking advisory check: cargo-audit is a second opinion from a different database, and a second opinion that can veto isn’t a second opinion, it’s a duplicate gate that will eventually disagree with the first and block you on the difference.

That’s the same rule from the other side. Those three haven’t earned the right to block, because blocking them would cost more than it ever caught.

The upshot

allow_failure: true is fine as a waiting room and corrosive as a destination. Every advisory check is a promise to make it blocking once it’s both meaningful and reliable, and the job is to keep the promise or admit you won’t. trivy earned its gate when the advisories cleared, the integration tests when they ran the same way every time, coverage when it had a runner with enough memory and a threshold set just below the current mark. The three I left advisory earned that standing too, by costing more to block than they’d catch. The YAML is one deleted line per job. Knowing which line to delete, and when, is the whole skill.

Moving this blog off Jekyll

Thu, 21 May 2026 00:00:00 +0000

The blog you’re reading used to be a Jekyll site on GitHub Pages, built on the lovely beautiful-jekyll theme. It isn’t any more: it’s Hugo now, published to GitLab Pages. The hosting move rode along with go-tool-base leaving GitHub for GitLab, but dropping Jekyll for Hugo was its own decision, and the more interesting one. Most of the migration was painless. Two bits were not, and they’re the two bits worth writing down.

Why leave Jekyll

Jekyll hadn’t done anything wrong, exactly. beautiful-jekyll is a genuinely nice theme and the site worked fine for years. But it had started to show its age. It hadn’t seen much improvement in a long while, and keeping it building meant staying on older versions of Ruby. Ruby is a perfectly good language, just never one I’ve much enjoyed living in, and I could feel one of those fork-it-and-drag-it-up-to-date afternoons coming, the kind I’d done before and didn’t fancy repeating.

So rather than patch up what I had, I asked the more interesting question: what else is out there? It came down to a shortlist of two, Astro and Hugo. Hugo won, fairly narrowly. Partly I just liked more of its out-of-the-box themes. And partly because it’s written in Go: one portable binary, no toolchain to wrangle, the sort of thing an engineer can drop onto any machine and run without a second thought.

The day every image on the blog tripled

The first proper snag was about where images live. Hugo would happily have let me keep Jekyll’s arrangement, one big /assets/images/ folder with every post linking into it by absolute path. But I’d picked the Stack theme, and Stack leans towards page bundles: each post is a directory, and the post’s own images sit right next to its index.md, referenced by plain relative name. The cover image becomes a resource of the post rather than a file in a shared bucket.

That’s a better model, and I decided to commit to it. Getting there, I managed to make a proper mess. The migration copied the old assets/images across, and Hugo’s static/ directory wanted a copy too, and then I started moving covers into the bundles, and at one humbling point a count turned up every image existing three times: once in assets/images, once under static/, and once in a bundle. A blog with a hundred-odd images had become a blog with three hundred-odd, most of them duplicates nobody referenced.

The fix was to go all the way to the bundle model: move each post’s images into its own directory, rewrite the references from absolute /assets/images/x.png paths to bare x.png, and delete the two shared piles entirely. Once the images lived with the posts, there was exactly one copy of each and the path was obvious. But for an afternoon the repository was a hall of mirrors, and the lesson was to pick the new tool’s model and go all the way to it, rather than carrying the old one alongside it and ending up with both.

The “extended” image that wasn’t new enough

The second one cost me the most time, and it’s the most transferable, so it gets the most words.

Hugo comes in two flavours, ordinary and extended, and the Stack theme needs extended because it compiles SCSS. So I reached for an off-the-shelf extended Hugo container image, wired it into the pipeline, and watched the build fail with an error about a template function the theme was calling that simply didn’t seem to exist.

I spent far too long suspecting the theme, my config, my content. The actual culprit was a version. The image I’d grabbed was a couple of minor releases behind, and Stack v4 uses .Site.Language.Locale, a Hugo feature that only landed in 0.157. The image was older than that, so the function genuinely wasn’t there, and the error was telling me the literal truth in a way I wasn’t ready to hear. “Extended” had told me the flavour was right and lulled me into not checking the version, which was the thing that actually mattered.

The fix was to pin a specific, recent extended image rather than trusting a floating “extended” tag to be new enough. The pipeline now runs on a pinned hugomods/hugo:debian-git-0.161.1, comfortably past the 0.157 the theme needs, and the build that had been failing on a missing function went green the moment the version was right. A theme has a minimum Hugo version the same way any dependency has a minimum, and “extended” is a feature flag, not a version number.

What it comes down to

Moving this blog from Jekyll to Hugo, and from GitHub Pages to GitLab Pages on the way, was mostly a pleasant afternoon, with two frustrations worth sharing. Commit fully to your theme’s page-bundle model rather than dragging Jekyll’s shared-assets layout along beside it, or you’ll briefly own three copies of every image. And pin your Hugo version explicitly, because a theme needs a recent enough Hugo, and the “extended” label tells you nothing at all about whether yours is.

If the site renders for you now, both got sorted. If it doesn’t, well, you’re reading this in a text editor, and I’ve some more debugging to do.

Two bugs that taught me the rules

Wed, 20 May 2026 00:00:00 +0000

Some bugs are interesting because they’re subtle. These two were interesting because they were the exact opposite… in each case the tool had a hard rule I simply didn’t know about, and its error message couldn’t be bothered to tell me what that rule was. Both came out of building the infrastructure toolchain, both cost me a good deal more time than they had any right to, and both are the sort of thing that looks blindingly obvious the moment you know it and utterly baffling until you do.

So here they are, written down, partly to save you the bother and partly so I don’t go and forget them myself.

Bug one: the rule-less job that skips your merge requests

The cicd gate components, in their first cut, shipped with no rules: block. They were dead simple jobs: lint, scan, validate. No conditions, because they should just always run. Obviously.

They ran on branch pipelines. On merge requests, they didn’t run at all! The gates that were the entire point of the components were simply absent from the one place you’d most want to see them… the merge request.

The cause is a GitLab CI rule that’s remarkably easy to go years without ever learning: a job with no rules: block runs only on branch and tag pipelines. It does not run on merge-request pipelines. So “no conditions” doesn’t mean “runs everywhere” at all. It means “runs everywhere except a merge request”, which is about the least intuitive default I can think of.

The fix is faintly absurd, and that’s exactly what makes it stick. You add an unconditional rule: rules: [{ when: on_success }]. The content of that rule does precisely nothing. It always matches. What actually matters is that the job now has a rules: block at all, because merely having one is what makes a job eligible for merge-request pipelines. A rule whose content is meaningless, added solely so the block exists. That’s the fix. I’ll admit I stared at it for a moment.

Bug two: the import block that only works at the root

The second one came from terraform-aws-security-baseline. The account-hardening module needed to adopt a resource that already existed in the account, which is exactly what OpenTofu’s import {} block is for. So an import block went into the account-hardening module, right next to the resource it was adopting. The natural home for it, surely.

OpenTofu disagreed, and rejected it outright. The rule: an import block is only allowed in the root module. It can’t live inside a child module. A module that wants one of its own resources imported can’t declare that import itself… the import has to be declared up at the root, and the root caller does the adopting.

The fix was to take the import block out of the module and document caller-side adoption instead. The module describes the resource, and the root configuration that calls the module is where the import actually lives.

Two unrelated bugs, in two completely different tools, and the same shape sitting underneath both of them.

In each case the tool has a hard structural rule. Where a block is allowed to live. What makes a job eligible for a particular kind of pipeline. And in each case the error told me the tool was unhappy without telling me which rule I’d broken, so the obvious next move (debugging my own logic) was the wrong move entirely. There was nothing wrong with the logic. The thing was simply in a place the tool doesn’t allow, or missing a block the tool quietly insists on.

The lasting lesson here isn’t the two specific rules, useful as they are to know. It’s the reflex. When something that should obviously work just doesn’t, and the error is unhelpful, stop debugging your logic and start suspecting a structural rule about where something is allowed to be, or whether a thing is eligible in the first place. GitLab CI and OpenTofu both have a handful of these, and you mostly learn them the hard way, by tripping over them. Knowing the shape of the category at least means the next one costs you an hour instead of a whole afternoon.

Worth remembering

Two bugs from building the toolchain, one shape. A GitLab CI job with no rules: block runs on branches and tags but silently not on merge requests, and the fix is an unconditional rules: block whose content does nothing and whose mere existence is the entire point. An OpenTofu import block gets rejected inside a child module, because imports are only legal at the root, so the caller adopts and the module just describes.

Neither error named the rule it was enforcing, and that’s the category to watch for. When sound logic fails against an unhelpful error, suspect a structural rule about where a thing may live or whether it’s even eligible… not a bug in what you actually wrote. It’ll save you an afternoon. It certainly cost me a couple.

Reviewed, then applied

Mon, 18 May 2026 00:00:00 +0000

The genuinely dangerous moment in infrastructure-as-code isn’t the apply. It’s the gap between the plan a human read and approved, and the change that actually runs a moment later. If those two are different computations (and by default they are) then nobody really reviewed the thing that touched your account. The infra repo closes that gap from both ends.

The gap between “reviewed” and “ran”

Here’s the moment in infrastructure-as-code where things go wrong.

Someone opens a merge request. CI runs tofu plan and the output is there to review: these three resources change, this one is destroyed. A human reads it, decides it’s correct, approves, merges. Then apply runs.

The trap is in what apply actually applies. If apply does its own fresh tofu plan and then applies that, the change that runs is not necessarily the change that was reviewed. State can have moved. A provider can have drifted. Someone else can have applied something in between. The reviewed plan and the applied change are two separate computations done at two different moments, and every difference between those moments is a change nobody looked at.

infra closes that gap from both ends.

Plan as an artifact

The first end is making the reviewed plan and the applied plan the same object.

The tofu-plan component runs the plan and saves it. It writes tfplan.cache, OpenTofu’s binary plan file, as a CI artifact. It also writes tfplan.json, which GitLab renders as a plan widget right in the merge request: the add, change and destroy summary, there to review without leaving the MR.

The tofu-apply component then does not re-plan. It applies that saved tfplan.cache. And OpenTofu itself enforces the safety net: applying a stale plan file, one captured against a state that has since moved, is rejected by the tool. So what reaches the account is provably the plan that was reviewed, or it’s nothing at all. There’s no third option where something unreviewed slips through.

Applying is a human decision

The second end is when apply runs.

infra is trunk-based: it dropped the develop branch and works on main. But a naive trunk setup auto-applies every push to main, which means there’s no human gate at all, just whatever the last merge happened to contain.

So the gate is built explicitly. releaser-pleaser keeps a release merge request open against main. Ordinary merges to main run plans but apply nothing. The apply happens only when a person merges the release MR. Merging it cuts a release tag, and the tag pipeline is what runs tofu-apply, against the plan banked by the latest main pipeline.

The effect is that the act of applying to the account is the deliberate, visible act of merging the release request. Nothing reaches the account because a commit landed. It reaches the account because a person decided a release should go out and merged it. (Which, after the accidental v2.0.0 that kicked off the whole GitLab move, is a discipline I’d freshly relearned the value of.)

The guard on the gate

There’s one more piece, because a gate is only as good as its precondition.

A verify-main-plan job blocks the release MR from being mergeable unless the latest main pipeline is green. You can’t cut a release, and therefore can’t apply, on top of a main whose plan didn’t even succeed. The human gate has its own gate: the thing you’re about to merge has to be standing on a known-good plan before you’re allowed to merge it.

The bottom line

The risk in infrastructure-as-code is the gap between the plan a human reviewed and the change that runs, because a re-plan at apply time is a different computation from the one that was approved.

infra closes it twice over. tofu-plan saves the plan as a tfplan.cache artifact and renders it as a merge-request widget; tofu-apply applies that exact artifact, and OpenTofu rejects it outright if the state has moved underneath it. And applying is gated on a human merging a releaser-pleaser release request, not on a push, with a verify-main-plan check making sure that request can only be merged on top of a green plan. What gets applied is what was reviewed, when a person decided it should be.

CI you include, not copy

Sat, 16 May 2026 00:00:00 +0000

Every infrastructure repo runs the same CI: lint the OpenTofu, scan it, validate it, plan, apply. The first repo, you write that .gitlab-ci.yml by hand. The second, you copy it. By the third, you’ve got three copies of the same pipeline quietly drifting apart, which is the exact problem you’d never tolerate in application code. The cicd repo is the fix, and it’s just the library-first instinct pointed at the pipeline.

The `.gitlab-ci.yml` you keep copying

The infrastructure repos in this series all run the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security issues and secrets, and on the deploy side, plan and apply.

The first repo, you write that .gitlab-ci.yml by hand. The second repo needs the same jobs, so you copy it. The third repo, you copy it again. Now there are three copies of the same pipeline, and they do what copies always do. They drift. A fix you make in one repo’s CI doesn’t reach the other two. A tightened scan rule lands in the repo you were working in and nowhere else. It’s the copy-paste problem, exactly as it shows up in application code, just written in YAML and therefore that bit easier to pretend isn’t code.

GitLab has a feature for exactly this

GitLab CI/CD Components are the answer to that problem. A component is a reusable, versioned piece of pipeline that you publish, and other projects pull in with an include: pinned to a version:

include:
 - component: gitlab.com/phpboyscout/cicd/tofu-lint@v0.5.0

That’s a library import, for pipeline. The component has a defined interface, a version, and a home in GitLab’s CI/CD Catalog. A consuming repo includes it instead of carrying its own copy, and when the component improves, the consumer moves a version pin rather than re-copying YAML.

Why a monorepo of components

The cicd repo holds all of the components together: tofu-lint, tofu-security, tofu-validate, tofu-plan, tofu-apply, and more. One project, not one project per component.

That’s a deliberate call, and the reason is how GitLab versions things. A version is a tag, and a tag belongs to a project. A component’s version is its project’s tag. So a monorepo of components, versioned together as one tag stream, is the natural unit: a consumer pins @v0.5.0 and gets a known-good set of components that were tested together, rather than juggling a separate version for each one.

Authoring discipline

A component is a file under templates/, and it opens with a spec: inputs: block: the typed inputs, their defaults, the component’s public interface.

The discipline that keeps the library usable is that a component must be consumer-agnostic. It never hardcodes a token, and it never names a particular consumer’s variable. Inputs have sensible defaults, and a consuming repo overrides them. A component that reaches out and assumes something about the repo including it is a component that works in one repo and surprises the next. An authoring guide in the repo keeps that consistent across everyone who adds a component.

The self-test you cannot fully write

The cicd repo tests its own components with a self-test pipeline. It’s worth knowing where that self-test stops.

When a repo tests its own components by running them in child pipelines, GitLab masks $CI_PIPELINE_SOURCE as parent_pipeline. A component’s rules:, which often branch on the pipeline source to behave differently for a merge request than for a branch or a tag, therefore can’t be exercised honestly by the self-test: the source they’d branch on has been flattened. The self-test covers what it can, and the component rules: are, in the end, validated by real consumers using them for real. That’s a genuine limit, and naming it is better than pretending the self-test proves more than it does. (It’s also, not coincidentally, the exact rules: quirk that bit me in one of the two bugs I closed the series with.)

The same instinct, again

This blog keeps circling the same instinct. go-tool-base exists because the same CLI scaffolding kept getting rewritten, so it was extracted into a library. cicd is that instinct pointed at the pipeline: the same gate jobs kept getting copied between repos, so they were extracted into a versioned, included library.

Stop copy-pasting. Publish, version, include. It’s true for CLI code, and it turns out to be just as true for the YAML that builds and ships it.

The gist

Every infrastructure repo needs the same CI, and copying the .gitlab-ci.yml between them produces copies that drift apart. GitLab CI/CD Components fix it: reusable, versioned pipeline that a repo include:s and pins, instead of carrying its own copy.

cicd is a monorepo of those components, versioned together as one tag stream, because GitLab tags a project and a component’s version is its project’s tag. Components are authored consumer-agnostic, with typed spec: inputs: and no hardcoded assumptions, and their rules: are validated by real use because the self-test can’t see the pipeline source. It’s the library-first instinct, applied to CI: publish it once, include it everywhere, fix it in one place.

A 403 you can't fix in IAM

Thu, 14 May 2026 00:00:00 +0000

The OIDC post explained the handshake that lets a GitLab pipeline deploy to AWS with no stored key. This is the story of the first time I got it wrong, and spent an afternoon fixing the wrong thing. The error was a flat 403 from AWS, and the maddening part is that no amount of editing the IAM policy was ever going to fix it.

A 403 on the first real run

The OIDC post covered the handshake: GitLab CI mints a signed token, AWS exchanges it for short-lived credentials against a role whose trust policy names the pipeline. During the GitLab migration I wired exactly that up for the infra repo, including a trust policy condition meant to let merge-request pipelines run a plan.

The first merge request that should have triggered tofu-plan didn’t run it. The job failed, and the error from AWS was a flat AccessDenied. A 403.

The instinct, and why it wastes an afternoon

The instinct on an IAM 403 is immediate and almost always right: the policy’s wrong, so go and edit the policy. Tighten the condition. Loosen the condition. Check the wildcard. Re-read the sub pattern character by character.

All of that was wasted, and it was wasted for a reason that took me far too long to see. The trust policy wasn’t matching the wrong value. It was matching a value that does not exist. No amount of editing a condition makes it match a thing that’s never present.

What is actually in the token

GitLab’s OIDC token has a sub claim that encodes the pipeline’s context, and part of that encoding is a ref_type. I’d assumed ref_type could be branch, tag, or mr, because a pipeline can certainly be a branch pipeline, a tag pipeline, or a merge-request pipeline. So the trust policy, for the plan job, matched a sub containing ref_type:mr.

That assumption was wrong. GitLab’s ref_type is branch or tag. That’s the entire set. There is no mr.

A merge-request pipeline doesn’t run against a merge-request ref. It runs against the source branch. So its token’s sub carries ref_type:branch, like any other branch pipeline. The trust policy condition asked for ref_type:mr, GitLab never puts mr in a token, the condition was therefore never true, and every merge-request pipeline got a 403. Forever, until the policy stopped asking for a claim that isn’t real.

The fix, and the lesson worth more than the fix

The fix is small once it’s visible: match ref_type:branch and narrow it down by branch name or project path instead. An afternoon of policy edits, and the actual change is one word.

The lesson is the part worth keeping. When an OIDC trust fails, the useful question is never “is my policy clever enough”. It’s “what’s actually in the token”. An OIDC trust policy can only ever match the claims the identity provider genuinely asserts, and the gap between what a provider asserts and what you assumed it asserts is precisely where this class of bug lives.

So the move, when an OIDC handshake 403s, is to get hold of a real token and decode it. Look at the actual sub, the actual claims, the actual values. Match what’s there. A 403 that survives every sensible edit to the policy is usually not a policy that’s too loose or too strict. It’s a policy matching a claim that was never going to be in the token.

The habit it left behind

I wired an OIDC trust policy to let merge-request pipelines plan, by matching a sub claim with ref_type:mr. The first real merge request got a 403, and no edit to the policy fixed it, because GitLab’s ref_type is only ever branch or tag. A merge-request pipeline runs on a branch ref, so the mr value the policy demanded was never in any token.

The fix was one word. The habit it left behind is the valuable bit: when an OIDC trust fails, stop editing the policy and go and read a real token. A trust policy can only match what the provider actually asserts, and “what I assumed it asserts” is where the 403 was hiding the whole time. (If this shape of bug feels familiar by the end of the series, that’s not an accident: I come back to it with two more from exactly the same family.)

Why go-tool-base left GitHub for GitLab

Mon, 11 May 2026 00:00:00 +0000

A botched version bump made me stop and actually look at where go-tool-base lived, and I didn’t much like what I saw. GitHub had spent months quietly falling over, and when Mitchell Hashimoto (GitHub user #1299, no less) publicly walked Ghostty off the platform, it stopped feeling like just my problem. I’ve been a GitLab fan for years, so the move was less a leap and more an overdue nudge. This is the why, not the how.

It started with a wrong number

Every migration has a trigger, and mine was embarrassingly small. A commit landed on main carrying a BREAKING CHANGE: footer it didn’t really deserve. Semantic-release did exactly what it’s told to do with that footer: it cut a major version. go-tool-base lurched from the v1 line straight to v2.0.0, and a chain of things that keyed off the version went sideways with it.

It was fixable. It wasn’t a disaster. But it was the kind of small, stupid breakage that makes you stop and actually look at your setup instead of just patching it and moving on. And when I looked, the version bump wasn’t the thing that bothered me. It was everything around it.

The platform had been quietly failing

I’d been losing time to GitHub for months. Not dramatically. No single outage you’d write home about, just a steady drip of Actions queues that wouldn’t drain, pull requests that wouldn’t merge, the occasional morning where the thing simply wasn’t there. You absorb it. You re-run the job. You make a coffee and try again. You tell yourself it’s a blip.

The trouble with a steady drip is that you stop counting it. It becomes weather.

The canary left the mine

Then, in late April, Mitchell Hashimoto (co-founder of HashiCorp, creator of Vagrant, Terraform and the Ghostty terminal) published Ghostty Is Leaving GitHub, and The Register picked it up a day later under the headline “GitHub ’no longer a place for serious work’”.

This is not a man with a casual relationship to GitHub. He’s, by his own account, user #1299, joined February 2008. He called it “the place that has made me the most happy”. And he still wrote this:

This is no longer a place for serious work if it just blocks you out for hours per day, every day.

The detail that landed hardest for me wasn’t a quote, it was a habit. He’d kept a journal for a month, marking an “X” on every day a GitHub outage had cost him working time. Almost every day had an X. Reading that, I realised I’d been having the same month. I’d just never been disciplined enough to write it down. He’d turned my vague “it’s been flaky lately” into a row of crosses on a calendar.

I want to ship software and it doesn’t want me to ship software.

When the person who’s been on the platform for eighteen years and loves it says that out loud, it stops being your private grumble. It’s the canary, and the canary has stopped singing.

Why GitLab, and not just “somewhere else”

Being annoyed at GitHub is a reason to leave. It is not, on its own, a reason to pick a destination. The destination has to be a positive choice.

For me GitLab was an easy one, because I’ve been a fan for years. Long enough, in fact, to have also been a reliable grumbler about their pricing tiers, which is how you know it’s a real relationship and not a honeymoon. What I’ve always rated is the model: GitLab treats source hosting, CI/CD, the package registry, releases and Pages as one integrated product, not a marketplace of bolted-on parts you assemble yourself.

That integration is the actual prize. On the old setup, “CI” meant a folder of separate GitHub Actions workflow files, each pinned, each its own little world. On GitLab it’s a single .gitlab-ci.yml pipeline with proper stages (lint, test, security, docs, release) and the release stage talks to the built-in package registry and Pages without me wiring up a single external credential. The CI job that builds the project can authenticate to the things the project needs because they’re the same platform.

There’s a second-order benefit too. A migration is a rare licence to fix things you’d never otherwise touch. Moving gave me the cover to reset go-tool-base’s versioning cleanly (back to a sensible v0.x line, the accidental v2.0.0 left behind as a cautionary tale) and to move the module path to its new home in one deliberate change rather than a thousand apologetic ones.

What I’m not going to claim

I’m not going to tell you GitHub is finished, or that GitLab never has a bad day, because it does, everyone does. This isn’t a teardown. GitHub gave go-tool-base a perfectly good home for its first year, and the archived mirror is still sitting there, read-only, pointing anyone who finds it at the new place.

What changed is simpler than a grand verdict. The friction crossed a line, someone I respect said the quiet part loudly enough that I couldn’t keep filing it under “weather”, and the place I’d have moved to anyway was sitting right there with a better model. Sometimes the prudent move and the move you secretly wanted turn out to be the same move, and you just need a wrong version number to give you permission.

Boiling it down

go-tool-base moved from GitHub to GitLab in May 2026. The proximate cause was a self-inflicted version-bump mess; the real cause was months of GitHub unreliability that I’d stopped consciously noticing until Mitchell Hashimoto’s very public departure named it for me. GitLab was a positive pick, not just an escape hatch: its integrated CI/CD, registry, releases and Pages are one product rather than a kit, and that integration is genuinely worth having. The migration also bought a clean versioning restart as a bonus.

If you’ve been absorbing a steady drip of friction and telling yourself it’s normal: try the calendar trick. Mark the X’s for a month. The page will tell you something you already half-know.