CI/CD, and getting things released

Pipelines you include rather than copy, a toolchain baked into one image, and a release that is just another merge request and tags the commit that actually landed. Plus the jobs that silently never ran, the cache that filled a disk, and a runner fleet that scales to zero.

23 posts

Start here

The two ideas the rest of this rests on: stop copying pipelines, and stop reinstalling the same tools.

  1. CI you include, not copyCopied .gitlab-ci.yml files drift out of sync. How GitLab CI/CD Components let a repo include and version-pin a shared pipeline instead. Pioneering
  2. Stop installing the same tools on every pipeline jobNearly every CI job began by fetching and compiling the same tools. Baking them into one image instead, and what that saved per pipeline. Pioneering
  3. One image for the whole toolchainBaking OpenTofu, tflint and trivy into a single version-pinned CI image, and publishing the scanned tarball with crane to avoid a kaniko rebuild. Pioneering

Releases

Making a release a reviewable change rather than a side effect of merging, and the orchestrator that tags the commit which actually landed.

  1. Nine tags that were never on the branchcolophon works out the version your commits have earned, opens the release as a merge request, and tags what actually landed. Not what your forge says did. First Light
  2. Exit 0 is the strongest claim a program can makeThe job told me, very politely, that it hadn't done its job, and then exited 0. Two questions were enough to undo the spec written to fix it. Pioneering
  3. Four lines cost a whole websiteTo append four lines to one YAML file, the release job was cloning this entire website into memory. The first fix for it failed without making a sound. Pioneering
  4. A release is just another merge requestMoving off tag-on-merge releases, where a release is a side effect of merging, to a model where the release is itself a reviewable change. Pioneering
  5. Same config, two answersPublishing a seventeen-crate Rust workspace to crates.io, and why the same release configuration produced two different answers. Pioneering
  6. Three traps release-plz sets for a Rust workspaceThree traps release-plz sets for a Rust workspace, starting with a default tag template that collides the moment you have more than one crate. Pioneering
  7. The version pin you can't lowerA standard-library security fix forced a Go version bump, and then the release failed because the build image shipped an older toolchain. Pioneering

Gates worth having

When an advisory job earns the right to block a merge, and what happens when one never runs at all.

  1. From allow_failure to blockingTurning three advisory CI jobs into real gates once they were trustworthy enough to block a merge, and how to tell when they are. Pioneering
  2. The build gate these sites never hadThree Hugo sites each hand-rolled a near-identical deploy job that only ever ran on merge, so nothing ever checked the build before it landed. Pioneering
  3. Two bugs that taught me the rulesTwo CI and infrastructure bugs with one shape: a GitLab job with no rules block skips merge requests, and an import block runs only once. Pioneering
  4. The component that fired on every scheduleA CI component gated on the default branch fired on every Renovate schedule too, because a scheduled run is also on the default branch. Pioneering
  5. The fastest job is the one you never runHalf my CI jobs ran for no reason on every merge request. Skipping them with rules:changes, and why that is trickier than the manual suggests. Pioneering

Where it runs

The platform, the credentials the pipeline does not hold, and the runners: a homelab box that kept filling up and a fleet that costs nothing while idle.

  1. Why go-tool-base left GitHub for GitLabAn accidental major version bump was the last straw after months of unreliability. Why a Go framework moved off GitHub, and what moved with it. Pioneering
  2. No access keys in CIReplacing long-lived AWS access keys in CI with OIDC federation, so the pipeline mints a short-lived token instead of holding a secret. Pioneering
  3. The secret that wasn't on my branchA secret scanner failed a merge request over a test key and a documentation PEM that the change did not contain. Scoping a scan properly. Pioneering
  4. The runner fleet that scales to zeroThe homelab finally ran out. Not disk this time, capacity. So the overflow went to AWS, on the condition that it costs nothing when nobody's pushing. Pioneering
  5. One cache key per lockfile filled my runner's diskThe documented cargo cache key gives every lockfile its own cache. On a self-hosted runner that fills the disk. Whose disk was the advice for? Pioneering
  6. The repo that downloaded 170 gigabytesAn hour to compile and 170GB of artefacts to clean up afterwards. Rust has a built-in answer to that, and I only used half of it. Pioneering
  7. The release that OOM-killed its own compilerA wedding-photo culler ran out of memory compiling an LLM SDK it has never once called. The release had been running for thirty-eight minutes at that point. The Expedition
  8. The builder had been archived for a yearI went looking for reproducible CI images and found the builder had been archived fourteen months earlier. Swapping it out turned out to be the easy part. Pioneering

Where to next

Everything, newest first → Follow this guide by RSS