Go-Tool-Base on PHP Boy Scout

I filed a feature request into my own framework

Fri, 03 Jul 2026 00:00:00 +0000

I’m building a tool called keryx, and the part of it that matters here is its studio: a browser app where the work happens, which saves everything you do into a git repository behind the scenes, the way a developer’s project lives in git with a history you can step back through.

I wanted that repository to be able to live entirely in memory. Cloned, edited, committed and pushed without ever writing a working copy out to a disk, for the times when you can’t, or would rather not, leave a checkout sitting around on the machine. It sounds exotic, but it’s something git libraries genuinely support, and it’s exactly what a browser studio running on a server somewhere wants.

Getting it working needed one small, awkward piece of plumbing in the middle. And a few lines into writing that piece, I stopped, because I realised I was writing it in the wrong repository.

The bridge I was about to vendor

Here’s the awkward bit. All of keryx’s file handling goes through afero, the standard filesystem interface in the Go world, the thing you hand your code so it neither knows nor cares whether it’s talking to a real disk, a test fake, or memory. It’s the interface go-tool-base hands you for filesystem work. But an in-memory git repository, the kind go-git gives you with its memfs, doesn’t speak afero. It speaks go-billy’s filesystem interface instead. Two perfectly good filesystem abstractions, and a worktree on the wrong side of the gap from all my code.

What I needed was an adapter: a bridge that makes a billy filesystem look like an afero.Fs, so the studio’s existing file handlers work unchanged over a repo that lives entirely in RAM. Twenty minutes of work, maybe. The obvious move was to write it inside keryx and get on with my afternoon.

And that’s the move I caught myself making. Because a billy-to-afero bridge is not a keryx thing. It’s not even a studio thing. It’s a general capability that any tool built on go-tool-base might want the moment it touches git. Vendor it in keryx and I’ve buried a reusable bit of plumbing inside one consumer, where it will drift away from the framework and get reinvented, slightly differently, in the next tool I build that needs it.

The bridge belonged in the framework. So that’s where I put it.

A feature request, against myself

I wrote the need up properly. Not a code comment, not a mental note, but an actual feature request, with a reference implementation sketched out, dropped into the go-tool-base repository as a document for the framework to act on.

There’s something slightly absurd about filing a feature request against your own project. The author and the customer are the same person. But that’s exactly what gives it its value. The most useful design input a framework gets is a real consumer hitting a real wall, and for once I was both: the person who maintains go-tool-base, and the person downstream of it who’d just discovered something it couldn’t yet do. The request wasn’t hypothetical or “wouldn’t it be nice”. It was “I am stuck on this right now, here is precisely what it can’t do yet.”

What came out the other side is pkg/vcs/repo/aferobilly, a first-class part of the framework as of v0.22.0. Its own description is the clearest summary of what it is:

// Package aferobilly adapts a go-billy/v5 Filesystem to an afero.Fs. It is the
// pure, reusable bridge behind pkg/vcs/repo's worktree-as-afero accessors, but
// works for any billy filesystem (memfs, osfs, chroot).

Alongside it, the worktree itself grew the accessors that hand you that view: WorkFS() for a live afero handle, and WithWorkFS() for an atomic sequence (worktree_fs.go, and the adapter itself). keryx then consumed it like any other framework feature, and the in-memory studio fell into place.

Two sessions, one dependency

The bit I’d actually recommend to anyone is what I did with my time while that got built.

I didn’t down tools and wait for the adapter. I handed the feature request to a separate agent session and let it build the framework feature from the spec, working in the go-tool-base repo, while my keryx session carried straight on with all the studio work that didn’t depend on the bridge. Two sessions running in parallel, deliberately sequenced around the one dependency between them: keryx needs the adapter, so the adapter session goes first, but only the last mile of keryx actually waits on it. When go-tool-base cut the release with the adapter in it, keryx pulled the new version and the final piece slotted in.

That’s a workflow the framework split makes possible. The thing that’s a shared capability gets built once, in its proper home, by one stream of work, while the thing that consumes it carries on in another. The dependency between them is real, so the order matters, but only at the very end.

The one rule that came with it

Upstreaming it also meant the tricky part got solved properly, once, with a warning attached, rather than learned the hard way in a consumer. The adapter is concurrency-safe by construction: it serialises every operation through a lock, so when that lock is the same mutex guarding the repo, a live afero handle over the worktree is genuinely safe to share. But that safety has a sharp edge, and the package says so plainly:

// A handle (and its open files) must NOT be used from inside a critical section
// that already holds the same locker (the repo mutex is non-reentrant — that
// would deadlock).

Use the handle inside a WithWorkFS callback and you’ll re-lock a non-reentrant mutex and hang yourself. That’s exactly the kind of footgun that, vendored in keryx, I’d have discovered at 11pm with a wedged process and no idea why. In the framework, it’s documented at the source, where the next consumer reads it before they trip over it.

The truest test of a framework

Building a real product on your own framework is the best test of it, and this is what that actually looks like in practice. The test is sharper than “does it work”. It’s “what does the product need that the framework doesn’t have yet”, and every real answer to that is a feature request waiting to be filed.

The discipline is filing it against the framework instead of patching around it in the app. Do that, and the awkward bridge has exactly one home, the deadlock warning gets written down once, and the next tool I build inherits all of it for free. The customer was me. The feature request was real. And go-tool-base is better for my having been stuck.

A stack trace is not an error message

Tue, 30 Jun 2026 00:00:00 +0000

The repair agent I’ve been building into go-tool-base narrates what it’s doing as it goes. It builds, it tests, it lints, it fixes, and it logs each step so I can watch it think. Mostly that log is a calm, readable trickle: tried this, that failed, reading the file, here’s the fix. Mostly.

The moment a build or a lint step failed, the calm trickle turned into a wall of Go stack frames, the same forty lines of runtime gubbins over and over, burying the one line I actually wanted to read.

Two different things we both call “the error”

The agent’s tools wrap their failures with cockroachdb/errors, which is a lovely library: it attaches a stack trace to an error the moment you create it, so when something goes wrong deep in the weeds you can see exactly how you got there. A failed go build comes back as one of these rich, wrapped values, carrying its message and its stack.

The line that recorded the failure looked like this:

l.Warn("Tool execution failed", "tool", name, "error", err)

Looks fine. It is not fine. The logger is charmbracelet/log, and when you hand a structured logger a cockroachdb error value as a field, it renders the whole thing: message, wraps, types, and every frame of that attached stack. So every failed step, and during self-repair there are plenty, printed a full traceback at WARN. The signal, the actual build error, was in there somewhere, wearing a forty-line coat.

The thing is, a stack trace and an error message are two different objects that we lazily both call “the error”. err.Error() is the message: short, human, “lint issues found…”. The value err is the message plus the evidence of where it came from. They serve different readers. The message is for whoever’s watching the loop run. The stack is for whoever’s debugging why the loop itself is broken. Hand the wrong one to the wrong reader and you’ve got either noise or a mystery.

Pick a reader per level

The fix is to stop making one log line serve both:

l.Warn("Tool execution failed", "tool", name, "error", err.Error())
l.Debug("Tool execution failure detail", "tool", name, "error", err)

The message goes to WARN, where someone’s watching the agent work and just wants to know what failed. The full wrapped value, stack and all, goes to DEBUG, where someone’s gone looking for trouble and wants every frame. Turn the level up and the evidence is right there; leave it at the default and the loop reads like prose again.

It’s a one-line change, give or take, and it lives in the shared chat tool-dispatch path rather than in the agent, so every tool-using client gets the quieter log for free. It’s the same loop I’d just taught to respect the linter; apparently I was determined to make it both honest and readable in the same fortnight.

Where the stack belongs

The stack trace was never the thing I needed to hide. cockroachdb/errors attaching it is exactly what I want; it’s the whole reason I use the library. The mistake was where I let it surface. A trace dumped at WARN on every routine failure isn’t observability, it’s wallpaper, and wallpaper is what you stop seeing. Keep the loud version for the level where someone’s actually gone looking for it, and leave the everyday log alone. The stack was never the noise. Printing it on every line was.

The agent said SUCCESS. The linter disagreed.

Fri, 26 Jun 2026 00:00:00 +0000

There’s a repair agent inside go-tool-base now. When you run gtb generate command, it doesn’t just spit out a file and wish you luck. An agent takes the generated code, builds it, runs the tests, and fixes whatever it broke, looping until the thing actually works (or until it’s tried the same fix five times and admits defeat). The whole point is that the generator hands you code that’s ready, not code that’s nearly ready and quietly now your problem.

So it stung a bit when I realised the agent had been holding itself to a lower bar than I’d hold any junior to. And I was the one who’d set the bar.

What “done” meant to the agent

The agent is a loop with real tools: it can build, test, read files, write files, tidy the module, and run golangci-lint. It works through them, and when it’s happy it replies with the word “SUCCESS” and the loop stops. On the Go side, the check is exactly that blunt:

if strings.Contains(strings.ToUpper(resp), "SUCCESS") {
 return nil
}

That’s the whole gate (agent.go). There’s no clever verification on my end that the agent actually did its homework. It does the work, it tells me it’s done, and I believe it. Which is fine, as long as the agent and I agree on what “done” means.

We didn’t.

The instruction that made lint optional

The agent decides it’s finished by following a numbered list in its system prompt. Here’s the line that did the damage:

If there are lint issues, use ‘golangci_lint’.

Read that the way the agent would. “If there are lint issues”… well, how would it know? The only way to find out is to run golangci-lint. But the instruction makes running golangci-lint the thing you do once you already know there are issues. It’s a chicken with no egg. And the SUCCESS condition at the bottom of the list never mentioned lint at all:

When the project builds successfully and tests pass, reply with “SUCCESS”.

So the agent did the sensible thing, given its orders. It built the code, ran the tests, saw both go green, and declared victory. golangci-lint was sat right there in its toolbox, unused, because nothing ever told it the job wasn’t finished until lint was clean too. I’d handed it a linter and then written a prompt that let it walk straight past it.

The galling part is that the linter was never the missing piece. The golangci_lint tool had been registered the whole time, and it even runs with --fix, so it’ll quietly clear the trivial stuff and only surface what actually needs a decision. The capability was there. The instructions just never required it.

The fix was words, not code

Here’s the part I find genuinely interesting. I didn’t add a check. There is no new gate in the Go. The fix is four lines of English:

Run ‘go_build’, ‘go_test’ and ‘golangci_lint’ in the project directory… Run all three; a clean build and passing tests do not imply clean lint.

Reply with “SUCCESS” only once ‘go_build’, ‘go_test’ AND ‘golangci_lint’ all pass with no errors and no reported issues.

That’s it. Lint moves from a remediation step you reach for once you somehow already know there’s a problem, into the gate itself. “Done” now means three green lights, not two.

It nags at me a little, that one. The reliability of an agent that writes and fixes real code came down to whether one sentence of instructions was precise enough. When your success criteria are a paragraph of prose, vagueness in that paragraph is a bug, the same as a vague type or an off-by-one. The spec just happens to be written in English, and the thing reading it is a language model that will cheerfully take the cheap reading if you leave it lying around. That’s the same lesson the goblin who wouldn’t stay dead taught me from the other direction: with these tools, what you say is what you get, and what you don’t say is fair game.

Leave it better, not just building

The Boy Scout Rule is the whole reason this blog exists, and I’d quietly exempted the robot from it. “Leave the campsite cleaner than you found it” had become “leave it building”, which is not the same thing and never was. If I’m going to put an agent in the loop precisely so it tidies up after the generator, then “tidy” has to mean what it would mean for a person on my team. Build, test and lint. No walking past the bin because nobody told you to pick it up.

The cobra hook I was sure I'd enabled

Wed, 24 Jun 2026 00:00:00 +0000

It came out of an audit. I’d recently pointed a small army of review agents at the whole go-tool-base codebase, back before that became a political problem, and one of the findings was that a subcommand could quietly skip the framework’s own start-up code. My first reaction was the dangerous one: surely not… we switched that on ages ago. So I asked for a second pair of eyes on the exact line.

There was no line. I was certain I’d enabled it, but I had simply never done it.

What the start-up hook is for

go-tool-base is built on cobra, the library most Go command-line tools are built on. In cobra, a command can carry a PersistentPreRunE: a function that runs before the command itself, and that, the name strongly implies, persists down to the command’s children. Think of it as the “before you do anything, get the tool ready” step.

go-tool-base uses exactly one of them, on the root command, to do all the humdrum setup: load and merge configuration, set up logging, ask about telemetry the first time you run, wire up the telemetry collector, and check whether there’s a newer release to install. Everything the tool does afterwards leans on that having happened. By the time your actual command runs, props.Config is meant to be populated and the collector is meant to exist.

The reasonable assumption (the one I’d made, anyway) is that “persistent” means it cascades. Define it once at the root and every mytool foo bar three levels down gets it for free.

“Persistent” promises less than it says

Here is the catch, and it is a good one to file away if you ever build a command tree. cobra runs only the nearest PersistentPreRunE it finds, walking up from the command you actually invoked. If a subcommand defines its own, that one runs and the root’s does not. Not as well as. Instead of. There’s no warning and no error; the child’s hook simply wins, and the parent’s is passed over in silence.

So the moment any command below the root declared its own PersistentPreRunE, the entire start-up for that branch, the config, the logging, the telemetry, the update check, would just not happen. props.Config would be nil. The collector would be nil. The first you’d hear of it is a nil-pointer panic a long way from the cause, or, worse, no panic at all and a tool running happily unconfigured.

EnableTraverseRunHooks is cobra’s opt-in to the behaviour most people assume is already the default: run every PersistentPreRunE from the root down to the leaf, in order. I’d assumed it was the default. It is not, and I’d never turned it on.

A landmine nobody had stepped on

The saving grace was that nothing was actually broken yet. In go-tool-base’s own command tree, the root is the only command that defines a persistent pre-run, so “root to leaf” and “nearest only” happen to produce the identical result. The flag being off changed nothing I could observe.

The bug was latent. It was a trap laid for the first person to do something entirely reasonable: add a PersistentPreRunE to one of their own subcommands. go-tool-base is a framework other tools are built on, so that person was never going to be me. The instant a downstream author did the obvious thing, their config and telemetry would vanish for that branch of their tool and nothing would tell them why.

That is the kind of bug I least like shipping. It compiled. It passed the tests. It would have demoed perfectly. And it sat there waiting to hand a stranger a debugging session with no breadcrumbs, for the crime of using a standard cobra feature the obvious way.

One line, and a note for whoever’s next

The fix is the line I was so sure already existed (root.go):

// Run every parent PersistentPreRunE from root to leaf rather than only the
// closest one. Without this, a downstream subcommand that defines its own
// PersistentPreRunE silently shadows the root bootstrap (config load,
// telemetry, update check) for that subtree. With it set, the framework
// bootstrap always runs first and a child hook runs after it.
cobra.EnableTraverseRunHooks = true

With it set, cobra runs the root start-up first and then the child’s hook, in order, so a downstream command adds to the setup instead of replacing it.

I didn’t want to stop there, because the next author to add a child hook still deserves to understand the ordering. So the change also drops a one-time debug line if it spots any command in the tree carrying its own PersistentPreRunE (the same file), saying out loud what’s going on:

l.Debug("a downstream command defines its own PersistentPreRunE; " +
	"it runs AFTER the framework bootstrap (config load, telemetry, update check), not instead of it")

And, belt and braces, the collector now defaults to a no-op so the few paths that do legitimately return early, like init and help, still satisfy the “always non-nil” promise the rest of the code relies on. The whole thing shipped with a pair of regression tests that assert the bootstrap really does run when a child hook is present, and that it runs first. It’s all written up in a short spec and landed in one commit.

Trust, but grep

There are two things worth taking away. The cobra one is portable: if you rely on PersistentPreRunE cascading down a command tree, set EnableTraverseRunHooks, because “persistent” means less than it sounds and the nearest hook wins by default.

The other is the one I keep having to relearn. The settings I’m most certain about are the ones I never check, precisely because the certainty is what stops me looking. Somewhere along the line I’d promoted “I meant to” straight to “I did”, with nothing in between… and then defended it out loud before I’d even gone to look. A review agent is good at exactly that blind spot: it has no memory of intending to do something, only the code in front of it. The best thing the audit turned up wasn’t a clever bug. It was a flag that was never there. Leaving the campsite better than you found it has to include the traps nobody’s stepped on yet.

Sign your own binaries with go-tool-base, part 7: rotation and break-glass

Tue, 23 Jun 2026 00:00:00 +0000

Most signing guides stop the moment the first release goes out the door, which is a shame, because the question that keeps you up at night comes later: what do you do when the key has to change? Keys get rotated on a schedule, keys get compromised, and one day you’ll want to move off RSA onto something newer. This last part of the signing series covers the bit everyone skips, swapping a signing key out from under a fleet of installed tools without locking a single one of them out, and the break-glass key for the day the primary is gone.

By Part 6 you’ve a pipeline signing every tagged release through a KMS key, a public key published over WKD, and that same key baked into the binary as its trust anchor. It all works. The catch nobody mentions is that you’ve now got a key embedded in software sitting on other people’s machines, and that key is the one thing in the whole chain you can’t quietly change. So let’s plan for changing it properly.

Why there’s no auto-rotate button

If you’ve used KMS for encryption, you’ll know it can rotate keys for you on a yearly tick. Asymmetric SIGN_VERIFY keys don’t get that, and the terraform-aws-signing-kms module sets enable_key_rotation = false on purpose. That’s not an oversight to work around; it falls straight out of how the key is built. The private half of a signing key never leaves the HSM and has no export path at all, so there’s no mechanism by which AWS could hand your verifiers a rotated public half and keep the old one verifiable. KMS auto-rotation works for symmetric keys precisely because you never see the key material; the instant you need a public key your clients pin against, rotation stops being something a cloud provider can do behind your back.

So rotation here is a runbook you maintain, not a checkbox you tick. That sounds like the worse deal until you weigh it: you’ve got a key with no exfiltration path, in exchange for rotating it by hand on the rare occasions you must. For a release-signing key that’s exactly the right trade. You don’t want this key rotating itself; you want to be standing right there when it happens.

Rotate by minting a new key, never by changing the old one

Here’s the rule that makes everything else fall into place. You do not edit the existing key. The module’s key_spec and name are immutable by design, and that immutability is a feature: the v1 key is a fixed point that stays verifiable while you stand up its replacement next to it.

So you mint a second key. A second module instance, named acme-release-signing-v2, with its own alias alias/acme-release-signing-v2:

module "signing_kms_v2" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v2"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn

 ci_subject_filters = [
 "project_path:acme/acme-cli:ref_type:tag:ref:v*",
 ]

 key_administrator_arns = [/* ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

Apply that and the v1 key carries on signing, undisturbed. Now mint its public half exactly the way you minted v1 back in Part 4, only the alias changes:

gtb keys mint \
 --backend aws-kms \
 --kms-region eu-west-2 \
 --key-id alias/acme-release-signing-v2 \
 --name "Acme Releases" \
 --email release@acme.dev \
 --created "2026-06-08T00:00:00Z" \
 --output signing-key-v2.asc

You’ve now got two real signing keys, both live, neither one a threat to the other. The whole rotation is the careful business of moving traffic from one to the other while every installed client keeps verifying.

The dual-publish, dual-sign window

This is the part that does the work, and it leans on one fact about how the verifier treats your keys. The trust anchor isn’t a single key, it’s a set: v1, v2, and the rotation authority all sit in internal/trustkeys/keys/ and all get embedded together. A release verifies if any key in that set validates its signature. That’s what makes a handover window possible, because during it a client might hold v1, or v2, or both, and verify happily whichever it has.

So you run an overlap. Three moves, in order.

First, publish both public keys over WKD. Same email bucket, same command you already know from Part 4, just with v2 added to the file list:

gtb keys wkd \
 --domain acme.dev \
 --email release@acme.dev \
 --output ./wkd-staging \
 signing-key-v1.asc signing-key-v2.asc rotation-authority.asc

Both signing keys share release@acme.dev in their UID, so they land in one hu/ bucket together, and the rotation authority rides along as it always has. Deploy that staging tree the way you deployed it before. Now the WKD endpoint serves the new trust anchor, so a client that fetches keys on its next update picks v2 up automatically.

Second, embed v2 alongside v1 and ship a release. Drop signing-key-v2.asc into internal/trustkeys/keys/ next to v1, cut a release, and that build now ships knowing about both keys. Installed clients pick the new trust anchor up as they update through the window. This is the slow bit, and it should be: you want the new key spread far and wide before you stop signing with the old one.

Third, sign with v2, wait, then retire v1. Point the pipeline’s GTB_SIGNING_KEY_ID at alias/acme-release-signing-v2 so new releases are signed by the new key. A client that’s already updated verifies against the embedded v2; one that’s lagging still has v1 in its set and, because you’re still publishing v1 over WKD, can still fetch and trust it. Nobody’s locked out.

Leave that overlap running long enough that you’re confident the slow movers have updated, weeks, not hours, depending on how often your users actually run the thing. Only then do you retire v1: drop its module instance, pull signing-key-v1.asc from the embed directory and the WKD file set, and ship a release that knows only about v2. The rotation is done, and at no point did an installed tool see a signature it couldn’t check.

The break-glass key, for when there’s no handover

Everything above assumes the old key can hand over to the new one: it’s still there, still able to sign, and you’re rotating on your own terms. The nightmare is the other case. The KMS key is gone, access revoked, or you’ve reason to think it’s compromised and you daren’t sign anything with it again. There’s no handover, because the thing that would do the handing is exactly what you’ve lost.

That’s what the rotation-authority key is for, and it’s why Part 4 had you mint it the moment everything else was calm. It’s an offline ed25519 key, generated once on a trusted machine, and the go-tool-base how-to, generate the rotation-authority key, walks the storage in detail. The short version: the private half never lives on a networked box. You print a paper backup with paperkey, write it to an encrypted USB stick, and the two go in a safe; the local copy gets shredded. The how-to even has you type the paper backup back in once before you walk away, because discovering your printer ate a stripe of pixels is a problem you want now, not eighteen months from now when the building’s on fire.

The public half, though, has been in your trust set the whole time, embedded in the binary and served over WKD right alongside the signing keys. That’s the trick. Because every installed client already trusts the rotation authority, it can vouch for a brand-new signing key outside the normal sign-with-the-old-key path. You bring the private half out of the safe, use it to authorise the new key, ship that, and installed tools adopt the replacement on their next update, all without the dead primary key ever having to sign a thing.

It is, deliberately, a key you hope never to touch. But a break-glass key you forgot to cut is just a pane of glass.

The whole chain, end to end

That’s the series. Step back and the shape is one clean line: a signing key is born inside KMS and never leaves it (Part 2); its public half is minted out of the HSM without the private bytes ever surfacing (Part 4); that public key is published off-platform over WKD, somewhere your release host can’t quietly rewrite (Part 4), and embedded into the binary as a required trust anchor (Part 5); every tagged release is signed through the key over short-lived OIDC credentials with no stored secrets (Parts 3 and 6); a stranger’s copy of your tool verifies its own updates against that anchor before trusting a byte (the cross-check); and when the day comes, the whole thing is rotatable without locking anyone out. A key that can’t be stolen, can’t be forged, and can still be replaced.

If you’ve followed all seven parts, you’ve built that. If you’ve dipped in for one piece, the pillar ties the lot together and points at the deep-dives behind the why. Either way, your users are getting updates they can actually trust, which was the whole point. Go and leave your supply chain better than you found it.

Sign your own binaries with go-tool-base, part 6: sign every release with GoReleaser

Mon, 22 Jun 2026 00:00:00 +0000

By now you’ve got all the pieces lying on the bench. A KMS key that signs but never hands over its private half (Part 2). A CI role you can assume over OIDC with no stored credentials (Part 3). A public key you’ve minted and published (Part 4), and embedded in the binary (Part 5). What you don’t have yet is the bit that makes it routine: a release that signs itself, every time, without you remembering to do anything. That’s this part. We wire signing into the tagged-release pipeline so that pushing a v* tag is the whole ceremony.

This is the part of the signing series where the chain stops being a set of commands you run by hand and becomes something the pipeline does for you. We’re using GoReleaser, which already builds your binaries, writes a checksums file and cuts the release. It needs one extra trick: sign the checksums on the way out, through the KMS key, using credentials that only exist for the length of the job. And you don’t hand-write that trick into your release config. gtb does.

You’ll need a working GoReleaser setup releasing a go-tool-base CLI, plus the KMS key, the signer role and the embedded public key from the earlier parts.

One command wires the signing in

In Part 5 you ran gtb enable signing to turn on the verifying side: embed the key, check every update against it. Now you give that same command the key the release pipeline should sign with, and it wires the producing side too:

gtb enable signing --key-id alias/acme-release-signing-v1

That records the KMS key in your .gtb/manifest.yaml and regenerates .goreleaser.yaml with a signs: block that calls gtb sign. The WKD email and everything else you set in Part 5 stay exactly as they were; you’re adding the key, not starting over. Because the block is generated, you don’t hand-edit the release config any more than you hand-edit the embed wiring. Change the key later (a new region, a rotated alias) and you re-run the command, not the YAML.

The generated block looks like this:

signs:
 - id: checksums
 cmd: gtb
 args:
 - "--ci"
 - "sign"
 - "--backend"
 - "aws-kms"
 - "--kms-region"
 - "eu-west-2"
 - "--key-id"
 - "alias/acme-release-signing-v1"
 - "--public-key"
 - "internal/trustkeys/keys/signing-key-v1.asc"
 - "--output"
 - "${signature}"
 - "${artifact}"
 artifacts: checksum
 signature: "${artifact}.sig"
 output: true

The region defaults to eu-west-2 and the public key to the internal/trustkeys/keys/signing-key-v1.asc you embedded in Part 5; pass --kms-region or --public-key if yours differ. The backend defaults to aws-kms, which is the one that matters in CI.

Why only the checksums

artifacts: checksum is the line doing the load-bearing work. It tells GoReleaser to run the signing command once, over the checksums manifest only, not over every binary and archive in the release.

Here’s why that’s enough. GoReleaser builds your binaries, then writes a checksums.txt listing the SHA-256 of each one. Every artefact in the release is named in that file by its hash. So if you sign the manifest, you’ve transitively vouched for everything it lists: change a single byte of any binary and its hash no longer matches the line in checksums.txt, and the moment you alter checksums.txt to cover for that, the signature over it breaks. One signature, the entire release covered, through the hash chain. The per-binary build stays completely untouched, which keeps reproducible builds reproducible. signature: "${artifact}.sig" names the output checksums.txt.sig, the detached, ASCII-armored OpenPGP signature your tool looks for on every self-update.

No shim, just `gtb`

If you go reading go-tool-base’s own release config, you’ll find its signs: block points at a scripts/sign-release.sh shim rather than calling gtb directly. Yours doesn’t, and the difference is worth understanding.

go-tool-base is signing the very binary it’s in the middle of building, so it can’t use an installed gtb to do it. Its shim runs go run ./cmd/gtb to build a throwaway signer from source. It also reads the key id, public key and region from environment variables, because the one config has to serve every build. Your tool has neither problem. It calls the gtb you already installed, and the generator already knows your key id, region and public-key path because you just told it. So there’s nothing for a shim to abstract: the whole invocation goes straight into args:, where you can read it.

The one thing you do still need is gtb on the release runner’s PATH. Install it in a before_script, or bake it into your CI image, the same way you would any other release tool.

Where the credentials come from

gtb sign asks KMS to sign. KMS will only oblige if the caller has credentials, and the whole point of keyless CI is that there are no stored credentials to leak. So they’re minted on the fly.

On GitLab, the release job declares an id_tokens: block. GitLab injects a short-lived OIDC token (a JWT) into the job, the before_script writes it to a file, and the AWS SDK’s default credential chain picks it up from there. No aws CLI call, no assume-role-with-web-identity you write yourself: set AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE and the SDK does the web-identity exchange the first time gtb sign touches KMS:

goreleaser:
 id_tokens:
 AWS_WEB_IDENTITY_TOKEN:
 aud: sts.amazonaws.com
 variables:
 AWS_REGION: eu-west-2
 AWS_ROLE_ARN: arn:aws:iam::…:role/acme-release-signing-v1-signer
 AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/oidc-token
 before_script:
 - echo "$AWS_WEB_IDENTITY_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"

The key id and region live in the generated signs: block now, not here, so CI only supplies the AWS credentials. AWS_REGION still earns its place: the --kms-region flag points the KMS client at the key, while the SDK uses AWS_REGION for the STS exchange that mints the credentials in the first place. The aud has to match the audience the signer role’s trust policy expects (Part 3 set this up; for the OIDC provider go-tool-base uses, that’s sts.amazonaws.com). On GitHub the moving parts are the same, you just let aws-actions/configure-aws-credentials do the token-to-credentials dance instead of writing the file yourself.

Those credentials are scoped tight. The signer role’s trust policy pins it to this project’s tag pipelines, so even if the role ARN leaked, nothing but a release tag on your repo can assume it.

Don’t sign when there’s nothing to sign with

A local goreleaser release --snapshot, or a CI run that isn’t a release, has no OIDC token and no business reaching for KMS. GoReleaser is told to skip the whole signing step in that case: the release job runs with --skip=sign unless the web-identity token is present. So a non-release build never so much as looks at KMS, which is exactly what you want when you’re iterating on your laptop. The signing only fires on the real thing.

Cut a release

With all of that in place, releasing is one push:

git tag v1.4.0
git push origin v1.4.0

The tag pipeline fires. GoReleaser builds the binaries, writes checksums.txt, calls gtb sign, gtb asks KMS to sign over OIDC, and checksums.txt.sig lands next to the manifest. Both get attached to the release. Nobody typed a signing command and no private key was anywhere near the runner.

A two-person gate, if you want one

There’s a window worth thinking about: a compromised CI runner during a release could, in principle, ride the OIDC credentials to get one malicious thing signed. You can shut that window with an approval gate in front of the signing job. On GitLab, a protected environment with a required approval (or a manual job) makes the release wait for a second pair of eyes; on GitHub, “required reviewers” on the release environment does the same. It’s optional, and it adds friction to every release, so weigh it against how exposed your runners are. For a lot of projects the OIDC scoping alone is enough; for anything where a forged release would be a genuine incident, the gate is cheap insurance.

Where this leaves you

Every release you cut from here on carries a signature made by a key you control and verifiable by anyone, and you got there by pushing a tag. That closes the loop the series has been building toward: the production side now actually produces signed releases.

Which means you can finally pull the trigger on the bit Part 5 left primed. We embedded the public key and left enforcement off, because turning it on before you ship signatures would brick every update. Now that signatures are shipping, Part 7 deals with the part everyone skips and nobody can afford to: rotating the key, and getting yourself out of trouble if one ever goes bad, without locking your users out.

Sign your own binaries with go-tool-base, part 5: embed the key and require verification

Sun, 21 Jun 2026 00:00:00 +0000

By now you’ve got a public key your tool can publish off-platform: minted from a KMS-held private key in Part 4 and served over WKD. That’s half the trust loop. The other half lives inside the binary itself: the tool has to hold a copy of the key it expects so that when an update lands, it can check the signature against something an attacker who owns the release page can’t quietly swap. This part bakes that trust anchor in, wires the self-updater to use it, and turns enforcement on without locking out the people who already have your tool installed.

That last clause is the one that bites, so we’ll come to it slowly. First, turning signing on.

Enable signing with one command

Your root command is generated by gtb, so you don’t wire signing in by hand-editing it. You turn the feature on and let the generator do the wiring:

gtb enable signing --email release@acme.dev

That one command does three things, all in generated, regenerable code you never touch:

scaffolds an internal/trustkeys package that //go:embeds your release keys and hands them to the self-updater;
wires Signing: props.SigningConfig{EmbeddedKeys: trustkeys.Keys()} into your generated root command;
writes a signing.go holding the enforcement defaults, generated from a signing block it adds to your .gtb/manifest.yaml. You change posture by re-running the command, never by editing the file.

Signing is off until you run this, on purpose: it needs a key and a published WKD endpoint, so a freshly generated tool doesn’t carry it uninvited. (If you’re curious what the embed package looks like, it’s a small //go:embed all:keys over an internal/trustkeys/keys/ directory with a Keys() [][]byte accessor. The all: prefix is load-bearing: a plain //go:embed keys won’t compile over a directory that holds only a dotfile, so the scaffold keeps a .gitkeep there. You don’t write any of it.)

Drop your key in

The scaffold gives you an empty internal/trustkeys/keys/. Put the public key you minted in Part 4 into it, alongside the break-glass key:

cp signing-key-v1.asc internal/trustkeys/keys/
cp rotation-authority.asc internal/trustkeys/keys/

The -v1 in the filename isn’t decoration: you’ll rotate one day and embed signing-key-v2.asc alongside it for a release or two (Part 7). The rotation-authority key rides along the same way. Rebuild, and trustkeys.Keys() now returns them. With no .asc present it returns nothing and verification stays dormant, so enabling signing before you have a key breaks nothing.

The WKD cross-check, via `--email`

The --email you passed is doing real work. The embedded key alone is a static anchor (how the verification works): it only ever says “this was the key on the day I was built.” Pairing it with the live WKD copy you published in Part 4 gives a second, independent source the release platform can’t reach. The default key source is both, an embedded-plus-WKD CompositeResolver, and the email is what lets the updater derive the WKD URL. Leave --email off and both quietly degrades to embedded-only.

For a locked-down tool that should refuse to update when it can’t reach WKD, rather than fall back to the embedded key, enable the strict cross-check:

gtb enable signing --email release@acme.dev --require-external-crosscheck

Most tools want the softer default, where WKD strengthens verification when it’s reachable and the embedded key still works when it isn’t.

Confirm the cross-check is actually firing

It’s easy to get this wrong by leaving the email out, and when you do, nothing complains: the updater quietly verifies against the embedded key alone and carries on. The way to know which anchors were actually consulted is to read the log. Every update prints a line naming the resolver it used:

INFO signature verified resolver=composite[embedded,wkd:openpgpkey.acme.dev]

That resolver= field is the whole tell:

composite[embedded,wkd:...] is what you want: both the embedded key and the WKD-served key were fetched, their fingerprints agreed, and the signature checked against the result. The cross-check is live.
embedded means only the baked-in key was used and WKD was never consulted. That’s the silent-degrade trap: key_source is "both", but with no external email there was no WKD URL to derive, so it fell back to a single anchor. If you see this after passing --email, the value didn’t take.
wkd:... on its own is the reverse: WKD was consulted but nothing was embedded.

There’s a matching update signature verification configured resolver=... line at the very start of an update, before any network call, if you’d rather see the choice before the fetch. Two failure shapes are worth recognising too. WARN composite resolver failed (RequireAll=false, continuing) means the WKD fetch fell over (a 404, a flaky network) and the update carried on against the embedded key alone, the soft default you can harden with --require-external-crosscheck. ERROR ErrKeyResolverMismatch is the one you want to see fire in anger: the embedded and WKD keys disagreed, which is exactly the tamper alarm the whole scheme exists to raise (the same mismatch the verification deep-dive walks through).

Require it, in the right order

Here’s the part everyone trips over. Enabling signing does not yet require it: require_signature stays off, and that’s deliberate. Turning it on too early breaks self-update for everyone who already has your tool.

Think about what an existing install holds. A user on an old version has a binary built before you enabled signing. It has no trust anchor. If the first thing it ever sees is a signed, signature-required release, it has nothing to check the signature against, and the update is refused. You’ve locked out exactly the people you were protecting.

The fix is to ship the key ahead of the requirement, so the anchor is already on the user’s machine by the time the first mandatory signature arrives. gtb follows the rollout in its own docs/development/phase2-signing-prep.md, across three releases:

Ship the key before you require the signature, or you lock out every install that predates it. In order:

Release N+1. Enable signing (above) and ship it, with require_signature off. Existing installs pull this update on checksum alone and pick up the embedded key as a side effect.

Release N+2. Ship your first signed release (Part 6 wires GoReleaser). Still not required: the signature is verified when present but not enforced, so nothing breaks.

Release N+3. Now, and only now, turn enforcement on.

When you reach N+3, it’s one command, not a code edit:

gtb enable signing --require-signature

That flips require_signature in the manifest and regenerates signing.go, so the change is tracked and reproducible. Skipping the middle release, requiring a signature before you’ve shipped one, is the exact mistake the ordering exists to prevent; gtb’s own rollout cites v0.12.2 as its first signed release for precisely this reason.

The checksum floor from Phase 1 sits underneath all this and is already on; signature verification adds to it, it doesn’t replace it. And even with signatures required, an end user genuinely stuck on a legacy release can escape with the update.require_signature config key or your tool’s <PREFIX>_UPDATE_REQUIRE_SIGNATURE=false. It’s an escape hatch, not the front door, but it means a requirement you turn on can’t permanently strand anyone.

Where this leaves you

Your binary now carries the key it expects, checks every update against that key and its live WKD twin, and refuses anything that doesn’t match, all without a gpg install on the user’s side and without stranding the installs that came before the key. The trust loop you built by hand in Part 1 now runs on its own, inside a stranger’s copy of your tool.

The one thing still missing is the signature itself on each release. Right now nothing is actually producing checksums.txt.sig. Part 6 wires the KMS signing from Parts 2 and 3 into a real GoReleaser pipeline, so every tagged release comes out signed without you touching a key.

Sign your own binaries with go-tool-base, part 4: mint and publish your public key

Sat, 20 Jun 2026 00:00:00 +0000

By the end of Part 3 your release pipeline can sign through a KMS key it never holds, over credentials that expire in minutes. The private half is locked away exactly where you want it. There’s a snag, though: a signature is no use to anyone who can’t get hold of the matching public key, and a KMS key won’t hand you one. KMS deals in raw signing operations, not OpenPGP entities. So this part does two things: it produces the published public key from the KMS key without ever touching the private bytes, then puts that key somewhere your release platform can’t reach.

That last bit is the part people skip, and it’s the part that does the real work. The whole scheme in a signature the platform can’t forge rests on the verifying key living somewhere an attacker can’t poison in the same breath as the release. We’ll come back to why that matters once the key is in hand.

Mint the public key from KMS

gtb keys mint builds an OpenPGP public key out of a signing backend. In Part 1 the backend was a .pem on disk; now it’s aws-kms, and only that flag changes:

gtb keys mint \
 --backend aws-kms \
 --kms-region eu-west-2 \
 --key-id alias/acme-release-signing-v1 \
 --name "Acme Releases" \
 --email release@acme.dev \
 --created "2026-06-02T00:00:00Z" \
 --output signing-key-v1.asc

INFO Minted OpenPGP key backend=aws-kms key_id=alias/acme-release-signing-v1 output=signing-key-v1.asc creation_time=2026-06-02T00:00:00Z fingerprint=...

Here’s the neat part, and it’s worth pausing on. An OpenPGP key isn’t just a lump of public-key material; the key carries a self-signature, a signature it makes over itself to bind the name and email to the key. So minting a public key normally needs the private key to do that signing. mint doesn’t. It wraps the backend in a crypto.Signer, and every signing operation, the self-signature included, becomes a kms:Sign call. KMS does the maths inside the HSM and hands back a signature; no private byte is ever exported, not even to stamp the key’s own identity onto itself. The mechanism is the subject of a signing key that never leaves KMS if you want to see how the signer is wired up.

Two things to get right. First, the AWS credentials: minting needs both kms:GetPublicKey and kms:Sign on the key, because it reads the public material and then signs the self-signature with it. The signer role you stood up in Parts 2 and 3 can do both; running this locally, your own credentials need the same. Second, and this is the same lesson Part 1 hammered on, pin --created. An OpenPGP fingerprint is derived partly from the creation time, so a different timestamp gives you a different fingerprint and, in effect, a different key as far as your tooling is concerned. Use the moment the KMS key was created and never let it drift: the key you embed in Part 5 and the key you publish here have to be byte-identical, and --created is what guarantees it.

Mint the rotation authority while you’re here

You won’t use it until Part 7, but the rotation-authority key is far easier to create now, alongside everything else, than to bolt on in a panic when you need it. It’s a break-glass key: an offline key whose only job is to vouch for a new signing key if the KMS one ever has to be replaced. The spare front-door key you tape to the back of a drawer, not the one on your keyring.

Because it’s break-glass, it isn’t a KMS key. You generate it on a trusted offline machine and the private half goes straight into cold storage:

gtb keys generate \
 --algorithm ed25519 \
 --name "Acme Rotation Authority" \
 --email release@acme.dev \
 --created "2026-06-02T00:00:00Z" \
 --output rotation-authority.asc

INFO Generated OpenPGP keypair algorithm=ed25519 public_output=rotation-authority.asc private_output=rotation-authority.priv.asc creation_time=2026-06-02T00:00:00Z fingerprint=...
WARN Move the private-half file to offline storage now. private_output=rotation-authority.priv.asc

For Ed25519 you get an armored public .asc and an armored secret-key block (.priv.asc, the same wire format gpg --export-secret-keys produces). Do what the warning says, and do it before you forget: move rotation-authority.priv.asc to offline storage immediately. An encrypted USB stick and a paper backup is not paranoid for a key you might not touch for two years and will desperately need when you do.

Notice it carries the same release@acme.dev user ID as the signing key, not a separate rotation@ address. That’s deliberate, and it’s the one place people trip: WKD groups keys by the email in their UID, so the matching address is what puts the rotation authority in the same bucket as the signing key, and into the same embedded trust set. Give it a different email and it quietly drops out of the published bucket, your embedded and WKD key sets stop matching, and the cross-check from Part 5 starts failing every update. Same release identity, two keys. The public half travels with the signing key from here on, published and embedded together.

Build the WKD tree

Now publish them. The way clients find a public key from an email address is the Web Key Directory: a fixed set of files under .well-known/openpgpkey/ on a web server, where each key lives at a path derived by hashing the local-part of its email. gtb keys wkd builds that tree for you, in pure Go, with no gpg or gpg-wks-client anywhere in sight:

gtb keys wkd \
 --domain acme.dev \
 --email release@acme.dev \
 --output ./wkd-staging \
 signing-key-v1.asc rotation-authority.asc

INFO WKD bucket email=release@acme.dev hash=... keys=2
INFO wrote path=wkd-staging/.well-known/openpgpkey/acme.dev/policy
INFO wrote path=wkd-staging/.well-known/openpgpkey/acme.dev/submission-address
INFO wrote path=wkd-staging/.well-known/openpgpkey/acme.dev/hu/...
INFO WKD tree complete output=wkd-staging method=advanced emails=1 files=3

Both keys carry release@acme.dev in their UID here, so they land in the same hu/ bucket, concatenated. (If you’d rather split them, give each a different email and pass --email twice.) The tree under ./wkd-staging holds a policy file (required by the spec, and empty), a submission-address file, and one hu/<z-base-32-hash> file per email with the keys inside.

One decision matters for where this gets served. The default --method is advanced, which serves the tree from a dedicated openpgpkey.acme.dev subdomain, which is why the path above has acme.dev nested inside it. Pass --method direct instead and the tree is served from acme.dev itself. Advanced is the modern default and what you want unless you’ve a reason otherwise; it does mean you’ll need DNS and TLS for openpgpkey.acme.dev.

Publish it somewhere the platform can’t reach

This is the bit that’s tempting to fudge, and the bit you mustn’t. Do not drop the WKD tree onto the same host, or under the same account, as your code and your releases.

Walk the attack through. Your binary will carry an embedded copy of the public key (Part 5). On every self-update it fetches the published key over WKD and checks the two agree before trusting a download. That cross-check is the entire defence. If an attacker who compromised your release platform could also rewrite the WKD tree, they’d swap both keys for one of their own, sign a malicious release with it, and the client would wave it straight through. The cross-check would be comparing a forged key against another copy of the same forged key. Worthless. The defence only holds if the published key and the embedded key come from infrastructure an attacker would have to breach separately. That argument is laid out in full in a signature the platform can’t forge; this is where you actually pay for it.

In practice that means a different host with its own credentials. A static host like Cloudflare Pages in Direct Upload mode does the job: you build the tree locally and push it with the Wrangler CLI under a token scoped to Pages edit and nothing else, no Git integration wired to your code repo. The token that can rewrite your keys has no power over your releases, and vice versa. For the advanced method you’ll also point DNS for openpgpkey.acme.dev at that host and let it terminate TLS. Whatever you pick, the test is simple: could one stolen credential change both the key in the binary’s update path and the key on the server? If yes, you’ve built a very elaborate way of trusting nobody.

Where this leaves you

You’ve turned a key locked inside KMS into a published, fetchable OpenPGP key without the private half ever surfacing, minted the offline rotation authority you’ll be glad of later, and put both somewhere your release platform can’t quietly rewrite. The verifying side of the loop now exists out in the world, ready to be checked against.

What’s missing is the checking. Part 5 bakes the public key into the binary as its trust anchor and turns enforcement on, so the tool refuses any update whose signature doesn’t hold, the moment that update lands.

Sign your own binaries with go-tool-base, part 3: keyless CI signing with OIDC

Fri, 19 Jun 2026 00:00:00 +0000

Part 2 left you with a KMS key your release pipeline can sign through and a role (<name>-signer) that’s allowed to call kms:Sign and nothing else. There’s one obvious question left hanging: how does a CI job become that role without an AWS access key stashed in a CI variable? Because a long-lived key sitting in your settings is exactly the kind of credential that ends up in a breach write-up. This part wires CI in with no stored credentials at all, on GitLab and GitHub.

The mechanism is OIDC federation, and the one-sentence version is: your CI platform already proves who it is to AWS, so let it. When a pipeline runs, GitLab or GitHub can hand the job a short-lived signed token (a JWT) that says “this is a tag pipeline on acme/acme-cli”. AWS trusts that token the same way a bouncer trusts a passport: it checks who issued it and what it says, and if the claims match a role’s trust policy, it swaps the token for temporary AWS credentials that live only for the job’s run. No key is stored anywhere; the credentials are minted on the spot and evaporate when the runner stops. The deep-dive on keyless CI covers the why and the threat model; here we just do the wiring.

Two things have to line up for this to work: the IAM OIDC identity provider (the thing in your AWS account that says “I trust tokens from this issuer”), and the signer role’s trust policy (which says “and only from these pipelines”). The terraform-aws-signing-kms module owns the second. The first comes from a sibling module.

Where the identity provider comes from

You don’t register the OIDC provider in the signing module. That’s deliberate: the same provider is shared across every role in the account that federates from CI, so it lives one level up, in terraform-aws-bootstrap. It takes a ci_provider input (github by default, or gitlab), registers the right IAM OIDC identity provider for that forge, and emits its ARN. You feed that ARN into the signing module. If you ran the bootstrap in Part 2 you already have it; if not, stand it up first and grab the oidc_provider_arn output.

That ARN is the only thing the two modules need to agree on. Everything else about which pipelines may sign lives in the signing module’s trust policy, which is what the rest of this part configures.

The GitLab path

Look the provider up by URL so you never hardcode the ARN, then pass it in along with the subject filter that scopes who can assume the role:

data "aws_iam_openid_connect_provider" "gitlab" {
 url = "https://gitlab.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn

 # oidc_issuer_host defaults to "gitlab.com"
 # oidc_audience defaults to "sts.amazonaws.com"

 ci_subject_filters = [
 "project_path:acme/acme-cli:ref_type:tag:ref:v*",
 ]

 key_administrator_arns = [/* ... from Part 2 ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

The interesting line is ci_subject_filters. GitLab stamps each CI token’s sub claim with the project path, the ref type and the ref. The pattern above reads as “tag pipelines on acme/acme-cli, for any ref starting v”. A branch pipeline or a merge-request pipeline carries ref_type:branch instead, so it simply doesn’t match, and the role refuses to be assumed. Your signer can only be driven from a release tag, which is the whole point: a dependency author opening an MR can’t trick CI into minting a signature.

On the pipeline side, the release job declares an id_tokens block so GitLab issues a token with the right audience, writes it to a file, and the AWS SDK picks it up:

goreleaser:
 rules:
 - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
 id_tokens:
 AWS_WEB_IDENTITY_TOKEN:
 aud: sts.amazonaws.com
 variables:
 AWS_REGION: eu-west-2
 AWS_ROLE_ARN: ${SIGNER_ROLE_ARN}
 AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/aws-token
 before_script:
 - echo "$AWS_WEB_IDENTITY_TOKEN" > /tmp/aws-token
 script:
 - aws sts get-caller-identity
 - goreleaser release --clean

AWS_ROLE_ARN plus AWS_WEB_IDENTITY_TOKEN_FILE is the convention the SDK recognises: it sees the two together and calls assume-role-with-web-identity for you, so by the time goreleaser runs it’s already the signer role. The actual signing job is Part 6; the aws sts get-caller-identity line is just a sanity check that federation worked. It should print the signer role’s ARN.

The GitHub path

Same shape, different issuer and a different sub format. GitHub’s sub support landed in module v0.1.2: earlier versions validated ci_subject_filters against GitLab’s format only and would reject a GitHub subject outright, so pin the version.

data "aws_iam_openid_connect_provider" "github" {
 url = "https://token.actions.githubusercontent.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.github.arn
 oidc_issuer_host = "token.actions.githubusercontent.com"

 # oidc_audience still defaults to "sts.amazonaws.com"

 ci_subject_filters = [
 "repo:acme/acme-cli:ref:refs/tags/v*",
 ]

 key_administrator_arns = [/* ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

Two differences from GitLab. The oidc_issuer_host has to change, because it’s the prefix on the trust-policy condition keys (token.actions.githubusercontent.com:sub rather than gitlab.com:sub). And the subject format is GitHub’s own: repo:<owner>/<repo>:ref:refs/tags/v* scopes the same way the GitLab pattern did, to tag refs only. The audience stays sts.amazonaws.com, because that’s the default aws-actions/configure-aws-credentials requests, so there’s nothing to override.

The workflow side is the official AWS action. It needs id-token: write permission to ask GitHub for the token in the first place, and contents: write so GoReleaser can create the release:

permissions:
 id-token: write
 contents: write
jobs:
 release:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: aws-actions/configure-aws-credentials@v4
 with:
 role-to-assume: ${{ secrets.SIGNER_ROLE_ARN }}
 aws-region: eu-west-2
 audience: sts.amazonaws.com
 - run: aws sts get-caller-identity
 - run: goreleaser release --clean

If you forget id-token: write, GitHub never issues a token and the action fails before it reaches AWS. It’s the most common GitHub-side trip-up.

The gotcha that wastes an afternoon: token audience

Here’s the one worth burning into memory, because the error message points you at the wrong thing. An IAM OIDC provider carries a client_id_list, and it rejects any token whose aud claim isn’t on that list before AWS even looks at the role’s trust policy. So if your CI token’s audience and the provider’s client ID don’t match, you get:

InvalidIdentityToken: Incorrect token audience

That reads like a role-trust problem, and you’ll go round in circles editing ci_subject_filters, but the trust policy was never consulted. The fix is to keep the audience aligned to sts.amazonaws.com everywhere: it’s the module’s oidc_audience default, it’s what terraform-aws-bootstrap puts on the provider’s client_id_list, it’s the aud in the GitLab id_tokens block, and it’s the default aws-actions/configure-aws-credentials requests. Leave all four alone and they agree by default. The moment you override one, override all of them, or you’ll meet that error.

Where this leaves you

There’s now no AWS access key anywhere in either forge. The pipeline proves it’s a release tag on your project, AWS hands it the signer role for the length of the run, and the credentials are gone the moment the runner stops. The signing module’s trust policy is the gate; the bootstrap module’s OIDC provider is the lock it hangs on. Both are verifiable in the public modules: terraform-aws-signing-kms for the role and terraform-aws-bootstrap for the provider.

The role can sign, but you still can’t verify anything yet, because nobody has the public half of that KMS key. Part 4 fixes that: gtb keys mint pulls the public key straight out of KMS, and gtb keys wkd publishes it somewhere the release platform can’t touch.

Sign your own binaries with go-tool-base, part 2: a signing key in AWS KMS

Wed, 17 Jun 2026 00:00:00 +0000

Part 1 left you with a working signing loop and one glaring weakness: the private key was a .pem on your laptop, and files get copied. This part fixes that. You’ll generate the production signing key inside AWS KMS, where it’s created and never comes out, and stand up a role that can sign with it. The key itself is the only thing we build here; wiring CI in over OIDC is Part 3.

The big idea is the same one the deep-dive spends its whole length on, so I’ll keep it short: you never hold the private key and you never sign with it. You ask KMS to sign on your behalf with kms:Sign, and the private half stays inside the HSM for its entire life. There is no export, no download, no “just this once” copy onto a runner. An attacker who owns your CI still can’t walk away with the key, because the key was never on the runner to begin with.

What you’ll need first

This part is OpenTofu (or Terraform, the module is fine with either). You’ll need:

An AWS account you can apply infrastructure into.
An IAM OIDC identity provider already registered in that account. KMS doesn’t need it, but the signer role we create trusts it, so it has to exist. If you haven’t got one, the sibling module terraform-aws-bootstrap provisions it (it’s the same family as the terraform-aws-security-baseline module). Its oidc_provider_arn output is exactly what we feed in below.

You don’t need a CLI, a pipeline or a public key yet. This is just the vault and the key that lives in it.

The module

The key, its alias, the signer role and the full key policy come from one public module, terraform-aws-signing-kms. Here’s the whole consumer block:

data "aws_iam_openid_connect_provider" "gitlab" {
 url = "https://gitlab.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 description = "Acme release binary signing"

 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn
 ci_subject_filters = ["project_path:acme/acme-cli:ref_type:tag:ref:v*"]

 key_administrator_arns = [ /* operator role + account root */ ]
 automation_role_arn = data.aws_iam_role.automation.arn
}

# Carry these forward: Part 3 wires the signer role into CI, Part 4 mints
# the public key from the alias.
output "signer_role_arn" {
 value = module.signing_kms.signer_role_arn
}

output "signing_key_alias" {
 value = module.signing_kms.key_alias_name
}

A few of those values are doing more than they look, so let’s walk them.

name is acme-release-signing-v1, with the v1 on the end deliberately. The name derives the role (<name>-signer) and the alias (alias/<name>), and both of those want to outlive the key. When you rotate to a new key in Part 7 you’ll mint a -v2 and repoint things, so bake the version in now rather than wishing you had.

ci_subject_filters is in GitLab’s OIDC sub format here, and it’s the line that says which pipeline is allowed to assume the signer role: tag pipelines for any v* ref on acme/acme-cli, and nothing else (no branch builds, no merge requests). It’s the heart of Part 3, so I’ll leave the full explanation there. For now, know it’s not optional: an empty list would trust every token from the issuer, and the module refuses to let you do that.

key_administrator_arns and automation_role_arn are the two roles that manage the key, and the distinction between them matters. More on that in a moment.

tofu init
tofu apply

What you just built

A single asymmetric KMS key, plus the IAM scaffolding around it. The key is created with key_usage = SIGN_VERIFY and the spec RSA_4096, which is the module’s default and the one you want.

The obvious question, if you’ve signed things before, is why RSA-4096 and not Ed25519, which is smaller and faster. Two reasons, and neither is preference. The first is that AWS KMS simply doesn’t offer Ed25519 for asymmetric signing, so it’s off the table the moment you decide the key lives in KMS. The second is that OpenPGP, the format your signatures end up in, ties its packet encoding to the signing algorithm: the algorithm isn’t a detail you can swap underneath, it’s written into the bytes. RSA-4096 is the spec that satisfies both constraints, so it’s the secure default and you shouldn’t need to touch it.

Two more things to note about the key itself:

enable_key_rotation is off, and that’s intentional. AWS’s automatic yearly rotation only works on symmetric keys; an asymmetric SIGN_VERIFY key can’t be auto-rotated, because a new key would mean a new public half and every embedded trust anchor breaking at once. Rotation for signing keys is a deliberate, staged operation (mint a new key, publish it, repoint the alias), which is its own part later in the series.
The deletion window defaults to 30 days, the longest AWS allows. For a key this important, the longest possible “oops, undo” window is the safe choice.

The module also creates a stable alias, alias/acme-release-signing-v1. Always reference the key through its alias, never the raw key ID. The alias is what survives rotation: when v2 arrives, the alias gets repointed and everything calling through it keeps working.

Four principals, one key policy

This is the part worth slowing down for. A KMS key is governed by its key policy, and this module writes one policy that names four classes of principal, each with deliberately different reach:

The account root keeps kms:*. That’s AWS’s recommended break-glass: if every other path is locked out, the account owner can still recover.
The key administrators (key_administrator_arns, typically your operator role plus root) can administer the key, schedule its deletion, that sort of thing. They are not signers, and they can’t sign.
The automation role (automation_role_arn, the role your infra apply pipeline assumes) can manage the key as a Terraform resource, read it, tag it, even change its policy. What it deliberately cannot do is kms:Sign. Think about why: the role that applies your infrastructure runs on every change, so if owning that role let an attacker mint signatures, you’d have handed the whole point of the exercise back. Managing the key and using the key are two different powers, and only the signer gets the second one.
The signer role can call kms:Sign, kms:GetPublicKey and kms:DescribeKey, on this one key, and that’s the entire list. It can’t read other keys, can’t administer this one, can’t delete anything.

Here’s the detail that catches people out: the signer role has no attached IAM policy at all. Its permissions live entirely in the key policy. That’s not an oversight, it’s the design. One document, the key policy, is the single source of truth for who can do what to this key, so there’s no second place to check and no way for an attached role policy to drift out of sync with the key policy and quietly grant something nobody intended. If you want to know who can sign, you read one file.

The signer role is assumable via OIDC (those ci_subject_filters again), which is what lets a CI job step into it without any stored credential. That federation is Part 3’s whole job.

The outputs you’ll carry forward

The module hands back everything the later parts consume. The ones you’ll actually use:

key_id and key_arn, the bare ID and full ARN of the key.
key_alias_name and key_alias_arn, the stable alias. key_alias_name is the one you’ll pass to gtb as the KMS key reference, because it survives rotation.
signer_role_arn and signer_role_name, the role CI assumes to sign. signer_role_arn becomes an environment variable in your pipeline next part.

Keep these to hand. Part 3 needs signer_role_arn; Part 4 needs key_alias_name to mint the public key out of KMS.

What this costs

Worth a quick word, because this is the one part of the series that puts a line on an AWS bill. An asymmetric KMS key runs about a dollar a month, plus a tiny per-signature charge on the kms:Sign calls. For release signing, where you sign a handful of checksums files a month, the per-signature cost rounds to nothing. A dollar a month for a key that can’t be stolen off a laptop is the cheapest security control in this entire series.

Where this leaves you

The key exists, it lives somewhere it can never leave, and there’s a role that can sign with it (and a separate role that pointedly can’t). What there isn’t yet is any way for your release pipeline to become that role without a long-lived credential sitting in CI.

That’s the gap Part 3 closes: federating GitLab and GitHub into the signer role over OIDC, so a tagged release can assume it for the length of one job and nothing is stored anywhere. The ci_subject_filters line we glossed over here is where it starts.

When you hand the same key to every call

Tue, 16 Jun 2026 00:00:00 +0000

I was building a tutorial, the kind where the whole point is that the reader runs every command and it just works. So I generated a fresh project with go-tool-base, added a command, then added a command underneath that command, and hit build. It didn’t.

pkg/cmd/hello/cmd.go: props.ChildCmd undefined
 (type *props.Props has no field or method ChildCmd)

My own generator, in my own framework, had just written code that referenced a thing that didn’t exist… which is a special kind of embarrassing.

A bug with a two-month alibi

git blame walked me straight to the commit that introduced the command middleware system back in March. Middleware here is the web-style idea of wrapping a command’s run function with cross-cutting behaviour, timing, auth, recovery, that sort of thing. To wire it in, the generator started emitting this for a nested command:

setup.AddCommandWithMiddleware(cmd, child.NewCmdChild(p), props.ChildCmd)

where the line before had simply been:

cmd.AddCommand(child.NewCmdChild(p))

The catch is that third argument. props.ChildCmd is meant to be one of a set of constants, but those constants are hand-declared for the framework’s built-in commands only (UpdateCmd, DocsCmd, and friends). The generator never declares one for a user’s child command, so the generated parent referenced a name that nothing had ever declared. Undefined. Won’t compile.

Here’s the part that should worry you more than the bug. It shipped in March and nobody noticed until late May. Partly because it only bites nested commands, a command under another user command; top-level commands register by a different path and were fine. But mostly because the generator’s tests checked the generated code as text, asserting that it contained the right strings, and never once ran go build on the result. CI was green for two months on code that could not compile. We were grading the essay without ever reading it aloud.

What the key actually was

Once I stopped staring at the missing name, the real problem came into focus… and it wasn’t the missing constant at all.

That third argument is a middleware lookup key. The framework keeps a table of middleware registered against each key, and the key tells it which to apply. It is not an on/off switch and it is not optional, so the generator had to supply one at every registration site. It was being asked to guess, on every call, a value it had no reliable way to produce for a user command.

And the tell was sitting right there in the same generator: everywhere else, the idiom was props.FeatureCmd("name"), a function that derives a key from a string. The nested-registration path was the one place that assumed a hand-declared constant instead. One call site out of step with all the others.

That is the actual lesson, and it has nothing to do with cobra or codegen. When you find yourself threading the same derived value through every single call site, and getting it wrong, the value is in the wrong place. The feature key was never the caller’s business. It belonged to the command.

Changing my mind about cobra

This is where I had to eat a helping of my own opinion.

go-tool-base is built on cobra, the de-facto Go library for building command trees, and I like it a great deal. I had deliberately not wrapped it. Every abstraction over a good library is a tax the reader pays, so my standing rule was: use cobra directly, don’t hide it behind something of mine.

The trouble is the middleware pattern kept growing, and the bigger it got the more plainly “don’t abstract cobra” was a position I was holding past its evidence. The very thing I’d refused to build was the thing the design had come to need. It helped that, having recently moved the project to GitLab, the version had reset to a 0.x prerelease, which makes a breaking change cheap. The window to stop patching and do it properly was open, and not for long.

Go’s composition model made it almost painless. You can embed a pointer to one struct inside another and the outer type inherits all the inner one’s methods for free, which is about as close to monkey-patching as a statically typed language gets. So a setup.Command became a cobra command plus the feature it belongs to:

type Command struct {
	*cobra.Command
	Feature props.FeatureCmd
}

func Wrap(feature props.FeatureCmd, cmd *cobra.Command) *Command {
	return &Command{Command: cmd, Feature: feature}
}

func (c *Command) Register(children ...*Command) {
	for _, child := range children {
		if child.Command == nil {
			continue
		}
		if child.RunE != nil {
			child.RunE = Chain(child.Feature, child.RunE)
		}
		c.AddCommand(child.Command)
	}
}

Because *cobra.Command is embedded, a setup.Command is a cobra command for every method cobra offers; the one place you need the raw pointer, you reach for .Command. The generated command now carries its own identity:

func NewCmdChild(props *props.Props) *setup.Command {
	cmd := &cobra.Command{Use: "child", RunE: ...}
	return setup.Wrap(props.FeatureCmd("child"), cmd)
}

and the parent registers it with nothing threaded through the call:

parent.Register(child.NewCmdChild(p))

The feature key now lives on the command, derived from the command’s own name, which is the one place the generator can always produce it correctly. The bug isn’t so much fixed as made unsayable: there’s no call site left to write the wrong thing into. The wiring got cleaner on the way past, too, each command’s run is wrapped exactly once with its own feature, instead of the old recursive pass that re-applied the parent’s feature down the whole subtree. And the old free function stays on as a deprecated shim that just calls Register, so nothing downstream breaks before v1.0.

Changing my mind about the tests

The redesign was the satisfying fix. The test was the important one.

The reason a non-compiling generator sailed through CI for two months is that its tests read the generated source as text. A generator is a program that writes a program, and we were checking that it wrote the expected words without ever asking whether the words formed a working program. So the redesign shipped with a different kind of test, one that scaffolds a real project, adds a nested command, and actually builds it. Its own comment says the quiet part out loud:

The previous test suite asserted file-content shapes but never tried to go build the generated module, so the nested-command path that referenced undefined props.<Name>Cmd symbols compiled cleanly in tests and broke only when downstream users built their tools.

It’s gated behind an integration flag, because it shells out to the Go toolchain and that’s too heavy for every unit run, but it closes the exact gap that hid the bug. The only real test of a code generator is whether its output compiles.

What it comes down to

Three times in one bug I had to change my mind. I’d decided cobra shouldn’t be abstracted; the evidence said abstract it. I’d reached for the one-line patch; the evidence said redesign. I’d trusted tests that read the generated code; the evidence said build it. None of those were comfortable, and the version reset is the only reason the timing was kind.

Both technical lessons are worth keeping. When the same derived value is threaded through every call, the abstraction is in the wrong place. And the only proof that something which writes code actually works is to compile what it writes. But the one underneath both, the one I apparently have to keep relearning, is simpler than either: don’t get so attached to an implementation that you can’t change your mind when the evidence says it doesn’t fit. The framework is better for it, and it now has a composition seam to hang the features cobra doesn’t give us natively. A nested command that wouldn’t build was just the thing that finally made me look.

Sign your own binaries with go-tool-base, part 1: sign and verify on your laptop

Mon, 15 Jun 2026 00:00:00 +0000

The quickest way to understand release signing is to do it once, by hand, with nothing but a key on disk. No cloud account, no CI, no cost. This first part of the signing series walks the whole loop on your laptop: make a key, sign a file, and verify the signature, including with plain gpg so you can see it isn’t locked to anything of ours. Everything you learn here maps one-to-one onto the AWS KMS workflow in the later parts; only where the key lives changes.

You’ll need the gtb CLI (installation docs). Make a scratch directory to work in, because we’re going to create a few files.

Make a signing key

gtb keys generate creates a keypair entirely inside the process and writes both halves to disk:

gtb keys generate \
 --algorithm rsa \
 --name "Acme Releases" \
 --email "release@acme.dev" \
 --created "2026-06-01T00:00:00Z" \
 --output signing.asc

INFO Generated OpenPGP keypair algorithm=rsa public_output=signing.asc private_output=signing.pem creation_time=2026-06-01T00:00:00Z fingerprint=...
WARN Move the private-half file to offline storage now. private_output=signing.pem

Two files come out: signing.asc (the public half) and signing.pem (the private half, a PKCS#1 PEM). The private half is the thing you guard. There’s no on-disk passphrase in this version of gtb, so keep it under filesystem encryption (LUKS, FileVault, or wrap it with age) rather than leaving it lying about.

One flag is doing quiet but important work: --created. An OpenPGP key’s fingerprint is derived partly from its creation time, so if you let it default to “now”, every run produces a different fingerprint. Pin it to a fixed instant and the key is reproducible, which matters the moment you start embedding it in a binary. Get in the habit now.

Mint the public key you’ll actually publish

You could hand signing.asc around as-is, but we’re going to produce the public key a slightly different way, with gtb keys mint:

gtb keys mint \
 --backend local \
 --key-id signing.pem \
 --name "Acme Releases" \
 --email "release@acme.dev" \
 --created "2026-06-01T00:00:00Z" \
 --output release.asc

INFO Minted OpenPGP key backend=local key_id=signing.pem output=release.asc creation_time=2026-06-01T00:00:00Z fingerprint=...

mint wraps a signing backend in OpenPGP framing and writes out the armored public key. Here the backend is local (a PEM file on disk), but in production it’ll be aws-kms pointing at a key you can’t hold. Minting the public key from the backend is the one habit worth forming early: it’s the only way to get the public half of a KMS key, so doing it the same way locally means the rest of the series is identical bar one flag. release.asc is the key you publish and embed from here on. (Because we pinned the same --created, its fingerprint matches the generated one exactly.)

Sign something

A real release signs its checksums.txt, so make a stand-in and sign it:

printf 'abc123 acme_linux_amd64\ndef456 acme_darwin_arm64\n' > checksums.txt

gtb sign \
 --backend local \
 --key-id signing.pem \
 --public-key release.asc \
 checksums.txt

INFO Signed file backend=local key_id=signing.pem public_key=release.asc input=checksums.txt output=checksums.txt.sig ...

That writes checksums.txt.sig, a detached, ASCII-armored OpenPGP signature. Note gtb sign takes --public-key: it cross-checks that the backend key matches the public key you claim to be signing as, and refuses if they diverge, so you can’t accidentally sign with the wrong key.

Verify it, two ways

First, the way your tool will do it on every self-update: against the public key. That path is the subject of a signature the platform can’t forge and we wire it into a real binary in part 5. For now, prove the signature is sound with something every machine already has, gpg:

gpg --import release.asc
gpg --verify checksums.txt.sig checksums.txt

gpg: Signature made ...
gpg: using RSA key ...
gpg: Good signature from "Acme Releases <release@acme.dev>"

Good signature is the whole point. The signature gtb sign produced is an ordinary OpenPGP detached signature, so anyone can verify it with the standard tool, no go-tool-base required. (gpg will warn the key isn’t certified in its web of trust; that’s expected and unrelated to whether the signature is valid.)

Now change a byte of checksums.txt and run the verify again. gpg reports BAD signature. That failure is the entire reason any of this exists: a tampered manifest no longer matches the signature, and a tool that requires a valid signature will refuse the update.

Where this leaves you

You’ve signed a file with a key you made and verified it independently. That’s the complete trust loop in miniature, and the shape never changes: a private key signs, a public key verifies, and the two are produced and checked the same way whether the private half is a .pem on your laptop or an HSM-held key in AWS.

The local key was the easy bit, and also the weakest: it’s a file, and files get copied. Part 2 moves the private key somewhere it can’t be copied at all, AWS KMS, and the only command that changes is the --backend flag.

Sign your own binaries with go-tool-base

Sat, 13 Jun 2026 00:00:00 +0000

If your CLI tool can update itself, it has a decision to make that nobody is watching: when it pulls down a new version, should it trust what just landed? A checksum tells it the bytes match a manifest. It does not tell it who wrote the manifest. Close that gap and your users get updates they can actually trust; leave it open and a compromised release host can hand them anything it likes. This series is the end-to-end “how”, using the signing tooling built into go-tool-base.

By the end you’ll have a CLI that ships releases signed by a key you control, verifies its own updates against that key, and does the whole thing with no gpg wrangling and no long-lived secrets sitting in CI. We did the why and the how it works in two deep-dives already, a signature the platform can’t forge and a signing key that never leaves KMS. This is the use-it counterpart.

What you’re protecting against

Nobody’s coming to clean your supply chain, so it’s worth being clear about the threat before you spend an afternoon on the fix. A checksum file sits next to the binary on the same release page. Whoever can swap the binary can swap the checksum in the same breath, and the hash still matches. A signature is different: it’s made by a private key the release platform never holds, and verified against a public key your tool fetches from somewhere the platform can’t reach. To forge a release that passes, an attacker would have to steal a key that, done right, was never anywhere they could get at it.

That “done right” is the whole series.

Two paths through it

You don’t need a cloud account to start. The series runs in two stages:

Learn it locally. Part 1 signs and verifies on your laptop with a plain key on disk. No AWS, no CI, no cost. It’s the fastest way to see every moving part for real.
Do it for production. Parts 2 onward move the private key into AWS KMS, where it’s generated and never leaves, and wire your release pipeline to sign through it over short-lived OIDC credentials.

Each part stands on its own and ends with something that works. They build in order, but you can stop after Part 1 with a genuinely useful skill and come back for the cloud parts when you need them.

Before you start

You’ll want a CLI built on go-tool-base to sign. If you haven’t got one, the Building a CLI with go-tool-base series gets you there in an afternoon; this one picks up where releases come in. You’ll also need the gtb CLI installed (the installation docs have the one-liner), and for the cloud parts, an AWS account and a GitLab or GitHub project to release from.

The parts

Sign and verify on your laptop: gtb keys generate, gtb sign, and gpg --verify, the whole loop with a local key.
A signing key in AWS KMS: stand up an asymmetric KMS key with the terraform-aws-signing-kms module.
Keyless CI signing with OIDC: federate GitLab and GitHub into the signer role, no stored credentials.
Mint and publish your public key: gtb keys mint from KMS, then gtb keys wkd to publish it off-platform.
Embed the key and require verification: bake the trust anchor into your binary and turn enforcement on safely.
Sign every release with GoReleaser: wire signing into a real tagged-release pipeline.
Rotation and break-glass: the part everyone skips, and how to do it without locking anyone out.

Start with Part 1. By the time you reach the end, the chain runs from a key born in a vault to a binary on a stranger’s machine checking, on its own, that the update it just fetched is really yours.

The goblin that wouldn't stay dead

Fri, 12 Jun 2026 00:00:00 +0000

Turn one, the player swings, the die comes up 20, and my AI dungeon master narrates the goblin falling silent, leaving the player alone in the corridor. Good. Turn two, another roll, a 6 this time, and the same dungeon master cheerily has the goblin “dance back” out of the dark to take another swing. The goblin I’d just watched die was up and fighting again, and the model didn’t so much as blink.

I didn’t feel cheated, or even surprised. I felt the small, familiar thud of oh, yeah, I forgot that bit. Because the model hadn’t gone rogue. It had done exactly what a language model does. The gap was mine.

This was the war story behind part four of the go-tool-base tutorial, the AI dungeon master. The tutorial shows the clean, final design and quietly moves on. It doesn’t show the three different ways I got it wrong first, which is a shame, because the wrong turns are where the actual lesson is.

Why a dungeon master at all

A word on why I was even here. I was trying to prove the chat component of the framework to myself. There’s a voice that pipes up whenever I build anything in this space, “LangChain exists, who do you think you are?”, and the answer I keep landing on is that LangChain is enormous and I wanted something small enough to hold in your head. The tutorial was the test: could a newcomer wire AI into a CLI with it and come out the other side with something that actually behaves?

That last word is the whole problem. A tutorial has to leave you holding something dependable, and dependability is the one thing AI fights you on. I also wanted it to be fun, a thing someone might keep poking at after the tutorial ends, maybe even the hook that gets a person other than me to use the framework. I batted hook ideas around and liked none of them, until the obvious one landed: I run a tabletop game on the odd weekend, so make the AI the dungeon master. Gamify the thing. Then watch it raise the dead.

Strike one: nothing to enforce

The first version was the naive one. I gave the model a roll tool, because the one thing you absolutely cannot let a language model do is pick its own numbers, and otherwise let it narrate freely. The conversation history carried from turn to turn, so it remembered the fight. I assumed remembering was enough.

It isn’t. Remembering and being held to it are different things. The history told the model a goblin had died; nothing stopped it writing the goblin back in when the next turn’s narration wanted a bit of jeopardy. Memory is not a constraint. The model will happily contradict its own past if you’ve given it room to, and I had given it nothing but room.

Strike two: a tool to read the state

The obvious fix, and I do mean obvious, the kind you reach for without thinking, was to give the model a state tool so it could check who was alive before it narrated. Hand it the facts on request and surely it’ll stop making them up.

What it actually did was dither. Handed a tool it could call to look things up, it called it. And called it. And called it again, turning a turn over in its hands without ever committing to an action, burning through its step budget on lookups and leaving the player staring at nothing. I’d cured the lying by inventing paralysis. A tool the model can call is a tool it will call, often instead of doing the thing you actually wanted.

Strike three: refereeing its own dice

When I did get it reading state cleanly, the third failure crept in, and this one was subtler. Once the model could see the goblin’s hit points, it started deciding the fight. It would read that the goblin had 12 HP and just narrate a killing blow, hits and damage and all, without calling the roll or attack tools at all. Why ask the dice when you can see the board and write whatever outcome the story wants? Give a model enough context and it stops being a narrator and starts being a referee, which is precisely the job I’d built tools to keep out of its hands.

The fix was less, not more

Three failures, and notice the shape of my fixes: each one added something. More memory, then a tool, then more context. Every instinct said the model needed more to work with. Every time, the extra capability was the new way to be wrong.

So I went the other way. The truth lives in a plain Go struct that I own, not the model. There’s no state tool to dither on, because the loop simply prepends the current state to every turn’s input, fresh, so the model never has to ask and never gets to drift. The mechanics, the dice and the damage, live in Go functions the model has to call, and the system prompt says in as many words that it must not decide a hit or damage itself. The model is left with exactly one job: narrate. The prose is its to invent. The maths, the state and the shape of the result are not.

That’s the line that turned three bugs into a feature. You don’t make a language model reliable by giving it more to work with. You make it reliable by giving it less to be wrong about.

The freedom I chose not to give it

There’s a real tension in that, and I want to name it rather than pretend the boxed-in version is the only true one. At my own table the rules are guidelines, not guardrails. I ignore them, bend them, improvise, reach for the “rule of cool” when the moment’s better for it. A great AI dungeon master would have that same freedom, and a few out there genuinely do, Old Greg’s Tavern is a lovely example of how far the free-form version can go.

But that freedom costs far more than a tutorial can spend, and it buys unpredictability I was specifically trying to teach people to avoid. So I made a deliberate trade: guardrails instead of guidelines. Simple, but not so simple it’s boring. The player still gets a “not on rails” game, they can try anything and the DM copes, but every outcome that matters runs through code I trust. That’s the right shape for a tutorial, and, not by coincidence, the right shape for most AI features you’d actually ship.

What the goblin taught me

The thing I keep coming back to is that the model never misbehaved. It resurrected the goblin because I gave it the freedom to. It dithered because I gave it a button to press. It refereed because I let it see the board. Every failure was a permission I’d handed over without meaning to. The reliability didn’t come from a cleverer prompt or a bigger model, it came from working out, one dead goblin at a time, exactly how little the model needed to be trusted with.

If you want the version where it all works first time, the tutorial has it, the tool-calling and the typed turns wired up properly. This was the road there. The goblin, you’ll be glad to hear, now stays down.

A signing key that never leaves KMS

Thu, 11 Jun 2026 00:00:00 +0000

The last post in this series walked through how a tool verifies a release signature the platform can’t forge. That post had a loose end dangling off the back of it, and I knew it the whole time I was writing. Because a signature has to be produced by a private key… and a private signing key is the single worst thing in this entire story to lose. Steal it, and you sign malware that sails through every check I spent two posts building, signature and all. So where does that key live? The answer I landed on is the one this whole post is about: inside AWS KMS, and it never comes out.

The only key you can’t steal

Think about where a signing key normally ends up. A file on a build server. A secret in CI. A key on the release engineer’s laptop, “just for the release, I’ll delete it after”. Every one of those is a copy, and every copy is one more thing somebody can read, exfiltrate, or quietly clone while your back is turned. You can wrap them in passphrases and vaults and rotation policies all you like, and you’re still standing guard over a thing that exists in a place you don’t fully control.

The way out is almost annoyingly simple to state: the only key nobody can steal is the one that was never anywhere to be stolen from. So don’t hold the key at all. Let something else hold it, somewhere it has no export path, and ask that thing to sign for you.

That thing is AWS KMS. This is the infrastructure side of the question I opened the signing series with, finally answered with real Terraform.

A key that’s born in the box and stays there

The signing key is an asymmetric KMS key, and the module that provisions it is small enough to read in one sitting:

resource "aws_kms_key" "this" {
 description = var.description
 key_usage = "SIGN_VERIFY"
 customer_master_key_spec = var.key_spec # default RSA_4096

 # Asymmetric SIGN_VERIFY keys do not support KMS-managed rotation;
 # rotation is handled by minting a new key (alias = `<name>-v2`) and
 # publishing the v2 public key alongside the v1 key (dual-sign window).
 enable_key_rotation = false

 policy = data.aws_iam_policy_document.key_policy.json
}

The private half of that key is generated inside KMS and there is no API that hands it back to you. You don’t sign with it the way you’d sign with a file. You call kms:Sign: the bytes you want signed go up, a signature comes back down, and the key itself never moves. An attacker who completely owns my CI, my account, my laptop, can ask KMS to sign things for as long as their access lasts… but they can’t walk off with the key and keep signing forever. The blast radius is “while I’m compromised”, not “until I rotate a key I didn’t know had leaked three years ago”.

Why RSA-4096 and not the Ed25519 I’d normally reach for? Because KMS asymmetric signing doesn’t offer Ed25519, and OpenPGP’s packet format is tied to the algorithm that signed it, so the choice of key spec ripples all the way out to the signature on the wire. RSA-4096 is the strong option KMS does offer, so RSA-4096 is what the workflow is built around. A constraint of the box shaped the cryptography, not the other way round, and I’d rather say so than pretend I picked RSA on purpose.

Minting an OpenPGP key from a key you can’t hold

Here’s the part I find genuinely neat. OpenPGP wants a private key to self-sign its own public key when you generate it. And I don’t have a private key in any form I can hand to a library… it’s sitting in KMS, behind a door with no handle on my side. So how do you produce a valid OpenPGP public key at all?

go-tool-base leans on a small Go interface, crypto.Signer: anything that can return its public key and sign a digest. A KMS-backed signer satisfies it by turning each Sign call into a kms:Sign request. Then pkg/openpgpkey builds the OpenPGP entity around that signer:

func Entity(signer crypto.Signer, name, email string, creationTime time.Time) (*openpgp.Entity, error) {
	rsaPub, ok := signer.Public().(*rsa.PublicKey)
	// ...
	pubPkt := packet.NewRSAPublicKey(creationTime, rsaPub)
	// Construct the private-key packet directly (rather than
	// packet.NewSignerPrivateKey, which panics on opaque signers):
	// the crypto.Signer drives the actual signing, so a KMS-backed
	// signer works here.
	privPkt := &packet.PrivateKey{PublicKey: *pubPkt, PrivateKey: signer}
	// ...
}

Look at that PrivateKey packet. The field where OpenPGP expects the secret key material holds the crypto.Signer instead, which is to say, a remote handle to KMS. When the entity self-signs its public key, that self-signature is computed by KMS. gtb keys mint runs exactly this and writes out an ASCII-armored OpenPGP public key, and at no point did a single byte of private key material exist on the machine that minted it. The OpenPGP “private key” is a phone line to a vault, not a key.

That public key is what gets published off-platform over WKD and baked into the binary, the two trust anchors that post cross-checks.

Access without a human and without a standing key

A key that never leaves KMS is only as good as the rules about who may call kms:Sign. The signer role is deliberately narrow: it can call kms:Sign and kms:GetPublicKey on this one key and nothing else, and it is assumable only over OIDC from specific CI subjects, the same keyless federation the rest of the estate runs on. No human holds it. No long-lived access key sits in a CI variable waiting to leak. A release job federates in for its few minutes, signs, and the credentials evaporate with the runner.

So the chain of “who can sign a release” has no standing secret in it anywhere. Not a key file, not an access key, not a console user. Just a short-lived token, scoped to two API calls, on a key that can’t be exported.

The real cost: rotation is manual

This isn’t free, and the bit it taxes you on is rotation. KMS won’t auto-rotate an asymmetric SIGN_VERIFY key, which is why the module sets enable_key_rotation = false rather than leaving a default on. Rotating means minting a new key (a -v2 alias), publishing its public key alongside the old one, and running a dual-sign window long enough that clients have picked up the new anchor before you retire the old. It’s manual, it’s a runbook, and pretending otherwise would be the kind of thing this series exists to argue against. The trade I made was: a key with no exfiltration path, in exchange for rotation I have to do by hand. For a release-signing key, that’s the right side of the trade.

Why this is a command and not a script I hid

The origin of all this is a good deal less tidy than the result. I was working through the key-generation runbook, creating the offline rotation key with a gpg command I’d copied straight off my own page… and it just hung. No error, no prompt, just a cursor blinking while gpg waited on something it never bothered to mention.

My first instinct was the lazy one: drop the minting script into a scripts folder in my infra repo and never speak of it again. Then it nagged. That repo’s private, so the recipe would live somewhere nobody else could ever reach, and I’d already half-promised myself a tutorial walking people through this exact setup. So it shouldn’t sit in infra at all. It should be a gtb command, with a pluggable backend so anyone can swap my KMS for whatever provider they happen to run.

The deeper objection is the one that actually shaped it, though. I didn’t want to be shelling out to gpg by hand in the first place. gtb is a tool I hand to other people, and every time it drops to the shell for some gpg incantation, that’s an environment I’m asking the next person to reproduce, a dependency to install, a fiddly step to get subtly wrong, all before they can sign a single thing. The aim was to keep as much of this inside the box as I could: mint the key, build the WKD tree, produce the signature, all in pure Go, with no gpg on the path and no gpg-wks-client either.

So gtb keys mint pulls the public half out of your KMS key and frames it as OpenPGP, the trick from earlier; gtb keys wkd builds the tree ready to upload; and gtb sign produces the detached signature through that same remote round-trip. What comes out is an entirely ordinary OpenPGP signature gpg --verify is happy with, so you’re not locked into anything of mine. And none of it is just for me: build your tool on go-tool-base and the same handful of commands stands you up with this exact model, pointed at your own KMS. No cloud KMS to hand? There’s a local backend, a plain key on disk, to wire the whole thing together on your laptop first. These are commands for you, the person shipping the tool. Your users never run mytool keys mint… they just get updates that quietly check themselves, which was the whole idea two posts ago.

That setup deserves a walkthrough of its own, and it’ll get one. For now, the ergonomics were the point, not a nicety bolted on afterwards. The safest setup in the world is no use to anyone if it takes a PhD to stand up.

Where this leaves the whole story

Step back and the full loop is finally closed. The private key is born in KMS and never leaves it. Its public key is minted from it, with KMS computing its own self-signature. That public key is published off-platform and embedded in the binary. Releases are signed by KMS, reached only through short-lived OIDC federation. And the client verifies against the embedded and WKD keys cross-checked against each other. At no single point in that chain is there a thing an attacker can grab that lets them forge a release, and the most dangerous thing of all, the private key, has no theft path because it has no export path.

That’s the thread running through the whole signing series, from the very first checksum to here: the strongest control isn’t a better lock on the key. It’s arranging things so the key was never somewhere you could lose it. Nobody is coming to clean your supply chain, so the least you can do is leave it nothing worth stealing.

Generate a command from a script or a sentence with go-tool-base

Thu, 11 Jun 2026 00:00:00 +0000

You’ve got a Python script that already does the job. It’s sat in a tools/ directory somewhere, it works, and every few weeks someone copies it onto a laptop that doesn’t have the right version of pandas and it falls over. You’d like it to be a proper subcommand of your tool, a real Go binary you can ship, but porting it means the cobra wiring, the options struct, a test file, and a fight with the linter before any of it lands.

Or you don’t even have the script. You’ve just got a sentence in your head: “something that pings a list of URLs and tells me which ones are slow.” The logic is five minutes of thought; the boilerplate around it is the afternoon.

gtb generate command is built for exactly that gap. Hand it a script or hand it a sentence, and it writes the Go, the tests and the docs, then sends an autonomous agent through the result to make sure the thing actually builds, passes its tests and survives golangci-lint before it ever reaches your working tree.

Two ways in, the same files out

There are two flags, and they’re mutually exclusive:

--script <file> converts an existing bash, Python or JavaScript script.
--prompt "<text>" (or a path to a file) generates from a plain-English description.

Both land in the same place. A generated command called csv-stats gives you:

pkg/cmd/csv-stats/cmd.go: the cobra registration. This one is read-only; the generator owns it and will regenerate it.
pkg/cmd/csv-stats/main.go: the implementation, where your logic lives and where you’re free to edit.
pkg/cmd/csv-stats/main_test.go: a test file.
docs/commands/csv-stats/index.md: AI-written docs for the command.

The provider and model come from your config (ai.provider) or the --provider / --model flags. Everything below was generated with Claude Opus. We’ll take each in turn.

From a script: `csv_stats.py` becomes `csv-stats`

Here’s the script I want as a native subcommand. It reads a CSV and reports, per column, the row count, how many values are empty, and min/max/mean for the numeric ones. Nothing exotic, but enough that porting it by hand is a chore. Copy it into a file called csv_stats.py if you want to follow along:

#!/usr/bin/env python3
"""Summarise a CSV file's columns.

For every column it reports the row count and how many values are empty; for
columns whose values are numeric it also reports min, max and mean. A single
column can be selected with --column.

usage: csv_stats.py [--column NAME] <file.csv>
"""
import argparse
import csv
import sys


def is_number(value):
 """True if value parses as a float."""
 try:
 float(value)
 return True
 except (TypeError, ValueError):
 return False


def summarise(path, only_column=None):
 with open(path, newline="") as handle:
 reader = csv.DictReader(handle)
 if reader.fieldnames is None:
 print("error: empty CSV", file=sys.stderr)
 return 1

 columns = list(reader.fieldnames)
 if only_column is not None:
 if only_column not in columns:
 print(f"error: no such column: {only_column}", file=sys.stderr)
 return 1
 columns = [only_column]

 counts = {c: 0 for c in columns}
 nulls = {c: 0 for c in columns}
 numbers = {c: [] for c in columns}

 for row in reader:
 for c in columns:
 value = row.get(c, "")
 counts[c] += 1
 if value is None or value.strip() == "":
 nulls[c] += 1
 elif is_number(value):
 numbers[c].append(float(value))

 header = f"{'column':<20}{'count':>8}{'nulls':>8}{'min':>12}{'max':>12}{'mean':>12}"
 print(header)
 print("-" * len(header))
 for c in columns:
 nums = numbers[c]
 if nums:
 cmin = f"{min(nums):.2f}"
 cmax = f"{max(nums):.2f}"
 cmean = f"{sum(nums) / len(nums):.2f}"
 else:
 cmin = cmax = cmean = "-"
 print(f"{c:<20}{counts[c]:>8}{nulls[c]:>8}{cmin:>12}{cmax:>12}{cmean:>12}")
 return 0


def main():
 parser = argparse.ArgumentParser(description="Summarise a CSV file's columns.")
 parser.add_argument("csvfile", help="path to the CSV file")
 parser.add_argument("--column", help="only summarise this column")
 args = parser.parse_args()
 return summarise(args.csvfile, args.column)


if __name__ == "__main__":
 sys.exit(main())

One command points the generator at it:

gtb generate command \
 --name csv-stats \
 --short "Summarise CSV columns" \
 --script ./csv_stats.py

Your browser doesn't support embedded video; the demo converts csv_stats.py into a Go command and the repair agent builds, tests and lints the result.

What lands is not a transliteration. The Python kept everything in one function; the Go that came out is decomposed into named pieces, opens the file through the project’s injected filesystem (props.FS, an afero Fs) rather than os, and reports through the structured logger rather than print:

func summarise(fs afero.Fs, path, onlyColumn string) ([]string, error) {
	handle, err := fs.Open(path)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to open CSV file %q", path)
	}
	defer func() {
		_ = handle.Close()
	}()

	reader := csv.NewReader(handle)
	reader.FieldsPerRecord = -1

	columns, indexByName, err := readColumns(reader, onlyColumn)
	if err != nil {
		return nil, err
	}

	stats := make(map[string]*columnStats, len(columns))
	for _, c := range columns {
		stats[c] = &columnStats{numbers: []float64{}}
	}

	for {
		record, readErr := reader.Read()
		if readErr != nil {
			if errors.Is(readErr, io.EOF) {
				break
			}

			return nil, errors.Wrap(readErr, "failed to read CSV record")
		}

		accumulate(stats, columns, indexByName, record)
	}

	return formatReport(columns, stats), nil
}

That decomposition, into readColumns, accumulate, formatReport, summaryValues and a couple of small formatting helpers, is the interesting part, and it didn’t come for free. The first thing the agent did after writing the code was build it, test it, and lint it. golangci-lint’s cyclop rule flagged a single fat summarise function well over its complexity ceiling of 10. So the agent read the file back, split the work into focused functions, and ran the checks again. It only stopped once the build, the tests and the linter were all clean. The tidy shape above is the agent arguing with the linter and winning, not the model’s first guess.

Then it just runs. In the demo I scaffolded the project without the init feature, so the tool reads sensible defaults and needs no config step, and csv-stats sample.csv prints real per-column counts, nulls and numeric stats (with the default features you’d run toolbox init, or pass --config, first). The full generated command, the three files and nothing else, is here: csv-stats-command.tar.gz.

From a sentence: a URL health-checker

No script this time. Just a description of the command I wish I had. --prompt takes a raw string, but a description with any detail to it is easier to read, and to keep, in a file, so I dropped it in healthcheck-prompt.txt:

Concurrently GET a list of URLs and report each one’s HTTP status and latency.

Flags:

--timeout: the per-request timeout

--file: read URLs from a file, one per line

--json: machine-readable output

Use httptest in the tests so they need no network.

The prompt describes what I want the command to do, including how the flags should behave. The flags themselves I declare up front with --flag (more on why that split matters below), and point the generator at the file:

gtb generate command \
 --name healthcheck \
 --short "Check URL health concurrently" \
 --flag "timeout:duration:per-request timeout:false:t:false:5s" \
 --flag "file:string:read URLs from a file, one per line" \
 --flag "json:bool:machine-readable output" \
 --prompt ./healthcheck-prompt.txt

Your browser doesn't support embedded video; the demo builds a concurrent URL health-checker from a natural-language description, then self-repairs until it builds clean.

And the flags feed straight in. RunHealthcheck reads the URL file from opts.File, the deadline from opts.Timeout, and the output format from opts.Json, then fans the requests out across goroutines, each writing into its own slot, exactly the way you’d write it by hand:

func RunHealthcheck(ctx context.Context, props *props.Props, opts *HealthcheckOptions, args []string) error {
	urls, err := collectURLs(props.FS, opts.File, args)
	if err != nil {
		return errors.Wrap(err, "failed to collect URLs")
	}

	if len(urls) == 0 {
		return errors.New("no URLs provided; pass URLs as arguments or via --file")
	}

	timeout := opts.Timeout
	if timeout <= 0 {
		timeout = defaultTimeout
	}

	client := &http.Client{Timeout: timeout}

	results := make([]result, len(urls))

	var wg sync.WaitGroup

	for i, u := range urls {
		wg.Add(1)
		go func(idx int, target string) {
			defer wg.Done()

			results[idx] = checkURL(ctx, client, target, timeout)
		}(i, u)
	}

	wg.Wait()

	return reportResults(props, opts.Json, results)
}

I asked for the tests to use httptest so they’d need no network, and they do. Each case spins up a local server, so go test is hermetic and the agent’s own test run during repair stays self-contained, and it wrote cases for the flags too, this one driving --json:

func TestRunHealthcheck_JSONOutput(t *testing.T) {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusNotFound)
	}))
	defer srv.Close()

	p := newTestProps()
	opts := &healthcheck.HealthcheckOptions{
		Timeout: 5 * time.Second,
		Json: true,
	}

	err := healthcheck.RunHealthcheck(context.Background(), p, opts, []string{srv.URL})
	if err != nil {
		t.Fatalf("unexpected error: %v", err)
	}
}

Same as before, it builds and runs: point it at a few URLs and it GETs them concurrently, reporting each status and latency. The full generated command is here: healthcheck-command.tar.gz.

What “self-repair” actually means

The agent isn’t a single shot at the model with a hopeful prompt. It’s a loop with real tools: it reads the project layout, reads the files it needs, and runs go build, go test and golangci-lint. When something fails, it reads the relevant code, rewrites it, and runs the checks again. It only declares success once all three pass with nothing outstanding. The repair agent’s instructions are deliberately blunt on that last point: a clean build and passing tests don’t count as done if the linter still has something to say.

A few flags shape how it runs:

--max-steps N raises the agent’s reasoning budget. The default is plenty for a command like these two, but a genuinely hairy conversion can run long, and this stops it stopping short.
--agentless skips the agent entirely and uses the older retry loop, if you’d rather keep the generation cheap and do the polishing yourself.
--non-interactive withholds the agent’s ability to ask you a question mid-run. It defaults on when the CI environment variable is set, so the thing never blocks a pipeline waiting for an answer that isn’t coming.

Flags you declare, logic it writes

The --timeout, --file and --json arrived as real flags on the command, but not because the prompt mentioned them. Flags are the generator’s job, not the prompt’s, and that split is deliberate. You declare each one with --flag (or the interactive wizard), as I did above, and the generator wires it onto the options struct and into the read-only cmd.go registration, which hands that struct straight to your Run function. The prompt is left to describe behaviour: what --timeout should bound, what --file should read, what --json should change.

So the agent, told exactly which option fields exist, wrote its logic against opts.Timeout, opts.File and opts.Json rather than inventing anything, and the finished command’s --help lists them with the 5s default and the -t shorthand I asked for. Leave the --flags off and it still works: the generator hands the agent an empty options struct, and it keeps those values as locals with sensible defaults, ready for a flag to be wired in later.

The one thing you don’t do is hand-edit cmd.go: it’s regenerated every time you add a flag or change the command, so reach for --flag, never the file. When a generation finishes, the quickest sanity check is the command’s own --help, which shows the flags it actually exposes.

One thing to keep in mind: the model isn’t deterministic. Run the same prompt twice and you’ll get two slightly different commands. If the first one isn’t quite right, regenerate, or nudge the prompt. Treat the output the way you’d treat a capable colleague’s first PR: read it, run it, and own what you merge.

And is it the best possible code, the best design? Probably not. That depends on the model you can afford to point at it, how much detail you put in the prompt, and a bit of luck on the day. What you can count on is a working starting point: something that builds, has tests, and uses proper Go idioms and the project’s own patterns, instead of a blank file and an afternoon of boilerplate. From there it’s yours to shape.

Where that leaves you

The generator does the boilerplate and has the argument with the linter so you don’t have to. What it can’t do is decide whether the command it built is the command you actually wanted. That part is still yours, which is rather the point. The full docs for both flags live in the AI conversion guide and the command generation reference, and they’re the place to go when you want the flags the prompt didn’t.

A signature the platform can't forge

Tue, 09 Jun 2026 00:00:00 +0000

A self-updating tool has a chicken-and-egg problem baked into it. The thing doing the updating is the thing being updated, so when it reaches out and pulls down a newer version of itself, it’s the one that has to decide whether to trust what just landed. No human in the loop, nobody to ask. I’ve been closing that gap in go-tool-base’s self-updater in two phases. The first gave it a checksum: download the new binary, hash it, compare it against the release’s checksums.txt. That catches the accidents, the truncated download, the flipped bit on a dodgy mirror. And I said at the time, plainly, that it does nothing about a determined attacker who owns the release platform… the checksums file sits right next to the binary, so whoever can swap one can swap both. I left that as an IOU. This second phase is me paying it.

The thing a checksum can’t do

A checksum is a promise that the bytes you got match the manifest. It says nothing about who wrote the manifest. So if GitLab, or my account, or a leaked CI token gets compromised, the attacker rewrites the binary and the checksums.txt in the same breath, and the hash matches perfectly, because they’re the one who computed it. It’s the same wall I keep walking into whenever I think about supply-chain trust: a checksum is only ever as good as whatever’s standing behind it, and the thing standing behind a checksum is the very platform that just handed you the file. Same hands, both times.

To get past that, you need a signature whose root of trust lives somewhere the platform can’t reach.

The crypto is the easy part

Here’s the bit that caught me slightly off guard while I was building this: the cryptography is the easy part. Verifying a detached OpenPGP signature is a library call, and go-tool-base’s TrustSet wraps it up in one method:

func (t *TrustSet) VerifyManifestSignature(manifest, signature []byte) error {
	// ...
	signer, err := openpgp.CheckArmoredDetachedSignature(
		t.entities, bytes.NewReader(manifest), bytes.NewReader(signature), nil)
	if err != nil {
		return errors.Wrap(ErrSignatureInvalid, err.Error())
	}
	if signer == nil {
		return errors.Wrap(ErrSignatureInvalid, "no signer in trust set matched")
	}
	return nil
}

Hand it the manifest, the detached signature, and a set of trusted public keys (the entities), and it tells you whether any one of them signed it. That’s the whole of the cryptography, and it’s genuinely not where the hard work lives.

The hard work is that set of trusted public keys. Where do they come from? Because if the answer is “we ship them right next to the binary”, well… you’re straight back to the checksum problem. Whoever can swap the binary can swap the key too, sign with their own, and the check waves it through none the wiser.

Pulling the two questions apart

So the design splits along exactly that seam. The verification half is fixed, and deliberately boring (the method above). The trust anchor, the actual keys, comes from a swappable KeyResolver:

// The interface separates "where the trust anchor comes from" from "how a
// signature is verified against it", so SelfUpdater can be wired with
// whichever resolver chain a tool needs without changing verification logic.
type KeyResolver interface {
	Name() string
	Resolve(ctx context.Context) (*TrustSet, error)
}

That little seam is really the whole game. Everything interesting about standing up to a compromised platform comes down to which resolver you hand the updater, and the verification code never has to know the difference.

Three answers to “where does the key live”

The first option is to embed it. Bake the public key straight into the binary at build time (NewEmbeddedResolver), so it rides along inside a release you already trusted enough to run. Tidy and self-contained. The catch is that a future malicious release could embed a different key, so on its own, embedding really just trusts whoever cut the most recent binary.

The second is WKD, the Web Key Directory. Fetch the key over HTTPS from a well-known path on a domain you control (NewWKDResolver), nothing to do with where the release itself is hosted. Now the key isn’t in the binary at all, so poisoning a release doesn’t touch it. You haven’t made the problem disappear, mind… you’ve moved the trust onto your domain’s host and its DNS. A different blast radius, but a blast radius all the same.

The third option is to do both, and make them agree. Run embedded and WKD, and insist they agree:

func (c *CompositeResolver) Resolve(ctx context.Context) (*TrustSet, error) {
	// ... run each child resolver concurrently ...
	if err := checkAgreement(successes); err != nil {
		return nil, err // ErrKeyResolverMismatch
	}
	return successes[0].ts, nil
}

Think of it as the two-key rule on a safe deposit box, or two witnesses who’ve never met telling you the same story. One source on its own you might quietly doubt. But if the key baked into the binary and the key sitting on my domain hand back the same fingerprint, that agreement is worth a great deal more than either of them alone. And if they ever come back different, that’s not a maybe, that’s an alarm: ErrKeyResolverMismatch. Poison one source and the mismatch is the thing that gives the game away.

That composite is the real answer, and it’s why the interface exists at all. There’s nothing a single attacker can get their hands on that holds the whole thing up by itself. The key is baked into a release you trusted, and fetched from a domain well off the release platform, and the two have to match before a single byte of the update is allowed through.

The separation is the whole point

It’s easy to nod along at “two sources” and miss the part that actually does the work. The agreement between the embedded key and the WKD key is only worth something if an attacker can’t reach both of them from the same place. If the key I bake into the binary and the key I serve over WKD both came out of the same release pipeline, whoever owns that pipeline swaps the pair of them, the fingerprints still match, and the cross-check happily waves the forgery through. Same hands, both times. Again.

So they don’t share a pipeline, and that’s the entire design, not an accident of how things ended up. The binary, and the key embedded in it, are built and signed in GitLab CI, which federates into AWS KMS to do the signing itself. The WKD key lives somewhere else completely: a Cloudflare Pages site serving openpgpkey.phpboyscout.uk, deployed by hand at rotation time with the Wrangler CLI and a token allowed to do nothing but edit that one Pages project. No Git integration, no webhook, nothing that lets a push to the repo or a run of the release pipeline so much as touch it. The Cloudflare account is even administered under a different email and a different second factor from the GitLab and AWS ones, so the three anchors really are independent rather than just feeling that way.

Which is what makes them fail independently, and that independence is the only thing that makes the agreement worth checking. To forge a release that survives the cross-check, an attacker doesn’t have to beat one system, they have to beat two unrelated ones, on different platforms, behind different credentials, in the same window, without either of them noticing.

There’s a quieter benefit in the cadence, too. Releases go out constantly and automatically; the WKD key changes rarely, and only ever by hand. So the busy, automated path, the one an attacker is most likely to prise open, is exactly the one with no power to rewrite the key everyone checks against.

Requiring it, without breaking everyone

Now, a check nobody ever switches on is just theatre. But switch it on before the keys are actually out there in people’s installs, and you’ve handed everyone a self-inflicted outage instead. So the default is deliberately timid. The framework ships DefaultRequireSignature = false: a tool built on go-tool-base doesn’t suddenly start rejecting its own updates the day its author bumps the framework version.

The tool author flips it to true in main(), but only after they’ve shipped a release that embeds the key, so every install out there already holds the trust anchor before the first release that insists on one. Ship the key, then turn the lock: the same leave-yourself-a-way-back discipline as any migration you’d like to still have a job after. And the end user still gets an override (update.require_signature, or an env var) for the day it all goes sideways and they need out.

What it actually buys

The first phase stopped accidents. This one stops the platform. And not because the cryptography is clever, OpenPGP checks the signature in a single call, but because the trust anchor is arranged so that nothing the attacker can actually reach holds the whole thing up on its own. A signature only ever proves the sender, never the contents. All of this is really about making “the sender” something a compromised release host can’t quietly fake its way into being.

Which leaves one last thread dangling. The verifying key gets fetched from somewhere, fine… but the signing key, the private half that actually produces these signatures, has to live somewhere the platform can’t reach either, or none of the rest holds up. That’s the capstone, and where this series ends: where that key lives, and why it never leaves the box it’s born in.

The consent you can't ask for

Sat, 06 Jun 2026 00:00:00 +0000

There’s a comfortable story going round about telemetry, and it goes like this. There are two kinds. There’s the creepy kind, the usage data a vendor harvests to work out who you are and what you do, and that kind needs your permission. And there’s the innocent kind, the operational data a service emits so the people running it can keep it up, and that kind is just plumbing, nobody’s business, no permission required. Two neat boxes, and only one of them has a lock on it.

I don’t think the boxes are that neat. And I think a fair few of the people drawing them that way know it.

Because there’s no clean line where operational data stops being personal. A web service’s logs carry IP addresses. Its traces carry the path you walked through the system, the ids of the things you touched, sometimes the very fields you sent. Point at almost any of it and a GDPR lawyer will cheerfully tell you it can be personal data, and that the law doesn’t much care whether you filed it under “analytics” or “observability”. The word you picked to describe the data was never the thing that decided whether it was personal. The data decided that, and a lot of operational data is personal.

So if you can’t hide behind the box marked “just plumbing”, what do you actually do?

Where I’m coming from

I should say up front that I haven’t always been this relaxed about it. I spent a good few years in righteous fury at every tool that phoned home, every “we collect anonymous telemetry to improve the product” I never agreed to. Then I started building the tools, and I needed the data myself: the kind that tells you which features people actually use and which command falls over on first run, the kind that lets you make the next decision with something better than a hunch. And it softened me. Not into thinking it’s fine to take it without asking. Into understanding why everyone wants to.

What the fury left me with, the one thing I’ve never talked myself out of, is being pro-choice. Not pro-collection, not anti. Pro-choice. Any tool I ask another person to run will never quietly opt them into sending me a thing. It asks. On first run it makes its case, says what it wants and why, and lets them say no and mean it. I’ll try hard to win the yes, because the data is genuinely useful and a tool gets better when people share it. But I won’t presume it. The choice is theirs, and the prompt exists so they actually get to make it.

The trouble with a service

Which is a lovely principle right up until you build a web service. Because who, exactly, do you prompt? An API doesn’t have a first run. It has a thousand callers a second, none of them sat at a terminal waiting to tick a box. You can’t show a consent dialog to a webhook. The answer the industry reaches for is “consent is implied by use”, and… maybe. It’s a grey area, full stop. Implied consent is the same hand-wave that gave us the cookie banner, the thing we all click through without reading. I’m not going to stand here and call it clean.

But there’s a version of the principle that survives the grey, and it’s the one I built the framework around. Consent belongs to whoever can actually give it. For a command-line tool, that’s the person running it, so you ask them. For a web service, the person who can give it was never the end user at all, because you can’t reach them. It’s the engineer who deploys the thing. They know what their service collects, who its users are, which law they sit under, whether they owe anyone a privacy notice. They are the one party in the whole chain who can make the call with any of the facts in front of them. So that’s where the choice goes.

Which is why, in go-tool-base, the web-service telemetry is a switch. On or off, the engineer’s hand on it, collecting only what you need to keep the lights on by default. There’s no consent prompt, not because consent stopped mattering, but because there’s nobody in the loop I could ask. The accountability sits with the person who can hold it.

The part I’ll own

I’m pro-choice on telemetry, which is exactly why I built a way to switch it off and a way to force it on. Because for a web service the person holding the choice was never the end user, it’s the engineer who ships it, and “pro-choice” has to mean putting the switch in their hand, not pretending a popup would have meant anything.

That force-it-on part is the bit I’ll answer for. I built a way for a tool author to bypass the first-run prompt entirely and bake the consent in. There’s a real use case behind it, the enterprise tool deployed under a policy where collection is contractual rather than optional. But I also know I’ve handed someone a way to take the choice away, and I did it deliberately. Rightly or wrongly, I made the framework flexible enough to do the wrong thing, and the line I care about is now only as safe as the judgement of whoever picks it up.

That’s the uncomfortable place this lands, and I’ve come to think it’s the true one. A framework can put the choice in the right hands. It cannot make the right choice. I can build the prompt, build the switch, set the defaults to the modest thing, and after that I have to trust the engineer on the other side to use it justly and with some wisdom, because there is nothing further down the stack that makes them. When the blame gets shared out, and it’s always shared, a piece of it has my name on it, for every escape hatch I left in.

I’m at peace with that, mostly. Not because the grey went away, but because the alternative, pretending there’s a clean line and that “operational” means “not your problem”, is the real dodge. I’d rather say it plainly: this data can be personal, the consent is real even when there’s nobody to ask, and the most a tool can do is hand the decision to the person who can make it, and trust them with it.

Telemetry that asks, and telemetry that doesn't

Thu, 04 Jun 2026 00:00:00 +0000

go-tool-base has had a thing called telemetry for a long while now. It’s the opt-in kind: the product analytics that asks a user’s permission before it phones a single byte home, sits there as a no-op until they say yes, and can be wiped on request. The whole package is built around consent.

Then the web-service series went and needed telemetry too. Not that telemetry. The other one, the one the rest of the industry means when it says the word: traces, metrics and logs of a running service. And the awkward thing about those two is that they share a name, they want to share a package, and they pull in exactly opposite directions on the one question that matters most.

This is the story of how 0.7.x grew a second telemetry without breaking the first, and where the line between them ended up getting drawn.

Why bother putting it in the framework at all

The starting point is that I could have left observability out. A reader could wire up OpenTelemetry in their own service and go about their day. But the six parts of the web-service series spent a lot of effort making the transports first-class: a gRPC server, an HTTP server, a gateway, TLS across all of them, each one a Register call against the controller. Turning a CLI into a real long-running service and then shrugging “observability is your problem” would have left a hole exactly where it hurts.

Because a service you can’t see into is a liability the moment it leaves your laptop. The series ended with a macguffin service that was typed, fast and served over TLS, and was also a black box: when it got slow, you had nowhere to look. Metrics and traces are how you get the lights on, and they deserved the same first-class treatment as the things they observe.

The other half of the reason is that the framework already had a foot in this world. The analytics package’s preferred backend speaks OTLP, the OpenTelemetry wire protocol. So OpenTelemetry was already in the building. Doing observability any other way would have meant two standards where one would do.

The catch: two telemetries, opposite instincts

Here’s where it gets interesting, and it’s the part worth slowing down on.

The analytics telemetry is about a user. It collects usage data, hashed machine id, which command ran, exit code, and the entire design assumes you have to ask first. It is off by default. The collector you get when it’s disabled is a no-op, so nothing is recorded until the user opts in, and there’s a deletion path for when they change their mind. That’s not an add-on, that’s by design.

The observability telemetry is about a service. It emits operational data, how long a request took, which span was slow, how many errored, to a collector the operator runs. And there is no user in the loop to ask. The operator deploys the service, points it at their collector, and that act is itself the consent. Asking would be nonsensical: whose permission, for data about their own service, on their own infrastructure?

So you have two things called telemetry, wanting to live in one package, with the opposite default on consent. One is off until someone says yes; the other is on the moment it’s configured. Get that wiring wrong and you fail in one of two ugly ways. Gate the operational telemetry behind the user’s analytics opt-in, and a service’s tracing silently does nothing because nobody ticked a box meant for something else. Or loosen the analytics gate to make observability flow, and you start leaking usage data the user never agreed to share. Neither is acceptable, and “just use two packages” throws away everything the two genuinely have in common.

Quite a lot, as it turns out, and all of it below the consent line.

Both ship their data over OTLP to a collector. Both need to describe who is emitting, the service name and version, the resource in OpenTelemetry’s terms. Both parse an endpoint, attach headers, decide whether the connection is plaintext. None of that has the faintest thing to do with consent. It’s just the plumbing of getting bytes to a collector, and the analytics backend already had all of it, written inline.

So the shape of the solution fell out of the problem. Lift the shared plumbing into one place, let both telemetries stand on it, and keep the consent decision firmly out of that shared layer. The structure under pkg/telemetry ended up like this:

pkg/telemetry/
 telemetry.go the analytics Collector (consent-gated)
 backend_otel.go its OTLP backend
 posthog/ datadog/ vendor analytics backends
 otelcore/ shared: OTLP endpoint, resource, telemetry.* config
 tracing/ observability signal
 metrics/ observability signal
 logs/ observability signal
 observability.go Setup: builds the enabled signals (implied consent)

The new otelcore is the keystone. It holds the three things both sides need and nothing they don’t: ParseEndpoint for the OTLP URL, Resource for the service identity, and Resolve for reading the shared telemetry.* config (a base endpoint, plus per-signal overrides, in the same cascade as the TLS config). It imports no signal exporter and knows nothing about traces, metrics, logs or analytics. It is deliberately dumb plumbing.

The refactor: making the old telemetry stand on the new core

This next part is where the old telemetry and the new one become a single thing. The analytics OTLP backend was the first user of OTLP in the framework, and it had grown its own copy of all that plumbing: a function that parsed the endpoint URL, split out the host and path, worked out the insecure flag, and built the resource from a service name. Exactly the code the three new signals were about to need.

So rather than write it a second time and let the two drift, the analytics backend was refactored onto otelcore. Its exporter builder, buildOTelExporterOpts, now calls otelcore.ParseEndpoint, the same function tracing, metrics and logs call, and the resource comes from otelcore.Resource, the same one they use. One implementation of “talk OTLP to a collector”, four callers: the analytics backend and the three observability signals. Change how the framework forms an OTLP endpoint, and every signal moves together.

The reassuring part was that the analytics tests didn’t budge. The refactor moved code without changing behaviour, and the consent machinery, the opt-in, the no-op-when-disabled, the deletion path, never came near otelcore. Which is exactly the point.

Where the line is

Because the shared core is the easy half. The half that earns its keep is the bit that isn’t shared, and it’s a single, deliberate line.

The analytics collector keeps its gate. The constructor, NewCollector, still returns a no-op the moment telemetry is disabled, so a user who hasn’t opted in gets a collector that silently discards everything. Informed consent, untouched.

Observability gets a different door entirely. Setup builds whichever signals the operator has switched on, and it is gated only by telemetry.tracing.enabled and its siblings, which the operator sets. It never consults the analytics opt-in. Turning on tracing doesn’t turn on analytics; disabling analytics doesn’t silence tracing. The two enable flags live under the same telemetry.* config root, sit next to each other, and never read each other.

So that’s the whole architecture in a sentence: one package, one OTLP export core, two consent models that share everything except the answer to “do we need to ask”. The principle underneath, the one that decided every one of these calls, is that the kind of data sets the consent model. Usage data about a person needs informed consent. Operational data about a service runs on implied consent. The CLI and the web service are just where each kind tends to live.

Where this leaves the framework

0.7.x came out the other side with both telemetries: the one that asks first, exactly as it was, and a new one that doesn’t, because it has nobody to ask. They share an export core, a config root and a name, and they part company on the only thing they were ever going to disagree about.

I’ve been careful here to describe how the two consent models are kept apart, not to argue why they have to be. That argument, that “the kind of data decides the consent model” is a line worth holding rather than a convenient bit of engineering, is a piece of its own, and it’s the one I’m writing next.

Building a web service with go-tool-base, part 6: seeing what your service is doing

Tue, 02 Jun 2026 00:00:00 +0000

On paper the macguffin service is finished. Part 5 left it typed, fast, documented and served over TLS. So you deploy it, traffic starts flowing, and a week later someone wanders over to say “it’s slow”.

Slow how? Slow where? Which endpoint, the gateway or the gRPC behind it, the store or the network in between? You open the logs and there’s a wall of “request completed” lines, no way to tell which ones belong to the same request, and not a word about the half-second one of them spent somewhere inside. The service is a black box that happens to be on fire.

That’s the gap this part closes. A long-running service needs to be observable: you need to see what it’s doing, how often, how fast, and where the time actually goes. And we get there without bolting on a separate stack, because go-tool-base speaks OpenTelemetry, and three signals fall out of the same wiring.

This part is written against go-tool-base v0.7.1, the release that brings the observability components in.

The three things you actually want to see

“Observability” is a grand word for three fairly down-to-earth questions. Logs answer what happened (you’ve had those since part 1). Metrics answer how often, and how fast: request rates, error counts, the spread of latencies. And traces answer where did the time go, by following a single request as it crosses from one piece of your system to the next.

For a service shaped like ours, that last one is the prize. A REST call hits the gateway, turns into a gRPC call, runs through the store, and comes back. A trace stitches that whole journey into one timeline, so “it’s slow” stops being a shrug and becomes “it’s slow in CreateMacguffin, in the store, for 400ms”. You can’t get there from a pile of disconnected log lines.

OpenTelemetry (OTel, if you’ve seen it bandied about) is the vendor-neutral standard for all three. You instrument once and ship the data to whatever backend you fancy (Jaeger, Grafana, Honeycomb, a plain collector) over one wire protocol, OTLP. go-tool-base does the instrumenting; you point it at a collector and pick your poison.

Turn it on

One call wires the lot. In the serve command from part 4, right after the controller exists:

// pkg/cmd/serve/main.go

import "gitlab.com/phpboyscout/go-tool-base/pkg/telemetry"

controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

// Build whichever signals are enabled, install them, flush them on shutdown.
if _, err := telemetry.Setup(ctx, p, controller); err != nil {
	return err
}

telemetry.Setup reads your config, builds whichever of the three signals you’ve switched on, and installs them as the process-wide OTel providers. It also hangs their shutdown on the controller, so when a SIGTERM lands the buffered spans and metrics flush before the process exits. That’s the same graceful-shutdown promise from part 1, now extended to your telemetry: a clean stop drops nothing, not even the trace that was mid-flight.

On its own that line shows you nothing. A signal you haven’t enabled is skipped, so an unconfigured service pays exactly nothing. You turn each on in config, which we’ll get to. First, the instrumentation.

Wire it onto the transports

Setup builds the pipeline; two short additions feed it. The gRPC server takes a stats handler:

// pkg/cmd/serve/main.go

grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger,
	gtbgrpc.OTelStatsHandler())

and the HTTP server takes a middleware. This is also where we finally meet go-tool-base’s middleware chain, which we’ve managed without until now. A chain is just an ordered stack of wrappers around your handler. We put the OTel one first so it opens the span, and the request logger second, so it runs inside that span and can see it:

// pkg/cmd/serve/main.go

chain := gtbhttp.NewChain(
	gtbhttp.OTelMiddleware("macguffin"),
	gtbhttp.LoggingMiddleware(p.Logger),
)

if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, mux,
	gtbhttp.WithMiddleware(chain)); err != nil {
	return err
}

That’s the whole code change. Both helpers, OTelMiddleware and OTelStatsHandler, lean on the standard OTel instrumentation libraries underneath (otelhttp and otelgrpc), so the spans and metrics they emit follow the conventions every backend already knows how to read. Every request now opens a span and records its own duration, on both transports, and you wrote none of it per handler.

Go deeper than the plumbing

The transport spans show the request crossing HTTP and gRPC, but they stop at the edge of your handler. To see where the time goes inside it, you open a span yourself, and that needs no go-tool-base API at all, just OpenTelemetry. Here’s the store doing it, opening a span around the work and tagging it with what it’s handling:

// internal/macguffin/store.go

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
)

var tracer = otel.Tracer("macguffinsvc/store")

func (s *Store) Create(ctx context.Context, name string, quantity int32) Macguffin {
	_, span := tracer.Start(ctx, "Store.Create")
	defer span.End()

	span.SetAttributes(
		attribute.String("macguffin.name", name),
		attribute.Int("macguffin.quantity", int(quantity)),
	)

	// ... store the macguffin, generate its id ...

	span.SetAttributes(attribute.String("macguffin.id", id))

	return m
}

The span starts as a child of whatever is already in the context, which is the gRPC server span the handler is running under, so it slots into the trace on its own. The one thing to get right is the context: give the store method a ctx context.Context first parameter and thread it down from the handler, and the span finds its parent. Tag it with the things you’d want when something’s wrong, the macguffin’s name and its id, and they show up on the span in the UI, right next to the timing.

Point it at a collector

The wiring’s on; now tell it where to send. The config lives under one root:

# config.yaml
telemetry:
 endpoint: "http://localhost:4318"
 insecure: true # plaintext: a local collector only
 tracing:
 enabled: true
 sampling: 1.0 # every trace while you're looking
 metrics:
 enabled: true
 logs:
 enabled: true

endpoint is your OTLP collector. Sampling defaults to a level-headed 0.1 in production (you rarely want every trace under real load), so turn it up to 1.0 while you’re developing or you’ll wonder where yours went. Stand up a collector (a local Jaeger handles traces and gives you a UI to poke at), start the service, and push some traffic through the REST side:

$ curl https://localhost:8443/v1/macguffins
$ curl -X POST https://localhost:8443/v1/macguffins -d '{"name":"the-grail","quantity":1}'

One request, the whole way through

Open the collector and there it is, the thing that justifies the exercise: a single trace for that POST, four spans deep. The HTTP server span on top, the gRPC call nested inside it, the gRPC handler inside that, and your Store.Create span at the bottom, with the macguffin’s name and id sat right on it. The REST request became a gRPC call, crossed the gateway we built in part 4, ran the handler and went into the store, and OTel followed it the entire way, because the trace context travels with the request and everything reads from the same providers. “It’s slow” finally has somewhere to point, and “slow where” has an answer.

The metrics ride the same pipeline: request counts, error counts, a latency histogram broken down by route and method. And your logs get an upgrade you didn’t ask for. Every “request completed” line now carries the trace_id and span_id of the request it describes, so a slow trace in the UI and its log line in your terminal share an id, and you can jump straight from one to the other. That correlation lands in your stderr logs and in the records shipped to the collector, both.

A word on the other telemetry

If you read Telemetry that asks first, you’ll know go-tool-base has a telemetry package for product analytics too: the opt-in kind that asks a user’s permission before it phones anything home. This is not that, and the difference is the whole point.

That analytics is about a user, and it’s gated behind informed consent: off until they say yes. This is about your service, emitted by you, to a collector you run, and it works on implied consent. You switched it on; there’s no end user in the loop to ask. Same package, deliberately separate gates, and never the twain shall meet. There’s a fuller post in that one distinction, and it’s the next thing I’m writing.

Where that leaves the service

Six parts in, the macguffin service starts cleanly, reports its health, serves gRPC and REST and interactive docs over TLS, shuts down without dropping a request, and now tells you what it’s doing while it runs, all from one annotated proto and a controller it registers against. Observability was the last missing sense. The thing was fast and correct before, but you couldn’t see inside it. Now you can, and it cost you about a dozen lines.

Building a web service with go-tool-base, part 5: docs that write themselves

Sat, 30 May 2026 00:00:00 +0000

The google.api.http annotations we added in part 4 have done one job so far: they told the gateway which REST calls map to which RPCs. But they describe the API precisely, the paths, the verbs, the request and response shapes, and a precise description of an API is most of an OpenAPI document. Feed that document to a viewer and you have an interactive docs site: every endpoint listed, every field typed, and a “try it” button that calls the real service.

So from one .proto we’ve had gRPC, then REST, and now documentation, none of it a separate thing to write or keep in sync.

The annotations, a third time

The pattern of this whole series is one source of truth and several outputs generated from it. The proto defined the gRPC service; the annotations on it generated the REST gateway; and those same annotations generate an OpenAPI document. Add a field to a message, and it shows up in the gRPC API, the REST API, and the docs, all at once, because all three are read from the proto.

Generate the OpenAPI document

This is one more buf plugin. A small wrinkle to know up front: grpc-gateway ships its own OpenAPI generator, but it emits OpenAPI v2 (the old Swagger format). For a v3 document we use kollalabs/protoc-gen-openapi, which reads the very same google.api.http annotations:

go install github.com/kollalabs/protoc-gen-openapi@latest

# buf.gen.yaml
plugins:
 # ... the go, go-grpc and grpc-gateway plugins from before ...
 - local: protoc-gen-openapi
 out: internal/docs/assets
 opt:
 - title=Macguffin API
 - default_response=false

go generate ./... now also writes an openapi.yaml. It’s the REST API described in full, drawn straight from the annotations:

# internal/docs/assets/openapi.yaml (generated, trimmed)
openapi: 3.0.3
info:
 title: Macguffin API
 version: 0.0.1
paths:
 /v1/macguffins:
 get:
 summary: ListMacguffins
 operationId: MacguffinService_ListMacguffins
 parameters:
 - name: pageSize
 in: query
 schema:
 type: integer
 format: int32
 responses:
 "200":
 description: OK
 content:
 application/json:
 schema:
 $ref: '#/components/schemas/ListMacguffinsResponse'

That GET /v1/macguffins operation is the get: "/v1/macguffins" rule from the proto, turned into OpenAPI. You wrote the annotation once; it now feeds three generators.

Serve it, viewer and all

A spec on disk is a means, not an end; people want to read it. The usual move is to bolt on a docs UI like Stoplight Elements or Swagger UI, which means vendoring a couple of megabytes of JavaScript into every project. go-tool-base’s openapi package does that part for you: the Stoplight Elements UI is embedded in the framework, so your project ships only its generated spec.

openapi.Register mounts both the document and the docs site onto a mux:

// internal/docs/docs.go

package docs

import (
	_ "embed"
	"net/http"

	"gitlab.com/phpboyscout/go-tool-base/pkg/openapi"
)

//go:embed assets/openapi.yaml
var spec []byte

// Register mounts /openapi.yaml and the Stoplight docs site (/docs/) onto mux.
func Register(mux *http.ServeMux) error {
	return openapi.Register(mux, spec, openapi.WithTitle("Macguffin API"))
}

The //go:embed bakes the generated spec into the binary (the same trick as part 3.5), and openapi.Register serves it at /openapi.yaml and the Stoplight site at /docs/. Wiring it into serve is one line, on the same mux the gateway is mounted on:

// pkg/cmd/serve/main.go (the existing HTTP wiring)

mux := stdhttp.NewServeMux()
mux.Handle("/v1/", gw) // REST, from part 4

if err := docs.Register(mux); err != nil {
	return err
}

Why it goes on the same server

That last point is doing more work than it looks. The docs, the spec, and the live REST API are all on the one HTTP server, so they share an origin. That’s what makes the “try it” console actually work: when you fill in a request in the docs and hit send, the browser calls /v1/macguffins on the same host it loaded the page from, with no cross-origin dance to configure. And because the certificate is your mkcert one, the page and its requests are all clean HTTPS, no warnings to click past, which is exactly why we set the local CA up back in part 2.

$ curl https://localhost:8443/openapi.yaml | head -4
# Generated with protoc-gen-openapi
openapi: 3.0.3
info:
 title: Macguffin API

Open https://localhost:8443/docs/ in a browser and there’s the service: every endpoint, every field, and a working console that calls the real thing.

One proto, three faces

Step back and look at what a single annotated .proto is now producing: a gRPC service for the things that speak it, a REST API for the things that don’t, and an interactive docs site for the people who have to consume either. One source, three faces, and nothing hand-maintained between them. That’s the whole argument for building it this way, and it’s why the annotations were worth the small ceremony.

The service is, by any reasonable measure, done: typed, fast, documented, and served over TLS. The last part is about what happens once it’s out there and taking traffic. In part 6 we add telemetry and logging, so you can see how it’s being used and why it’s slow, without bolting on a separate observability stack.

Building a web service with go-tool-base, part 4: REST for free, with the gateway

Fri, 29 May 2026 00:00:00 +0000

A quick tally of where part 3 left us. One domain, the Store. One gRPC service over it, mapping the domain to proto with toProto. And then a whole second transport, the REST layer, with its own routing and its own toDTO mapping the very same domain into the very same shape, by hand. Two encodings of one thing, drifting apart the moment anyone adds a field and forgets the other side.

I promised that doubling would go away. This is the part where it does, and the thing that does it is the grpc-gateway.

What the gateway actually is

The grpc-gateway is a reverse proxy, generated from your .proto, that speaks REST on the front and gRPC on the back. A JSON request comes in, the gateway turns it into the matching gRPC call, hands it to your gRPC server, and turns the gRPC response back into JSON on the way out.

Read that again with part 3 in mind, because it’s the whole point. The gateway does the JSON-to-proto-and-back encoding for you, using the proto types your gRPC server already produces. You wrote domain → proto once, in the gRPC adapter. The gateway supplies proto → JSON. There is no second hand-written encoding to keep in step, because there is no second implementation: REST becomes a generated front door onto the gRPC service you already have.

So the plan is short. Tell the proto which HTTP calls map to which RPCs, regenerate, wire the gateway in, and delete the part-3 REST layer entirely.

Map HTTP onto the proto

gRPC has no opinion about URLs and verbs; REST is all URLs and verbs. The bridge is an annotation, google.api.http, that you attach to each RPC to say “this one is GET /v1/macguffins/{id}”. Here’s the service with those rules added:

// proto/macguffin/v1/macguffin.proto

import "google/api/annotations.proto";

service MacguffinService {
 rpc GetMacguffin(GetMacguffinRequest) returns (Macguffin) {
 option (google.api.http) = {get: "/v1/macguffins/{id}"};
 }
 rpc ListMacguffins(ListMacguffinsRequest) returns (ListMacguffinsResponse) {
 option (google.api.http) = {get: "/v1/macguffins"};
 }
 rpc CreateMacguffin(CreateMacguffinRequest) returns (Macguffin) {
 option (google.api.http) = {
 post: "/v1/macguffins"
 body: "*"
 };
 }
}

Each rule is small but exact. {id} in the path binds to the id field of the request message. body: "*" on the create says the whole JSON body maps onto the request. The list takes no body and no path parameter, just the verb and path. This is the same information part 3’s hand-written routes carried, except now it lives next to the RPC it describes, and a generator reads it instead of you.

These rules go a good deal further than the three cases we need: query-string parameters, several URL bindings for a single RPC, custom verbs, choosing which field becomes the response body. When you reach for those, the grpc-gateway docs walk through the mapping, and the canonical reference is the HttpRule message that google.api.http comes from, its comments document every option.

The annotations.proto import comes from Google’s common protos, so tell buf where to find them by adding a dependency, then fetch it:

# buf.yaml
deps:
 - buf.build/googleapis/googleapis

buf dep update

Generate the gateway

This is another buf plugin, exactly like part 2’s. Install it:

go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest

and add it to the generators:

# buf.gen.yaml
version: v2
plugins:
 - local: protoc-gen-go
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-go-grpc
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-grpc-gateway
 out: internal/gen
 opt: paths=source_relative

go generate ./... now also writes macguffin.pb.gw.go, the gateway: a RegisterMacguffinServiceHandler function that, given a connection to your gRPC server, mounts the REST routes the annotations described.

Wire it on

The gateway needs to call your gRPC server, which means dialling it like any other client, over the same TLS, with credentials that trust its certificate. That’s fiddly to get right by hand, so go-tool-base’s gateway package does it for you. gateway.New opens the connection (matching your server’s transport security) and hands you a mux to register the generated handlers on:

// pkg/cmd/serve/main.go

package serve

import (
	"context"
	stdhttp "net/http"

	"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
	"google.golang.org/grpc"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	"gitlab.com/phpboyscout/go-tool-base/pkg/gateway"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	gtbhttp "gitlab.com/phpboyscout/go-tool-base/pkg/http"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	store := macguffin.NewStore()

	// gRPC: the one implementation, mapping the domain to proto.
	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	// REST, for free: the gateway proxies JSON/HTTP to the gRPC server above.
	gw, err := gateway.New(ctx, p.Config,
		func(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error {
			return macguffinv1.RegisterMacguffinServiceHandler(ctx, mux, conn)
		})
	if err != nil {
		return err
	}

	mux := stdhttp.NewServeMux()
	mux.Handle("/v1/", gw)

	if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, mux); err != nil {
		return err
	}

	controller.Start()
	controller.Wait()

	return nil
}

The only macguffin-specific line is the one inside the callback, RegisterMacguffinServiceHandler. Everything around it, the dial, the credentials, the mux, is the framework’s. Mount the result under /v1/, register it on the same controller and HTTP server as before, and you’re done.

Delete the duplication

Here’s the satisfying bit. The hand-written REST adapter from part 3, the resthand package, the routes, the toDTO, all of it, comes out. You don’t need it: the gateway serves the same REST surface, backed by the gRPC service, from the proto. The serve command shrinks to one gRPC server and one gateway, and your codebase now has a single place where a macguffin becomes JSON.

See it work

The gateway answers REST, and it’s the same store the gRPC service uses:

$ curl https://localhost:8443/v1/macguffins
{"macguffins":[{"id":"m-1","name":"maltese-falcon","quantity":1}]}

$ curl -X POST https://localhost:8443/v1/macguffins -d '{"name":"the-grail","quantity":1}'
{"id":"m-2","name":"the-grail","quantity":1}

Create over REST, and the macguffin is there over gRPC a moment later, because both are the same implementation over the same Store:

$ grpcurl localhost:50051 macguffin.v1.MacguffinService/ListMacguffins
{ "macguffins": [ { "id": "m-1", ... }, { "id": "m-2", "name": "the-grail", ... } ] }

Errors, and changing them

Error handling comes across too. When a gRPC handler returns a status code, the gateway maps it to the matching HTTP status. The codes.NotFound we returned back in part 2 arrives as a 404, with a JSON error body, and we wrote none of it:

$ curl -s -o /dev/null -w '%{http_code}\n' https://localhost:8443/v1/macguffins/nope
404

That default mapping is the sensible one you’d reach for anyway. A few of the common codes:

gRPC code	HTTP
`InvalidArgument`	400
`Unauthenticated`	401
`PermissionDenied`	403
`NotFound`	404
`AlreadyExists`	409
`Internal`	500
`Unavailable`	503

So the rule of thumb is simply to return the right codes.* from your gRPC handlers, and the REST side gets the right status for free.

When the default shape isn’t what your clients expect, a {"error": {…}} envelope, a trace id header, a tweak to one particular status, you supply your own error handler. The grpc-gateway takes one as a runtime.ServeMuxOption, and gateway.New passes those straight through:

gw, err := gateway.New(ctx, p.Config, register,
	gateway.WithMuxOptions(runtime.WithErrorHandler(myErrorHandler)),
)

myErrorHandler receives the error and the http.ResponseWriter and writes whatever your API promises. That same WithMuxOptions hatch takes the gateway’s other knobs, header matchers, custom marshalers, and the rest; they’re all in the grpc-gateway docs.

Where this leaves us

This is the shape the series was building towards. One domain, one gRPC implementation, one domain → proto mapping you wrote by hand, and a REST API generated from the same proto that needed no second implementation and no second encoding. The things that speak gRPC get gRPC; the browser, the webhook and the curl get JSON; and there’s exactly one place to change when a macguffin grows a field.

Those google.api.http annotations have one more trick in them. They describe your REST API precisely enough to generate an OpenAPI document, and in part 5 we serve that as a live, clickable docs site, from the very same proto.

Building a web service with go-tool-base, part 3.5: the same server, now a website

Thu, 28 May 2026 00:00:00 +0000

The HTTP server from part 3 serves JSON. But net/http doesn’t care what you hand it: HTML, an image, a stylesheet, a whole little site, it’s all just bytes with a content type. So before we get back to the API in part 4, a short detour to prove the point and pick up a couple of genuinely useful tools: we’ll turn the macguffin service into a tiny website.

This is a bonus, off to the side of the API arc, but it earns its place. Real services nearly always grow a bit of HTML eventually: a status page, a landing page, a small admin view, an embedded docs site (we’ll do exactly that in part 5). The mechanics are the same every time, and worth having in hand.

A page from `html/template`

Go’s html/template renders HTML from a template and your data, and it escapes that data on the way out, so a macguffin called <script> becomes text rather than a problem. Here’s a page that lists the catalogue:

<!-- internal/site/templates/index.html -->
<!doctype html>
<html lang="en">
 <head>
 <meta charset="utf-8" />
 <title>Macguffins</title>
 <link rel="stylesheet" href="/static/style.css" />
 </head>
 <body>
 <h1>The macguffin catalogue</h1>
 <ul>
 {{range .}}
 <li>{{.Name}} <span class="qty">&times;{{.Quantity}}</span></li>
 {{end}}
 </ul>
 </body>
</html>

{{range .}} walks the slice we pass in, and {{.Name}} / {{.Quantity}} read each macguffin’s fields. The data is the same Store from part 2, so the page is a view onto the very same domain the gRPC and JSON APIs serve.

Shipping the files inside the binary

A template and a stylesheet are files, and you do not want to deploy a folder of loose assets next to your binary and hope they line up. Go’s embed package bakes them into the binary at build time, so the whole thing ships as one file.

// internal/site/site.go

package site

import (
	"embed"
	"html/template"
	"io/fs"
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

//go:embed templates static
var content embed.FS

var tmpl = template.Must(template.ParseFS(content, "templates/index.html"))

type Site struct {
	store *macguffin.Store
}

func New(store *macguffin.Store) *Site {
	return &Site{store: store}
}

func (s *Site) Routes() *http.ServeMux {
	static, err := fs.Sub(content, "static")
	if err != nil {
		panic(err) // the embedded path is a compile-time constant
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /{$}", s.index)
	mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.FS(static))))

	return mux
}

func (s *Site) index(w http.ResponseWriter, _ *http.Request) {
	if err := tmpl.Execute(w, s.store.List()); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Three things are doing the work. //go:embed templates static pulls both folders into the content filesystem. template.ParseFS parses the page from it once at startup. And http.FileServer(http.FS(static)) serves the stylesheet (and anything else under static/) straight from the embedded files, with content types set for you, so GET /static/style.css comes back as text/css.

The GET /{$} pattern is worth a note: the {$} anchors it to the exact root path, so / renders the page but /anything-else doesn’t accidentally match it.

If you’d rather edit templates without rebuilding during development, swap the embedded filesystem for the real one: http.FileServer(http.Dir("static")), and template.ParseGlob instead of ParseFS. Embed for release, disk for the edit-refresh loop; the handlers don’t change.

On the same server

Routes() hands back a *http.ServeMux, which is an http.Handler, so it registers exactly like the JSON API did, on the same controller, with the same TLS:

// pkg/cmd/serve/main.go (or a dedicated command)

if _, err := gtbhttp.Register(ctx, "site", controller, p.Config, p.Logger,
	site.New(store).Routes()); err != nil {
	return err
}

Because the certificate is the mkcert one from part 2, opening https://localhost:8443/ renders the page, stylesheet and all, with a clean padlock on any machine that trusts your local CA (where you ran mkcert -install); anywhere else, the browser flags the cert, exactly as it should.

The same hardened server, the same graceful shutdown, the same /healthz, now serving a website instead of (or alongside) JSON.

Back to the API

That’s the whole trick: the HTTP server is just net/http, and it will serve whatever you point it at, escaped and content-typed properly, shipped inside the binary. We’ll use exactly this in part 5 to serve interactive API docs.

Detour over. In part 4 we get back to the API and finally deal with that duplicated REST layer, the one we wrote twice and promised to delete.

Building a web service with go-tool-base, part 3: a REST service, by hand and by spec

Wed, 27 May 2026 00:00:00 +0000

The gRPC service from part 2 is the right core for service-to-service traffic. It is also useless to a great many of the things that might want to call it: a browser, a webhook from some SaaS, a partner who will not touch protobuf, a curl in a 2am runbook. They all want the same thing, plain JSON over HTTP.

So we need a REST face as well. The only way we know how so far is to build one, a second implementation of the macguffin service, this time over HTTP. We’ll do it two ways, by hand and from a spec, and wire it onto the very same controller from part 1. And yes, we’ll feel the duplication, because the next part is about making it disappear.

Why not just `net/http`?

A fair question before we register anything: go-tool-base’s HTTP server is net/http underneath, so why not stand up a http.Server{} yourself? You can, and nothing here hides the standard library from you. What gtbhttp.Register adds is the dull, easy-to-botch scaffolding that goes around the handler:

It’s still net/http. You write ordinary http.Handlers. There’s no bespoke router and no framework to learn underneath, just the standard library doing the serving.
Timeouts and limits you’d otherwise forget. A bare http.Server{} has no ReadTimeout or WriteTimeout, which is a Slowloris waiting to happen. You get sane read/write/idle timeouts and caps on header and body sizes.
The hardened TLS from part 2, the same shared certificate and the same 1.2-minimum AEAD config, with no extra setup.
Lifecycle for free. It registers on the same controller as your gRPC server, so a SIGTERM drains in-flight requests and stops both together, instead of you re-writing the signal-and-Shutdown dance for every service.
Health endpoints, /healthz, /livez and /readyz, backed by the controller and ready for an orchestrator to probe, with nothing to write.

You write the handlers; it owns the lifecycle and the hardening. With that settled, let’s write some handlers.

By hand, on the standard library

net/http’s ServeMux acquired method and path patterns in Go 1.22, so a small REST surface needs no router and no dependencies at all. And the hard part is already behind us: the Store from part 2 is our domain, and the HTTP handlers are just another thin adapter over it, exactly as the gRPC server was. A small type holds the store, and the routes hang off it:

// internal/resthand/server.go

package resthand

import (
	"encoding/json"
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

type API struct {
	store *macguffin.Store
}

func (a *API) Routes() *http.ServeMux {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /macguffins", a.list)
	mux.HandleFunc("GET /macguffins/{id}", a.get)
	mux.HandleFunc("POST /macguffins", a.create)

	return mux
}

Each pattern names a method and a path, and {id} is a named wildcard a handler reads back with r.PathValue("id"), no third-party router required. And because the domain Macguffin already carries JSON tags, this adapter can encode it straight to the response, with no separate type to map to:

// internal/resthand/server.go (same file)

func (a *API) get(w http.ResponseWriter, r *http.Request) {
	m, ok := a.store.Get(r.PathValue("id"))
	if !ok {
		http.Error(w, "macguffin not found", http.StatusNotFound)
		return
	}

	writeJSON(w, http.StatusOK, m)
}

list and create are the same shape: ask the store, encode the result with a small writeJSON helper. Routes() hands back a *http.ServeMux, which is itself an http.Handler, so it slots straight into the controller in a moment.

There’s nothing clever in any of it, and that’s the appeal: total control, no tooling, and because we serve the domain type directly, nothing to map. The cost is the kind that creeps up on you. Every route, every bit of marshalling, every status code is yours to write and keep correct, and as the surface grows, so does the area for small mistakes.

By spec, with oapi-codegen

The other road is the OpenAPI mirror of part 2’s proto. You describe the API in an OpenAPI document, and a generator turns it into a Go interface for you to implement. The tool is oapi-codegen.

First the contract, api/macguffin.openapi.yaml (trimmed to one path here):

# api/macguffin.openapi.yaml
paths:
 /macguffins/{id}:
 get:
 operationId: getMacguffin
 parameters:
 - name: id
 in: path
 required: true
 schema: { type: string }
 responses:
 "200":
 description: OK
 content:
 application/json:
 schema:
 $ref: "#/components/schemas/Macguffin"
 "404":
 description: Not found
components:
 schemas:
 Macguffin:
 type: object
 required: [id, name, quantity]
 properties:
 id: { type: string }
 name: { type: string }
 quantity: { type: integer, format: int32 }

Install the tool and tell it what to emit with a small config. We want the standard-library server, so std-http-server:

go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@latest

# api/oapi-codegen.yaml
package: restapi
output: internal/restapi/macguffin.gen.go
generate:
 std-http-server: true
 models: true

As in part 2, wire it into go generate so it can’t drift, this time sitting right alongside the buf directive:

// gen.go (at your module root): add the second line alongside part 2's

//go:generate buf generate
//go:generate oapi-codegen -config api/oapi-codegen.yaml api/macguffin.openapi.yaml

go generate ./... now regenerates both the gRPC and the REST code in one go. What oapi-codegen writes is the message models and, the important part, a ServerInterface, one method per operation:

// internal/restapi/macguffin.gen.go (generated)

type ServerInterface interface {
	ListMacguffins(w http.ResponseWriter, r *http.Request)
	CreateMacguffin(w http.ResponseWriter, r *http.Request)
	GetMacguffin(w http.ResponseWriter, r *http.Request, id string)
}

If that feels familiar, it should: it’s the exact shape from part 2. The generator hands you an interface and your job is a type that honours it, an adapter over the same Store. But here’s the wrinkle that earns this part its keep. oapi-codegen generated its own Macguffin type, a DTO, so the adapter has to map the domain into it. (The path parameter, at least, arrives already typed as id string rather than fished out by hand.)

// internal/restapi/server.go

package restapi

import (
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func (a *API) GetMacguffin(w http.ResponseWriter, _ *http.Request, id string) {
	m, ok := a.store.Get(id)
	if !ok {
		http.Error(w, "macguffin not found", http.StatusNotFound)
		return
	}

	writeJSON(w, http.StatusOK, toDTO(m))
}

// toDTO maps the domain type to oapi-codegen's generated Macguffin DTO.
func toDTO(m macguffin.Macguffin) Macguffin {
	return Macguffin{Id: m.ID, Name: m.Name, Quantity: m.Quantity}
}

That toDTO is the second mapping of this kind we’ve written. Part 2’s gRPC adapter had toProto; this one has toDTO. The same domain data, encoded twice, into two generated shapes, kept in step by hand. Hold that thought: part 4 is where that second mapping stops being something you write.

A generated helper turns your implementation into an http.Handler:

handler := restapi.HandlerFromMux(restapi.NewAPI(store), http.NewServeMux())

So which road? By hand when it’s a handful of endpoints you fully control and you’d rather not own a generator. Spec-first when the contract carries weight, several teams consume the API, the shape changes often, or you simply want the OpenAPI document to exist, which (spoiler) we’ll be serving as live, clickable docs in part 5.

Wire it onto the controller

Whichever road you took, you’re holding an http.Handler. We don’t replace the gRPC server from part 2; we add the HTTP one beside it, on the same controller, both reading from one shared store. This is the controller from part 1 doing exactly what it exists for: two transports, one lifecycle.

// pkg/cmd/serve/main.go

package serve

import (
	"context"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	gtbhttp "gitlab.com/phpboyscout/go-tool-base/pkg/http"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
	"gitlab.com/myorg/macguffinsvc/internal/resthand"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	// One shared domain behind both transports.
	store := macguffin.NewStore()

	// gRPC, from part 2.
	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	// HTTP, new this part. (Or the oapi-codegen handler; either is an http.Handler.)
	handler := resthand.New(store).Routes()
	if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, handler); err != nil {
		return err
	}

	controller.Start()
	controller.Wait()

	return nil
}

The HTTP Register is the counterpart of part 2’s gRPC one: same controller, so a single SIGTERM drains and stops both together and /healthz reports on both. The HTTP server reads server.http.port, the gRPC server server.grpc.port from part 2, and both take their TLS from the shared server.tls block, the one certificate:

server:
 grpc:
 port: 50051
 reflection: true
 http:
 port: 8443
 tls:
 enabled: true
 cert: ./localhost+2.pem
 key: ./localhost+2-key.pem

A browser away

Here’s where the mkcert groundwork from part 2 pays off. Because that certificate is signed by a CA your machine already trusts, the service answers over HTTPS with no --cacert flag and, more to the point, no browser warning:

$ curl https://localhost:8443/macguffins
[{"id":"m-1","name":"maltese-falcon","quantity":1}]

$ curl https://localhost:8443/macguffins/m-1
{"id":"m-1","name":"maltese-falcon","quantity":1}

And /healthz now reports both transports, since they registered against the one controller:

$ curl https://localhost:8443/healthz
{"overall_healthy":true,"services":[{"name":"grpc","status":"OK"},{"name":"http","status":"OK"}]}

Open https://localhost:8443/macguffins in an actual browser and it just loads, green padlock and all. That matters more than it sounds, and it’s the reason we set the local CA up early: in part 5 the API docs are a web page, and a docs page behind a cert warning is a docs page nobody trusts.

Where this leaves us

Step back and count what we actually built. There’s one domain, the Store, and it didn’t change at all this part. What we added was a second delivery adapter over it: routing, marshalling, and, on the spec-first road, a toDTO mapping sitting beside part 2’s toProto. The same data, encoded into two generated shapes, kept in step by hand. That’s the real duplication, not the logic, but the transport scaffolding wrapped around it.

And it’s the itch the next part scratches. We built a second transport and a second encoding of the same domain, and kept them in step by hand. Part 4 is where that whole second layer, adapter and encoding both, stops being something you write at all. We felt the cost first on purpose; now we get to remove it.

Building a web service with go-tool-base, part 2: a gRPC service, with TLS

Tue, 26 May 2026 00:00:00 +0000

The heartbeat from part 1 runs, ticks along, and shuts down politely when you ask it to. It also talks to absolutely no one. A service people can actually call needs an API, and for a typed, fast, streaming-capable one, gRPC is the obvious first move.

The catch is that a production-grade gRPC server is rather more than grpc.NewServer(). You want health checks an orchestrator understands, reflection so you can poke at it without the .proto file in hand, a graceful shutdown that doesn’t guillotine calls that are still in flight, and TLS, which is where most people’s first attempt quietly goes wrong. The good news: part 1 already gave us the thing that carries all of it. A gRPC server is just another service to register on the controller.

Why gRPC, and not just REST

Worth a moment on why we’re reaching for gRPC at all, because for plenty of services a plain JSON-over-HTTP API is the right call and less faff. gRPC earns its place when a few of these matter:

A contract that’s enforced, not hoped for. The .proto is the single source of truth, and both ends are generated from it. You don’t hand-write JSON marshalling, and you don’t find out at runtime that the client and server disagree about a field’s type. Evolve the schema carefully (add fields by number) and old clients keep working.
Clients in any language, for free. The same .proto generates a Go server and a Python, TypeScript, Rust or Java client with nobody writing an SDK by hand. For an internal service that several teams call, that one point can decide it.
It’s built for service-to-service traffic. Binary protobuf is smaller and quicker to encode than JSON, calls multiplex down a single HTTP/2 connection, and streaming (from the client, the server, or both at once) is a first-class thing rather than something you bolt onto REST with websockets.
Deadlines, cancellation and a health protocol come built in, rather than conventions you reinvent for every service.

The trade-offs are real. A browser doesn’t speak gRPC natively, and a binary protocol is fiddlier to poke at than a JSON endpoint you can curl (which is exactly why reflection and grpcurl exist). That’s not a reason to avoid it; it’s the reason this series doesn’t stop at gRPC. In part 4 we put a REST/JSON face on this very service, so the things that call it get the typed, fast core and the things that can’t speak gRPC still get a friendly HTTP surface. You don’t have to pick a side.

Define the contract

gRPC starts with a schema. Here’s a small macguffin service, macguffin.proto:

// proto/macguffin/v1/macguffin.proto

syntax = "proto3";

package macguffin.v1;

option go_package = "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1;macguffinv1";

service MacguffinService {
 rpc GetMacguffin(GetMacguffinRequest) returns (Macguffin);
 rpc ListMacguffins(ListMacguffinsRequest) returns (ListMacguffinsResponse);
 rpc CreateMacguffin(CreateMacguffinRequest) returns (Macguffin);
}

message Macguffin {
 string id = 1;
 string name = 2;
 int32 quantity = 3;
}

message GetMacguffinRequest { string id = 1; }
message ListMacguffinsRequest { int32 page_size = 1; }
message ListMacguffinsResponse { repeated Macguffin macguffins = 1; }
message CreateMacguffinRequest { string name = 1; int32 quantity = 2; }

From proto to Go

If gRPC in Go is new to you, this is the part that catches people out: you don’t write the Go for those messages and that service interface, you generate it from the .proto. The proto is the source of truth; a compiler turns it into Go you import and build against. Same goes for a client in any other language, all from the same file.

That compiler is protoc, and on its own it’s a faff. You install it, then a separate plugin for each output you want (protoc-gen-go for the message types, protoc-gen-go-grpc for the client and server stubs), keep their versions in step, and drive the lot with a command line of -I include paths and --*_out flags that’s easy to get subtly wrong.

buf is the friendlier way to run exactly that. It wraps protoc and its plugins behind a couple of small config files, handles the plugin versions, and turns that gnarly invocation into a single word. It’s become the usual way to work with protobuf in Go, and it’s what we’ll use here.

At a minimum you need three binaries on your PATH: buf itself, and the two plugins it drives. go install is the quickest way to get them:

go install github.com/bufbuild/buf/cmd/buf@latest
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Those land in $(go env GOPATH)/bin, so make sure that’s on your $PATH. Then describe what you want generated in a buf.gen.yaml:

# buf.gen.yaml
version: v2
plugins:
 - local: protoc-gen-go
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-go-grpc
 out: internal/gen
 opt: paths=source_relative

Each part of that earns its place. version: v2 is buf’s config format. The plugins list names the generators to run, and we run two, because gRPC in Go arrives in two halves:

protoc-gen-go turns the messages into Go structs, the Macguffin type and the request and response types, in a macguffin.pb.go file.
protoc-gen-go-grpc turns the service into the client and server scaffolding, in a macguffin_grpc.pb.go file, including the MacguffinServiceServer interface you’re about to implement.

out: internal/gen is where the files land, and paths=source_relative lays them out mirroring the proto’s own folders (so proto/macguffin/v1/... becomes internal/gen/macguffin/v1/...) rather than deriving the path from the go_package line. Then run it:

buf generate

Both files appear under internal/gen/macguffin/v1, and we’re ready to write the implementation.

Running that by hand once is fine; remembering to run it every time the .proto changes is where it goes wrong, and the generated code quietly drifts out of step, usually right before a demo. Wire it into go generate instead. Drop a one-line directive in a file at your module root, say gen.go:

// gen.go (at your module root)

package macguffinsvc

//go:generate buf generate

Now go generate ./... regenerates everything from the proto, and it’s the same one command for any other generator you add later. Run it whenever the schema changes, and in CI if you want to catch a stale checkout.

If OpenAPI is your map of the territory

If your mental model of an API contract is an OpenAPI (Swagger) document, a .proto is the same idea wearing fewer clothes: a typed, language-neutral description of a service that both ends generate from. The thing you notice first is how much less of it there is. Here’s that Macguffin message again, and the same shape written as an OpenAPI schema:

message Macguffin {
 string id = 1;
 string name = 2;
 int32 quantity = 3;
}

Macguffin:
 type: object
 properties:
 id:
 type: string
 name:
 type: string
 quantity:
 type: integer
 format: int32

And that pattern holds across the whole service. The proto above, three calls and five messages, is about twenty lines. Describe the same surface in OpenAPI and you’re closer to a hundred, because OpenAPI also pins down the HTTP verbs, paths, status codes and content types: the transport details a proto leaves out on purpose. That isn’t OpenAPI being bloated; it’s describing more. But when the contract is the thing you care about, the proto says it with less ceremony, and it doesn’t wed your API to HTTP, which is exactly what lets us serve this same service over gRPC now and REST later. (We’ll generate a real OpenAPI document from this proto in part 5, for the readers who still want one.)

Implement it

Generating the code gave you the message types and, more to the point, an interface to satisfy. Open macguffin_grpc.pb.go and you’ll find MacguffinServiceServer, one method per RPC in the proto:

// internal/gen/macguffin/v1/macguffin_grpc.pb.go (generated)

type MacguffinServiceServer interface {
	GetMacguffin(context.Context, *GetMacguffinRequest) (*Macguffin, error)
	ListMacguffins(context.Context, *ListMacguffinsRequest) (*ListMacguffinsResponse, error)
	CreateMacguffin(context.Context, *CreateMacguffinRequest) (*Macguffin, error)
	mustEmbedUnimplementedMacguffinServiceServer()
}

That interface is the server-side contract. Each method takes the request message you defined and hands back the response message, plus an error. Writing the type that honours it, the actual logic behind each call, is the part that’s yours: the proto pins down the shape of the conversation, and this is what the service actually does when one happens.

The one curious line is mustEmbedUnimplementedMacguffinServiceServer(). Alongside the interface, buf generated an UnimplementedMacguffinServiceServer struct with a do-nothing stub for every method, and you embed it in your own type. It earns its keep twice over. It satisfies that unexported method, so your type counts as a real implementation. And it future-proofs you: add a new RPC to the proto later and your existing server still compiles, falling back to the stub (which returns a clean “unimplemented” error) until you write the real method.

Before we satisfy that interface, one separation worth making up front. The gRPC server is a delivery mechanism, not the place the data lives. If we stuff the map of macguffins straight inside it and then build an HTTP server next part, we’d have two servers each hoarding their own copy of the same data. So keep the domain, the macguffins and what you can do with them, in its own type, and let each transport be a thin layer over it.

Here’s that domain: an in-memory store standing in for the repository a real service would have. Nothing in it knows about gRPC, HTTP, or any wire format.

// internal/macguffin/store.go

package macguffin

import "sync"

// Macguffin is the domain type. The JSON tags will let a hand-written HTTP
// handler serve it directly in part 3.
type Macguffin struct {
	ID string `json:"id"`
	Name string `json:"name"`
	Quantity int32 `json:"quantity"`
}

type Store struct {
	mu sync.Mutex
	items map[string]Macguffin
	seq int
}

func (s *Store) Get(id string) (Macguffin, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()

	m, ok := s.items[id]

	return m, ok
}

List and Create are the same shape, and NewStore seeds it with a single maltese-falcon. Now the gRPC server is thin: it embeds the stub, holds a store, and each method calls the store and translates the result into the generated protobuf type.

// internal/grpcsvc/server.go

package grpcsvc

import (
	"context"

	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

type Server struct {
	macguffinv1.UnimplementedMacguffinServiceServer

	store *macguffin.Store
}

func New(store *macguffin.Store) *Server {
	return &Server{store: store}
}

func (s *Server) GetMacguffin(_ context.Context, req *macguffinv1.GetMacguffinRequest) (*macguffinv1.Macguffin, error) {
	m, ok := s.store.Get(req.GetId())
	if !ok {
		return nil, status.Errorf(codes.NotFound, "macguffin %q not found", req.GetId())
	}

	return toProto(m), nil
}

// toProto maps the domain type to the generated protobuf DTO.
func toProto(m macguffin.Macguffin) *macguffinv1.Macguffin {
	return &macguffinv1.Macguffin{Id: m.ID, Name: m.Name, Quantity: m.Quantity}
}

ListMacguffins and CreateMacguffin are the same: call the store, map the result. The one habit worth keeping is to return real gRPC status codes (codes.NotFound here) rather than bare errors, so callers get something they can branch on.

That toProto step is worth a second look, because it comes back round later. The domain has one shape, the proto has its own generated Macguffin, so the adapter maps between them. It’s a small price for a single transport. In part 3 we add a second transport with its own generated type and pay that price again, and part 4 is where we stop paying it twice.

Wire it onto the controller

This is the part that earns its keep. First, generate a serve command, the same way the CLI series generated its commands:

gtb generate command \
 --name serve \
 --short "Run the macguffin service"

That scaffolds two files: pkg/cmd/serve/cmd.go (generated, and wired into your command tree for you) and pkg/cmd/serve/main.go, which holds a RunServe function for your logic. Fill it in:

// pkg/cmd/serve/main.go

package serve

import (
	"context"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	store := macguffin.NewStore()

	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	controller.Start()
	controller.Wait()

	return nil
}

That’s the whole server. gtbgrpc.Register does four things in one call: it builds a *grpc.Server, wires the standard gRPC health service to the controller’s health reports (the ones we met in part 1), registers Start, Stop and Status against the controller so the lifecycle is handled, and hands you back the server to register your own service on, which is the RegisterMacguffinServiceServer line. After that it’s the same Start() / Wait() we used for the heartbeat.

It reads its port from config (server.grpc.port, falling back to server.port), so a minimal config is:

server:
 grpc:
 port: 50051
 reflection: true

Poke it

Build, run mytool serve, and reach for grpcurl. Reflection is on, so you don’t need the .proto to hand:

$ grpcurl -plaintext localhost:50051 list
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection
macguffin.v1.MacguffinService

$ grpcurl -plaintext localhost:50051 macguffin.v1.MacguffinService/ListMacguffins
{
 "macguffins": [
 { "id": "m-1", "name": "maltese-falcon", "quantity": 1 }
 ]
}

And the health service is already answering, wired straight to the controller, without you registering a thing:

$ grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
{ "status": "SERVING" }

That’s the lifecycle work from part 1 paying out: the controller’s health is the gRPC health, and a SIGTERM still drains and stops the server cleanly.

Now turn on TLS

Here’s the bit people brace for. Plaintext gRPC is fine on a laptop and unacceptable the moment it leaves one. With go-tool-base it’s a config change, not a code change.

The fiddly part of local TLS is usually the certificate. A hand-rolled self-signed one means passing a -cacert to every client and clicking past browser warnings. mkcert makes that go away: it creates a local certificate authority and installs it into your system’s (and your browser’s) trust stores, so anything it signs is simply trusted. Set the CA up once:

mkcert -install

Then mint a certificate for the names the service answers on:

mkcert localhost 127.0.0.1 ::1

That writes localhost+2.pem (the certificate) and localhost+2-key.pem (the key), signed by your now-trusted local CA. Doing this properly now pays off later: in part 3 the HTTP server, and in part 5 the API docs in a browser, both lean on that certificate being trusted with no warning.

Point the tool’s config at the pair, under the shared server.tls block:

server:
 grpc:
 port: 50051
 reflection: true
 tls:
 enabled: true
 cert: ./localhost+2.pem
 key: ./localhost+2-key.pem

No code changes. Run mytool serve again and it comes up over TLS:

INFO starting gRPC server tls=true addr=:50051

Because the certificate is signed by a CA your machine already trusts, the client needs no extra flags:

$ grpcurl localhost:50051 macguffin.v1.MacguffinService/GetMacguffin -d '{"id":"m-1"}'
{ "id": "m-1", "name": "maltese-falcon", "quantity": 1 }

A plaintext client is now refused, as it should be. (In production you’d point those same two config keys at whatever your real CA issues; the wiring doesn’t change.) Two details are worth knowing about what just happened, because both are easy to get wrong by hand. The server uses a hardened TLS config (1.2 minimum, AEAD cipher suites, X25519), so you’re not accidentally shipping a weak handshake. And the listener advertises HTTP/2 over ALPN, the h2 protocol gRPC rides on, which sounds like a footnote until you discover that recent gRPC clients refuse a TLS connection that doesn’t offer it. The framework sets it for you; it’s the single most common reason a hand-rolled gRPC-over-TLS server works with old tooling and mysteriously rejects a current client. All of that lives in the shared pkg/tls package.

I put the certificate under server.tls rather than server.grpc.tls deliberately. That shared block is the cert every transport falls back to, so the HTTP server in the next part and the transports after it can use the same one, with a per-transport override only where you actually need it.

The short version

A few files in, you have a real gRPC API: a typed contract, an implementation, health an orchestrator understands, reflection for poking, a clean shutdown, and TLS, and the only part that was actually yours to write was the service logic. The reference for the server helpers is the gRPC component doc, and the add-a-gRPC-service how-to has the manual-wiring path if you ever want it.

Next part puts an HTTP face on the very same controller, REST handlers and the same health endpoints an orchestrator probes, sharing that one certificate.

Building a web service with go-tool-base, part 1: lifecycle and graceful shutdown

Mon, 25 May 2026 00:00:00 +0000

A CLI does its job and gets out of the way. You run it, it prints something or writes a file, the process exits, done. Then one day you want the opposite: a thing that stays running. A server answering requests, a worker chewing through a queue, something that sits there until you tell it to stop. And the moment a process is long-lived, a pile of fiddly problems lands on your desk that a short-lived command never had to think about.

How does it shut down when Kubernetes sends it a SIGTERM, without dropping the requests it’s halfway through? How does anything outside it know it’s alive, or ready for traffic? When one part falls over at 3am, does the whole thing come down, or pick itself back up? None of that is your actual service. It’s the plumbing around it, and it’s the sort of plumbing that’s easy to write almost right and only notice the gap in during an incident.

This is a new series, a companion to the one on building a CLI. That series gets you a working command-line tool; this one turns it into a web service, a piece at a time: gRPC, HTTP, a gateway that bridges the two, TLS across all of them, and live API docs. But every one of those is a long-running process, so we start with the part they all stand on.

Same shape as before, each part stands alone. By the end of this one you’ll have a process that starts cleanly, reports its own health, and shuts down without dropping anything, and you won’t have written the lifecycle code yourself. The series is written against go-tool-base v0.6.0, the release that brings the web-service components in.

What every long-running process needs

go-tool-base’s answer to all of the above is the controls package, and its centrepiece is the Controller. You hand it a set of services, things with a way to start, a way to stop, and a way to report health, and it owns their lifecycle. It starts them, watches for the operating system asking the process to quit, drives a graceful shutdown in the right order, and keeps a running picture of whether everything is alright.

A “service” here is deliberately loose. An HTTP server is one. A gRPC server is one. So is a background worker that wakes every few seconds, or a queue consumer. The controller doesn’t care what’s inside; it cares that it can start it, stop it, and ask after its health. That’s the whole trick: get those three verbs right once, in one place, and everything you bolt on later inherits them.

A service in thirty lines

Let’s build the smallest useful one: a heartbeat that logs a tick every second. It isn’t exciting, but it’s a real long-running service, and it shows every moving part without a transport getting in the way.

// main.go

package main

import (
	"context"
	"os"
	"sync/atomic"
	"time"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	"gitlab.com/phpboyscout/go-tool-base/pkg/logger"
)

func main() {
	log := logger.NewCharm(os.Stderr, logger.WithTimestamp(true))

	// The controller owns the process lifecycle: it starts registered services,
	// watches for SIGINT/SIGTERM, and drives a graceful shutdown.
	controller := controls.NewController(context.Background(), controls.WithLogger(log))

	var beats atomic.Int64

	controller.Register("heartbeat",
		// Start launches the work. The context is cancelled when the controller
		// shuts down, so the goroutine just watches ctx.Done().
		controls.WithStart(func(ctx context.Context) error {
			go func() {
				ticker := time.NewTicker(time.Second)
				defer ticker.Stop()

				for {
					select {
					case <-ctx.Done():
						return
					case <-ticker.C:
						log.Info("beat", "count", beats.Add(1))
					}
				}
			}()

			return nil
		}),
		// Stop runs during shutdown for any explicit cleanup.
		controls.WithStop(func(_ context.Context) {
			log.Info("heartbeat stopping", "total_beats", beats.Load())
		}),
		// Status reports health. Here we're healthy as long as we're ticking.
		controls.WithStatus(func() error { return nil }),
	)

	controller.Start()
	controller.Wait()
}

A few things are earning their keep there. NewController takes a context and some options, here just a logger. Register names a service and gives it its three verbs through functional options: WithStart launches it (and is handed a context that gets cancelled when the controller shuts down, which is the hook the goroutine watches), WithStop is called during shutdown for cleanup, and WithStatus answers “are you alright?”. Then controller.Start() launches everything and controller.Wait() blocks until the whole thing has stopped. The three options are all in pkg/controls.

Build it and run it:

go run .

INFO beat count=1
INFO beat count=2
INFO beat count=3

A service, running. Now for the half that’s easy to get wrong.

Shutting down on purpose

Press Ctrl-C, or send the process a SIGTERM the way an orchestrator would, and watch what it does:

WARN [Controller] : received signal signal=terminated
WARN [Controller] : Stopping Services
INFO heartbeat stopping total_beats=3
INFO [Controller] : Stopped

Nothing in our thirty lines handled a signal. The controller registered its own handlers for SIGINT and SIGTERM, and when one arrived it cancelled the context that every service’s Start is watching, gave them a window to finish, ran each Stop, and exited cleanly. That cancel-the-context-then-Stop order is the thing: it’s exactly what stops an HTTP server from dropping requests it’s mid-way through when the pod rolls. We’ve got it here for a heartbeat that does nothing on the way out, and we’ll get the same order for free for every real transport we add later.

You can tune the window with WithShutdownTimeout, and turn the signal handling off entirely (handy in tests) with WithoutSignals. The defaults are the right ones for a service in a container.

Health, before anything’s asking

That third verb, WithStatus, is the start of the health story, and it’s worth seeing now even though nothing’s calling it yet. The controller can report three separate things: overall status, liveness, and readiness. Those aren’t the same question. Liveness is “is this process wedged and in need of a restart”; readiness is “should traffic come here yet”. An orchestrator uses them differently, which is why they’re kept apart, and a service can answer them separately by adding WithLiveness and WithReadiness alongside WithStatus.

Right now nothing asks, because we’ve no transport. But this is the quiet payoff of putting lifecycle first: when we add an HTTP server in part 3, these reports are what back its /healthz, /livez and /readyz endpoints, and when we add gRPC in part 2 they back the standard gRPC health service, with no re-plumbing on our side. The controller also carries a restart policy for services that should pick themselves back up, and standalone health checks for things like “can I still reach the database”, but those earn their place once we’ve something worth checking.

Where this leaves us

A few lines in, we’ve a process that starts, ticks along, answers for its own health, and stops cleanly when the platform asks it to, on a controller that the real transports will register against unchanged. The heartbeat is a stand-in. Next part we swap it for a proper gRPC service, give it TLS, and the controller barely notices the difference, which is the entire point of it.

If you want to read ahead, the controls component has the full interface, and the service-orchestration deep-dive covers how the startup ordering and shutdown actually work underneath.

Building a CLI with go-tool-base, part 5: a CLI that updates itself

Sun, 24 May 2026 00:00:00 +0000

You ship version one. A week later someone finds a bug, you fix it, you cut version two. Now for the awkward part: how does the person who installed version one ever get version two? Email them? Hope they wander back to the install page? For a CLI that lives on people’s machines, “go and re-download it” is the answer that quietly strands half your users on old, broken builds. This part closes that gap, and like most of this series, the work is already done for you: your tool has shipped with an update command since part 1.

As before, this is written against go-tool-base v0.6.0 (gtb version).

The command is already there

update is one of the default features, so it’s been in your binary all along. Your users run:

mytool update

and the tool fetches the newest release, checks it, and replaces itself in place. No package manager, no re-download, no instructions. The rest of this part is about what that one command actually does, and how to make sure the binary it pulls down is the one you shipped.

Where it looks for releases

A tool can’t update itself without knowing where its releases live. That’s the --repo you passed back in part 1: it filled in your tool’s release source, the platform, owner and repository it checks. For --repo myorg/mytool that’s github.com/myorg/mytool, and mytool update looks at that project’s releases.

go-tool-base speaks more than one platform here, GitHub, GitLab, Gitea, Codeberg, Bitbucket, or a plain HTTP server, so the same command works whether you publish on github.com or your own GitLab. If you ever need to point somewhere else (a mirror, a private host), the custom release source how-to covers it; for a private repository it reads a token the same way the rest of the tool does.

What `update` does, step by step

When a user runs it, the command walks a short, careful path:

Resolve the latest release from your release source.
Compare versions. It reads the version baked into the running binary and compares it, as semver, against the latest. If you’re already current, it says so and stops: already running latest version, v1.2.0. (If your build somehow reports a version ahead of the latest published, it tells you off in character: your tardis travelled too far into the future...)
Download the right archive for the user’s OS and architecture.
Verify it before trusting it (the next section).
Replace the running binary with the new one, in place.
Bring the config along. If your tool has the init feature (it does by default), the update then runs the new binary’s init over the user’s config directory to fold in anything the release added.

That last step is easy to miss and matters more than it looks. A new version often ships new config: a key for a feature you just added, a changed default. Rather than leave the user a version behind, with code that expects settings their config file has never heard of, update re-runs init against their existing config once the swap is done, non-interactively (it passes --skip-login --skip-key, so nobody gets re-prompted for a token). It’s the same initialiser system from part 2, reused: the merge keeps what the user set and adds what the new version introduced, so the binary and its config move forward together. Turn the init feature off and this step is simply skipped, there’s no config to keep in step with.

There are two flags worth knowing. --version v1.3.0 targets a specific release instead of the latest, handy for pinning or rolling back. And --force updates even when the version check thinks you don’t need to. Most of the time, a bare mytool update is the whole story.

Downloaded isn’t the same as trusted

A binary that arrives over the network is a binary you didn’t build on the machine it’s running on, and a self-updater that swaps itself for whatever the server sent is a lovely way to ship a corrupted or tampered build straight into your users' hands. So before the swap, update verifies what it downloaded against a checksum manifest, the checksums.txt GoReleaser produces alongside your binaries. If the hash of the downloaded archive doesn’t match the one in the manifest, the update aborts and nothing gets replaced.

By default this is best-effort: a release that ships a checksums.txt is verified, but a release without one is updated with a warning rather than a hard stop. When you want the guarantee, make it mandatory:

# in your tool's config
update:
 require_checksum: true

Now a missing or mismatched checksum is a refusal, not a shrug. I wrote up why this matters, and exactly what it does and doesn’t buy you, in verifying your own downloads.

The limit is worth stating plainly, because it’s the whole reason there’s a “part two” to this story. A checksum proves the binary matches the manifest on the same release page. It catches a corrupted download or a botched upload cold. What it cannot catch is an attacker who owns the release platform and swaps both the binary and its checksum in the same breath, because then the two still agree. Closing that gap needs a signature whose trust root the release host can’t reach, which is a different piece of machinery (and a post of its own). go-tool-base now does exactly that: self-update signature verification has shipped, the binary checking a detached signature against a key it both embeds and fetches over WKD (how it works). Until you turn signing on for your own tool, checksums are the floor, and a worthwhile one.

Seeing it work without publishing anything

Here’s the catch with writing about self-update: you can’t update from a release you haven’t published, and your tutorial tool isn’t on anyone’s GitHub. There’s a flag for exactly this, meant for offline and air-gapped installs but perfect for a look under the hood: --from-file installs from a local release archive instead of the network.

Build a snapshot of your tool the way your release pipeline would (GoReleaser’s --snapshot builds the archives without publishing), then point update at one:

goreleaser release --snapshot --clean
mytool update --from-file ./dist/mytool_Linux_x86_64.tar.gz

You’ll watch the same extract-and-swap the network path uses, with nothing published and no release source involved. It’s also genuinely useful in its own right, for shipping into environments that can’t reach the internet.

The real loop

In production the cycle is the one part 1 already set you up for. The project gtb scaffolds ships a GoReleaser config and a release pipeline, so the flow is:

Tag a version and push the tag.
CI builds the binaries for every OS and architecture, generates checksums.txt, and publishes them as a release on your source.
Your users run mytool update and get it, verified.

You write git tag v1.3.0 && git push --tags; everyone who installed v1.2.0 is one command away from the fix. That’s the whole point of putting the update channel inside the tool: shipping a fix becomes tagging a release, and nothing else.

What this buys you

A tool that updates itself turns “please go and reinstall” into mytool update, and a tool that verifies what it updates to turns “I hope that download was clean” into a checked guarantee. Both came with the scaffold; the only work was understanding them. The full reference, including the config keys and the per-platform release sources, is in the update command docs and the auto-update concepts page.

Next part is the last one, and it’s about what happens after your tool is out there doing its job: telemetry and logging, so you can see how it’s actually being used without spying on the people using it. Until then, tag a release and watch your tool catch up to itself.

Building a CLI with go-tool-base, part 4: an AI dungeon master

Sat, 23 May 2026 00:00:00 +0000

I run a Dungeons & Dragons game on the odd weekend, so when I sat down to put an AI feature inside a CLI, my first instinct wasn’t a chatbot. It was: could the tool run a little adventure, with an AI as the dungeon master? It turns out that’s a near-perfect way to learn the chat client, because the thing that makes a game trustworthy, rules the players can’t break, is exactly the thing that makes any AI feature trustworthy. So this part builds mytool adventure: a tiny dungeon you play in your terminal, narrated by an AI that is firmly on a leash.

Part 3 pointed AI at your CLI from the outside (an agent driving your commands over MCP). This part goes the other way: AI inside your tool, as a feature you write. The worry everyone has about that is fair, AI output is unpredictable, and a CLI is meant to be dependable. The whole lesson here is how you square those two: you don’t hope the model behaves, you box it in with rules it can’t escape and mechanics it doesn’t get to invent.

As before, this is written against go-tool-base v0.6.0 (gtb version).

Behind the DM screen

A turn of our game looks like this: the player types what they want to do, the AI dungeon master narrates what happens and offers a few choices, and round it goes until the adventure reaches an end. The trick is where the truth lives. The model’s job is the prose, and only the prose. Everything else is yours:

The rules live in the system prompt: what the DM may and may not do.
The mechanics live in Go functions the model calls as tools (dice, combat). It never makes a number up.
The state lives in a Go struct you hand the model fresh every turn, so it never has to remember, and can’t quietly rewrite history.
The shape of each turn is a typed Go struct the model fills in, so your code always gets back something it can render, never a wall of prose to parse.

Two go-tool-base capabilities do the heavy lifting: the AI calling your Go functions, and the AI handing back a typed struct instead of text you have to regex. The game is just a fun excuse to use both at once.

Wiring a provider

The chat client (pkg/chat) is a library you import; you don’t need any special feature flag for it. It does need an API key, and it’ll find one from a few places. The simplest, for now, is the well-known environment variable for your provider:

export ANTHROPIC_API_KEY="sk-ant-..." # or GEMINI_API_KEY, OPENAI_API_KEY

That’s the bottom of the client’s lookup chain, which is fine for playing locally. For a tool you actually ship, go-tool-base has the ai feature and its mytool init wizard (the same initialiser system from part 2) to store the key properly, and there’s a whole post on where a CLI should keep your keys. For learning the client, an env var is plenty.

Scaffold the command

You know this step from part 1:

gtb generate command --name adventure --short "Play a dungeon adventure"

Everything below goes in the RunAdventure function the generator left you in pkg/cmd/adventure/main.go, plus a couple of types and helpers in the same package.

The state is yours, not the model’s

Start with the truth. The game state is a plain Go struct that you own. The model never holds it; instead you hand it the current state at the top of every turn (more on that in the loop). This is the part to grow: start small, then add rooms, items, NPCs, quest flags, whatever your adventure needs. Nothing else in the design has to change when you do.

// GameState is the single source of truth for the game. Extend it freely.
type GameState struct {
	PlayerHP int
	Location string
	Inventory []string
	Foes map[string]int // foe name -> remaining hit points
}

// summary renders the state into a line the model is given each turn.
func (g *GameState) summary() string {
	foes := make([]string, 0, len(g.Foes))
	for name, hp := range g.Foes {
		foes = append(foes, fmt.Sprintf("%s (%d HP)", name, hp))
	}
	return fmt.Sprintf("You have %d HP, at %s, carrying %s. Foes: %s.",
		g.PlayerHP, g.Location, strings.Join(g.Inventory, ", "), strings.Join(foes, ", "))
}

And the shape of a turn, the thing the model has to produce:

type Turn struct {
	Narration string `json:"narration"`
	Choices []string `json:"choices"`
	GameOver bool `json:"game_over"`
}

The dungeon master’s tools

A tool in pkg/chat is a chat.Tool: a name, a description the model reads to decide when to use it, a parameter schema, and a handler. The handler gets the model’s arguments as raw JSON and returns any value (which the framework JSON-encodes back to the model) or an error.

The simplest possible one is a die roll. This is the canonical “give the model something it’s bad at” tool, because language models cannot be trusted to roll fairly or even add up:

func rollTool() chat.Tool {
	return chat.Tool{
		Name: "roll",
		Description: "Roll a die with the given number of sides; returns 1..sides.",
		// Use an anonymous struct so the schema's properties sit at the top level,
		// which is where SetTools looks. A named type would hide them behind a $ref.
		Parameters: jsonschema.Reflect(struct {
			Sides int `json:"sides" jsonschema:"description=number of sides on the die"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			var a struct {
				Sides int `json:"sides"`
			}
			if err := json.Unmarshal(args, &a); err != nil {
				return nil, err
			}
			if a.Sides <= 0 {
				a.Sides = 20
			}
			return rand.Intn(a.Sides) + 1, nil
		},
	}
}

That comment about the anonymous struct matters, by the way. Reflect a named type and jsonschema emits a top-level reference with the real fields tucked inside, and the tool ships with no parameters at all. An anonymous struct inlines them where the framework expects. It’s the one sharp edge in the whole exercise.

Combat is where state actually changes, so combat is a tool too. Note it takes the foe by name and looks it up in Foes, so it works for the goblin and for any creature you add later, without touching this function:

func attackTool(game *GameState) chat.Tool {
	return chat.Tool{
		Name: "attack",
		Description: "Resolve the player's attack on a named foe. Rolls to hit, applies damage.",
		Parameters: jsonschema.Reflect(struct {
			Target string `json:"target" jsonschema:"description=the name of the foe being attacked"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			var a struct {
				Target string `json:"target"`
			}
			if err := json.Unmarshal(args, &a); err != nil {
				return nil, err
			}
			hp, ok := game.Foes[a.Target]
			if !ok {
				return map[string]any{"error": "no such foe: " + a.Target}, nil
			}
			if rand.Intn(20)+1 < 10 {
				return map[string]any{"hit": false, "foe": a.Target}, nil
			}
			dmg := rand.Intn(6) + 1
			hp -= dmg
			if hp < 0 {
				hp = 0
			}
			game.Foes[a.Target] = hp
			return map[string]any{
				"hit": true, "foe": a.Target, "damage": dmg,
				"foe_hp": hp, "defeated": hp == 0,
			}, nil
		},
	}
}

A bad target comes back as a plain error string, which the framework hands to the model so it can recover (apologise, pick a real foe) rather than crash.

That’s the whole tool set, and there’s deliberately nothing here for reading the state. The model never fetches it. Instead the loop hands it the current state at the top of every turn, which we wire up shortly. A language model has no memory you can rely on, so rather than trust it to remember the fight, you give it the truth each time.

The turn is a tool too

Here’s the neat part. The chat client won’t let a single call both run tools and return a typed struct, they’re separate modes. So instead of asking for the struct afterwards, we make submitting the turn into a tool of its own. The dungeon master ends its turn by calling submit_turn, and its handler captures the typed Turn into a variable we hold:

func submitTurnTool(out *Turn) chat.Tool {
	return chat.Tool{
		Name: "submit_turn",
		Description: "End your turn. Call this exactly once, last, with the turn's outcome.",
		Parameters: jsonschema.Reflect(struct {
			Narration string `json:"narration" jsonschema:"description=two-sentence narration of what just happened"`
			Choices []string `json:"choices" jsonschema:"description=the actions the player may take next"`
			GameOver bool `json:"game_over" jsonschema:"description=true only if the game has ended"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			if err := json.Unmarshal(args, out); err != nil {
				return nil, err
			}
			return "turn recorded", nil
		},
	}
}

So the turn’s structure is enforced by a schema, same as any other tool’s parameters. Your loop gets a populated Turn every round, never prose.

The rules

This is where you bound the model. The system prompt is the rulebook, and it leans hard on the tools so the DM has no room to freelance the mechanics:

const dmRules = `You are the dungeon master of a short fantasy adventure. Each turn
you are given the current game state and the player's action.

Resolve the action and end the turn:
- If the player attacks, you MUST call the attack tool with the foe's name to
 resolve it. Do not decide the hit or the damage yourself.
- For any other chance event, call the roll tool and use its result.
- For simple actions, just narrate them.
- Then call submit_turn exactly once: a two-sentence narration, two or three
 choices, and game_over.

Trust the state you are given; never contradict it. A foe at 0 hit points is dead
and stays dead. The game ends when the player's hit points reach 0 (they lose), or
when the player reaches a satisfying ending. When it ends, set game_over and narrate
the finish.

Keep the tone light and quick.`

Two of those lines carry the weight. Trusting the state you are given, and never contradicting it, is what keeps the world consistent: the state is handed in fresh every turn (the next section), so the model works from the truth instead of from a memory it does not reliably have. And you MUST call the attack tool is what stops it quietly deciding hits and damage itself when it would rather just narrate. Those two are the difference between a game with rules and a model telling a story.

The loop

Now stitch it together. Create the client with the rules as its system prompt, register the tools once, and run a turn each time the player acts:

func RunAdventure(ctx context.Context, props *props.Props, opts *AdventureOptions, args []string) error {
	game := &GameState{
		PlayerHP: 20,
		Location: "the mouth of a damp cave",
		Inventory: []string{"a short sword", "a guttering torch"},
		Foes: map[string]int{"goblin": 12},
	}
	var turn Turn

	client, err := chat.New(ctx, props, chat.Config{
		SystemPrompt: dmRules,
		MaxSteps: 8, // roll/attack, then submit_turn
	})
	if err != nil {
		return err
	}

	if err := client.SetTools([]chat.Tool{
		rollTool(),
		attackTool(game),
		submitTurnTool(&turn),
	}); err != nil {
		return err
	}

	action := "I step into the cave."
	for {
		turn = Turn{}
		// Hand the model the current truth, then the player's action.
		input := fmt.Sprintf("State: %s\nThe player: %s", game.summary(), action)
		if _, err := client.Chat(ctx, input); err != nil {
			return err
		}

		fmt.Println("\n" + turn.Narration)
		if turn.GameOver {
			return nil
		}
		action, err = chooseAction(turn.Choices)
		if err != nil {
			return err
		}
	}
}

The same client runs every turn, so the conversation and the tools carry through the whole game; and the State: line you prepend is always current, because the attack tool mutated game last turn. The model is never trusted to remember, only to narrate.

The one helper I glossed is chooseAction. A bare fmt.Scanln would do, but we can do much better with almost no effort, and make a point while we’re at it. The framework already leans on Charm’s huh for its init wizard, you met it in part 2, so we’ll use the same library for a proper menu, with one deliberate addition:

func chooseAction(choices []string) (string, error) {
	const other = "__other__"

	opts := make([]huh.Option[string], 0, len(choices)+1)
	for _, c := range choices {
		opts = append(opts, huh.NewOption(c, c))
	}
	opts = append(opts, huh.NewOption("Something else...", other))

	var pick, custom string
	form := huh.NewForm(
		huh.NewGroup(
			huh.NewSelect[string]().
				Title("What do you do?").
				Options(opts...).
				Value(&pick),
		),
		// A second step that only appears when the player chose "Something else".
		huh.NewGroup(
			huh.NewInput().
				Title("Describe your action").
				Value(&custom),
		).WithHideFunc(func() bool { return pick != other }),
	)
	if err := form.Run(); err != nil {
		return "", err
	}

	if pick == other {
		return custom, nil
	}
	return pick, nil
}

The select gives the player a tidy arrow-key menu instead of typing a number, but the addition that earns its keep is the last option. “Something else…” is always there, and choosing it unfolds a second step (huh shows or hides a group with WithHideFunc) where the player types whatever they actually want to do. That free text goes straight to the dungeon master as the next turn’s input, and because the DM is an AI bound by the rules rather than a switch statement over three fixed choices, it just copes. Bargain with the goblin, search your pockets, set the cave alight: the model narrates it within the rules you gave it, rolling and applying damage through the same tools. That is the agency a scripted game can’t offer, and it’s the natural place to start building your own richer interactivity on top.

Play it

Set your key, build, and go:

export ANTHROPIC_API_KEY="sk-ant-..."
just build
./bin/mytool adventure

A turn looks like this (your wording will differ every time; the mechanics won’t):

You swing your short sword at the goblin, the blade whistling through the damp cave
air. The creature snarls as it tries to dodge your blow.

What do you do?
> Attack the goblin again
 Try to push deeper into the cave
 Retreat to the entrance
 Something else...

Your blade whistles through the air, but the nimble goblin dances back just in
time. It lunges forward with a rusty dagger in return, yet its clumsy strike only
finds empty air.

What do you do?
> Swing your sword again!
 Try to intimidate the creature
 Retreat from the cave
 Something else...

Behind that, the dungeon master called attack each turn (a hit, then a miss), the goblin’s hit points changed in the GameState you own, and the next turn handed that updated state straight back to the model. The prose is the model’s; every number is yours.

The pattern under the game

Strip the dungeon away and you’re left with the thing worth keeping. An AI feature you can ship is one where you’ve kept the model away from everything that has to be right: the rules live in the system prompt, the mechanics in typed Go tools the model must call, the state in a struct you hand it fresh each turn rather than trust it to remember, and the output in a struct it fills in rather than free text. Do that and the model’s unpredictability is confined to exactly where you want it, the wording, and walled out of everywhere you don’t, the maths, the state, the shape of the result.

Two honest limits worth knowing. There’s no temperature dial on the client (the setting that would let you turn the model’s randomness down), so you can’t make the prose reproducible; you make the mechanics reproducible instead, which for most features is what you actually needed. And a tool calling loop is several round-trips to the model per turn, so it’s not free, keep MaxSteps tight for anything interactive.

That’s the foundation, and the state struct is already sized for more than one fight: it carries a location, an inventory and a map of foes you’ve barely touched. Add a move tool that updates Location, a use_item tool that reaches into Inventory, a second creature in Foes, even a give_quest flag, and the adventure grows without the architecture changing. The model just gets more tools to call and more truth to read. Saved games come nearly free, too: the client can snapshot and resume a conversation. Next part leaves AI behind and gets the tool ready to look after itself: shipping signed self-updates, so a new release reaches your users safely. Until then, go explore the cave.

Building a CLI with go-tool-base, part 3: expose your CLI to AI agents

Fri, 22 May 2026 00:00:00 +0000

“Make it work with AI” is the request that lands on your desk with a thud and no further detail. The first time it landed on mine I braced for a treadmill of integration work: an adapter for this assistant, a wrapper for that one, one per client, forever. Then I looked at the hello command we built back in part 1. It has a name, a one-line description, and (once you give it some) typed, documented flags. That is exactly the shape an AI agent needs to call a tool. You already did the hard part.

This part wires that up: turning the CLI you’ve been building into something an AI assistant can drive, with no AI code of your own. The how-it-works behind it is in your CLI is already an AI tool; here we just use it.

A version note, as in the earlier parts: this is written against go-tool-base v0.6.0 (gtb version). The tool is young and moving, so if you’re on a newer release and a command or its output has shifted, check there first.

The translator you already have

An AI agent that wants to call local tools needs three things: a list of named operations, a description of each so it knows when to reach for them, and a typed parameter schema for each so it knows how to call them. A good CLI is already all three. The only missing piece is a translator between “this is a CLI” and “this is a set of tools an AI can call”, and that translator is the Model Context Protocol (MCP), an open standard every serious assistant now speaks.

Your tool already ships it. mcp is one of the default features, so it’s been in your binary since you scaffolded in part 1, no flag required. Check:

./bin/mytool mcp --help

You’ll see subcommands you never wrote. The rest of this part is just three of them.

See what the agent sees

Before you connect anything, look at what your tool would expose. This writes the tool definitions to a file:

./bin/mytool mcp tools

{
 "tools": [
 {
 "name": "mytool_hello",
 "description": "Say hello",
 "inputSchema": {
 "type": "object",
 "properties": {}
 }
 }
 ]
}

That’s your hello command, seen from an agent’s side of the glass. The name is your tool’s name and the command path joined with an underscore; the description is the Short you gave it in part 1; the inputSchema is empty because hello has no flags yet. Add a flag and it shows up here as a property, with the type and help text you already wrote. There’s no second schema to keep in sync, because the command tree is the schema.

A few things are deliberately left off this list: hidden and deprecated commands, pure command groups that don’t do anything themselves, and the mcp, help and completion plumbing. So mcp tools doubles as an audit: it’s exactly what an agent can reach, and nothing else.

Run the server

One command turns the whole thing on:

./bin/mytool mcp start

It doesn’t print a banner and march off doing things. It sits quietly, speaking MCP as JSON-RPC over standard input and output, waiting for an assistant to talk to it. You won’t run this by hand much; the assistant launches it for you. But it’s worth knowing what happens when the agent calls one of your tools: the server re-runs your own binary with the arguments the agent supplied, captures the output, and hands it back. The agent isn’t poking at your internals. It’s running mytool hello, the same command a human would, and getting the same result.

Point an assistant at it

The quickest way is to let the tool write the client config for you. For Claude Desktop:

./bin/mytool mcp claude enable

There are cursor and vscode variants too. Restart the assistant and your CLI is in its toolbox.

If you’d rather wire it by hand (or your client isn’t one of those three), the config is small. Point the client at your binary with mcp start as its arguments, using the absolute path:

{
 "mcpServers": {
 "mytool": {
 "command": "/absolute/path/to/bin/mytool",
 "args": ["mcp", "start"]
 }
 }
}

Claude Desktop keeps that in claude_desktop_config.json (under ~/Library/Application Support/Claude/ on macOS, %APPDATA%\Claude\ on Windows); Cursor uses ~/.cursor/mcp.json; VS Code’s Copilot reads github.copilot.mcpServers in your settings. The shape is the same everywhere. From here, ask the assistant to say hello and watch it call mytool_hello.

The agent’s reach is exactly your CLI’s reach

This is the part worth being calm about. Exposing your CLI over MCP doesn’t widen its surface by an inch. The agent can call the commands you shipped, with the parameters you defined, and nothing else. It can’t invent a command or pass a flag you never wrote. The boundary of what it can do is the boundary you drew when you built the tool, and mcp tools shows you that boundary in full. If there’s a command you don’t want an agent touching, mark it hidden and it drops off the list.

For a long-running or remote setup there’s also ./bin/mytool mcp stream, which serves the same tools over HTTP instead of stdio; the mcp reference has the details. For most desktop assistants, start over stdio is all you need.

What it comes down to

You turned the CLI you’ve been building into an agent-callable tool with one command and zero lines of AI code, because the real work, naming your operations and documenting their inputs, you finished the moment your --help was any good. Every command you add from here is a new tool the agent gets for free.

Next part goes the other way: instead of letting an assistant drive your tool from outside, we put AI inside it, wiring up a provider and building a feature against go-tool-base’s chat SDK. Until then, add a command or two and watch them appear in mcp tools. The agent’s toolbox grows as your CLI does.

Building a CLI with go-tool-base, part 2: configuration your tool can trust

Thu, 21 May 2026 00:00:00 +0000

In part 1 you scaffolded a tool and gave it a hello command. It says the same thing every time, which is fine for a first command and useless for a real one. The moment a tool does anything worth doing it needs settings: an endpoint, a default, a token, a log level. And the moment you have settings, you have the problem nobody warns you about. You set one in a file, the tool ignores it, the code that reads it looks perfectly correct, and an hour later you find you’d typed tiemout. Nothing in the whole stack thought that worth a word.

The good news is you don’t have to build any of this. Your scaffold already wired up a config system in part 1, the same one the rest of go-tool-base uses. This part puts it to work: where a setting’s value actually comes from, how to ship sensible defaults alongside the command they belong to, how to layer files so a team and a laptop can disagree politely, and how to turn a fat-fingered key from a silent shrug into an error that tells you exactly what you got wrong.

The same version note as part 1, since each of these stands on its own: everything here is written against go-tool-base v0.6.0 (gtb version will tell you what you’re on). The tool is young and still changing shape, so if you’re on a newer release and a detail has drifted, that’s the first thing to check. I’ll flag anything that breaks across versions as it comes up.

You already have a config system

The root command loads configuration for you before any of your command code runs, merges every source together, and hands the result to each command through Props. By the time your RunHello runs, props.Config is populated and ready.

A value can arrive from several places at once, so there’s an order. Highest wins:

Command-line flags
Environment variables (your tool’s prefix plus the key, so hello.greeting reads MYTOOL_HELLO_GREETING, with the dots turned into underscores)
Config files (on disk, in the order they were loaded)

That ladder is the mental model for what beats what: a flag beats an env var, an env var beats a file. The files are worth pinning down, though, because there’s more than one and they don’t all come from the same place. This is the bit that’s easy to trip over:

Embedded defaults are baked into the binary, one slice per command. You don’t read these at runtime directly. The init command (coming up) bakes them into your config file for you.
The file init writes, ~/.mytool/config.yaml, is the default the tool reads, along with a machine-wide /etc/mytool/config.yaml if one exists.
Files passed with --config replace those defaults for that run rather than adding to them. Name one or more and the tool reads exactly those.

We’ll set each of these up in turn. The full reference lives in the config docs.

Reading a value is one call, and it’s typed:

greeting := props.Config.GetString("hello.greeting")
timeout := props.Config.GetDuration("server.timeout")
debug := props.Config.GetBool("verbose")

Give a command a setting

Let’s make hello configurable. Open pkg/cmd/hello/main.go (your file, the one the generator leaves alone) and read the greeting from config instead of hard-coding it:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	greeting := props.Config.GetString("hello.greeting")
	props.Logger.Info(greeting)
	return nil
}

Build and run it:

just build
./bin/mytool hello

ERRO failed to load config: no configuration files found
please run init, or provide a config file using the --config flag

Not what you expected, maybe, but it’s the right instinct from the tool. It has no configuration to read yet, and rather than guess, it stops and says so. Which brings us neatly to where settings actually come from.

Defaults belong to the command

You could drop a default into the project’s central config, and for something truly global like the log level that’s the right home. But a setting that belongs to hello should live with hello, not in a shared file you have to remember to edit every time you add a command. The generator does this for you, you just have to ask. Back in part 1 you generated hello without config support, so run the same command again with --assets:

gtb generate command --name hello --short "Say hello" --assets

This is safe to re-run. The generator honours the code you’ve already written: it refreshes the boilerplate cmd.go, adds the asset scaffolding, and leaves your main.go, and the RunHello you’ve been editing, completely alone. One thing to hold off on here: don’t reach for --force. Force rewrites everything, including that main.go, which is exactly the work you want to keep.

You now have pkg/cmd/hello/assets/init/config.yaml, and the generator has already opened it under the command’s own namespace:

hello:

Fill in your defaults under it:

hello:
 greeting: Hello
 style: plain

Those values are embedded into the binary as an asset, and the generated cmd.go registers them with Props for you (props.Assets.Register("hello", &assets)), so the config system knows where your command’s defaults live. A quick word on style, since we’ll lean on it shortly: it’s a second setting I’m giving a default now so it’s ready when we need it. Plain says the greeting as written; loud will shout it.

That per-command home comes with one rule worth taking seriously: namespace your keys. Notice the generator opened the file under a hello: key rather than at the top level. Copy that. Every command ships its defaults in its own embedded file, and those files are all merged together to build the config, but the order they merge in is not guaranteed. If two commands both defined a top-level timeout, which one won would be a toss-up that could flip between builds. Keep each command’s settings under its own name (hello.greeting, report.timeout) and the clash can’t happen in the first place. The generator namespacing the file for you is a hint worth taking.

One thing the defaults file does not do is set values through struct tags. If you later add a default:"info" tag to a config field, that’s documentation for the error messages, nothing more. Real defaults live here, in the embedded YAML. It’s an easy thing to assume otherwise and then wonder why your default never applied.

First run: init

So your defaults are baked into the binary. The tool still needs an actual config file to read, and that’s what init is for. It’s one of the features your tool shipped with, so it’s already there:

./bin/mytool init

INFO Configuration initialised in /home/you/.mytool/config.yaml

Open that file and you’ll find your command’s defaults waiting in it, merged with the framework’s own:

hello:
 greeting: Hello
 style: plain
log:
 level: info

That’s the missing piece. init gathers every command’s embedded defaults through the Assets layer, writes them to ~/.mytool/config.yaml, locks the file down to 0600 (it may hold secrets later), and drops in a .gitignore so nobody commits it by accident. Now hello has something to read:

Prefer no init step? init is a feature, and you can leave it out of your tool’s feature set. With it off, the tool loads its embedded defaults directly and runs with no config file at all, you’d only add one to override something. That suits a small, self-contained tool. This tutorial keeps init on, which is the default and the right call while a tool is finding its feet, so the rest of the article assumes it.

./bin/mytool hello

INFO Hello

Setup that needs a human: initialisers

Static defaults cover the values you can decide for the user. Some you can’t: a token, an API key, an endpoint that differs per person. Writing a blank or guessed value for those is worse than useless. This is where go-tool-base does something I’ve not seen many CLI frameworks bother with: it lets a command bring its own first-run setup, and wires it in for you. It’s one of the genuine reasons to build on the framework rather than roll your own, so it’s worth a proper look.

Generate a command with --with-initializer:

gtb generate command --name greet --short "Greet someone" --with-initializer

Alongside the usual files you get an init.go. It’s generated and marked DO NOT EDIT, and it does all the wiring. Here’s the heart of it:

// Code generated by gtb. DO NOT EDIT.
package greet

func init() {
	setup.Register(props.FeatureCmd("greet"),
		[]setup.InitialiserProvider{func(p *props.Props) setup.Initialiser {
			if skipGreet {
				return nil
			}
			return &GreetInitialiser{}
		}},
		[]setup.SubcommandProvider{func(p *props.Props) []*cobra.Command {
			return []*cobra.Command{NewCmdInitGreet(p)}
		}},
		[]setup.FeatureFlag{func(cmd *cobra.Command) {
			cmd.Flags().BoolVar(&skipGreet, "skip-greet", false, "skip initializing greet configuration")
		}},
	)
}

type GreetInitialiser struct{}

func (i *GreetInitialiser) Name() string { return "greet" }

func (i *GreetInitialiser) IsConfigured(cfg config.Containable) bool {
	return cfg.IsSet("greet")
}

func (i *GreetInitialiser) Configure(p *props.Props, cfg config.Containable) error {
	return InitGreet(p, cfg)
}

That package init() registers three things with the framework the moment your command is imported, with no central setup file for you to edit: the initialiser itself, an init greet subcommand so the user can reconfigure just this command later, and a --skip-greet flag on the main init. IsConfigured is how the framework avoids nagging: if the greet key is already in the config, init leaves it be and moves on.

All of that is generated for you. The one piece that’s yours is the InitGreet function in main.go, which starts as a stub:

func InitGreet(p *props.Props, cfg config.Containable) error {
	// TODO: Implement custom initialization logic for greet
	return nil
}

Fill it in with whatever the setup needs. go-tool-base leans on huh for prompts, the same library its own GitHub and AI setup use, so a one-question form looks like this:

func InitGreet(p *props.Props, cfg config.Containable) error {
	var greeting string

	form := huh.NewForm(
		huh.NewGroup(
			huh.NewInput().
				Title("What greeting should greet use?").
				Value(&greeting),
		),
	)
	if err := form.Run(); err != nil {
		return err
	}

	cfg.Set("greet.greeting", greeting)
	return nil
}

Set the value on cfg and you’re done. After the initialisers run, init writes the whole config out to disk, so the answer persists into ~/.mytool/config.yaml with everything else. Run mytool init on a fresh machine now and it stops to ask for the greeting; run it again and it sails past, because IsConfigured sees the key is already there. Need to redo just this one command’s setup? mytool init greet. The framework hands each command its own setup step, its own subcommand and its own skip flag, and asks you for a single function in return. That’s the trade worth making: static defaults in your embedded YAML, anything that needs a human in an initialiser.

Overriding: the environment and layered files

With a config file in place, the other sources come into their own. The quickest override is an environment variable. Remember the prefix you set when scaffolding in part 1: hello.greeting maps to MYTOOL_HELLO_GREETING, the prefix and key joined up, uppercased, dots turned to underscores:

MYTOOL_HELLO_GREETING="Hello from mytool" ./bin/mytool hello

INFO Hello from mytool

You didn’t register that variable anywhere; the config system binds it for you. The prefix is what keeps it from colliding with some other tool’s LOG_LEVEL on the same machine, which is exactly why it’s worth having.

Files are the other half, and they’re where that precedence list earns a closer look. A single config file is fine until two people, or two machines, want slightly different settings, and then you’re copying files around by hand. The --config flag fixes that: pass it more than once and the tool merges the files in order.

./bin/mytool hello \
 --config ./config.yaml \
 --config ./config.local.yaml

Between the files you name, the rule is later wins on a clash, and every key that doesn’t clash is kept. If config.yaml sets hello.greeting: Hello and config.local.yaml sets hello.greeting: Oi, you get Oi, but keys that appear in only one file survive untouched. It’s a merge between them, not a replacement.

The edge to remember is what --config does to the default locations: it replaces them. The moment you name a file, ~/.mytool/config.yaml drops out of the picture unless you name it too. So you pass the whole stack you want, a shared base and a local override together, and let precedence settle it. Commit a config.yaml with the team’s settings, keep an untracked config.local.yaml for your own, run with both, and your local tweaks win without anyone editing a shared file. Leave --config off and you’re back on the defaults init wrote: ~/.mytool/config.yaml plus that machine-wide /etc/mytool/config.yaml if it’s there. Whichever set of files you land on, environment variables and flags still sit on top.

The typo that does nothing

Now for the failure I keep circling. Say you want to change the greeting. Open your config, but fat-finger the key:

hello:
 greting: Oi  # meant to be greeting

Run it, and you get a blank line. The greeting you set never applied: the misspelled key was read, matched nothing, and was silently dropped, and the real greeting is now nowhere to be found. Nothing said a word. For a greeting it’s a shrug. For a timeout or a retry count it’s the bug you chase at 2am, and I wrote up the why of it in the config key that quietly did nothing.

go-tool-base won’t catch this for you by default, and that’s a choice rather than an oversight. There’s no central schema that knows every key your tool could ever take, because keys belong to the commands that use them. What you get instead is a way to opt a command in, so it validates its own slice and nobody else’s.

Making mistakes loud

Tell the generator you want validation for a command and it scaffolds exactly this (gtb generate command --name hello --with-config-validation). Since hello already exists, it’s a small file to add by hand. Create pkg/cmd/hello/config.go:

package hello

import "gitlab.com/phpboyscout/go-tool-base/pkg/config"

// HelloConfig describes the config keys the hello command consumes.
type HelloConfig struct {
	Greeting string `config:"hello.greeting" validate:"required"`
	Style string `config:"hello.style" enum:"plain,loud" default:"plain"`
}

// ValidateHelloConfig checks the hello config against its schema.
func ValidateHelloConfig(cfg config.Containable) error {
	return config.ValidateStruct[HelloConfig](cfg)
}

The tags carry the rules. validate:"required" means the key has to be present and non-empty. enum:"plain,loud" means style has to be one of those two words. config.ValidateStruct[HelloConfig] does the rest: it derives a schema from those tags and checks the config against it, returning a readable error if anything is off. It takes props.Config as it is, the Containable interface, so there’s no casting to a concrete type. Call it at the top of RunHello, before you trust any of the values, and use the style while you’re there:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	if err := ValidateHelloConfig(props.Config); err != nil {
		return err
	}

	greeting := props.Config.GetString("hello.greeting")
	if props.Config.GetString("hello.style") == "loud" {
		greeting = strings.ToUpper(greeting)
	}

	props.Logger.Info(greeting)
	return nil
}

(You’ll add strings to the imports at the top of main.go.)

Now make a real mistake. Set the style to something that isn’t allowed:

hello:
 greeting: Hello
 style: shout

ERRO config validation failed:
 hello.style: value "shout" is not allowed (hint: allowed values: plain, loud)

That’s the difference. The command stops and tells you the key, the bad value, and what it would have accepted. The same check catches a misspelled greeting: the moment the real key goes missing, required fails with hello.greeting: required field is missing instead of quietly running on nothing. Set style: loud and you get HELLO, because the value finally passes and the code downstream can trust it.

If you switch on the optional config feature (it isn’t in the default set, so you opt into it), you also get a ready-made mytool config validate command that runs these checks without you wiring anything into a command at all. Either way, the principle holds: the program already knows what good config looks like, so make it say so when the config is bad.

The upshot

Your hello command now reads a real setting, ships a sensible default that init writes into place, honours overrides from the environment and from layered files in a predictable order, and refuses to run on a value it doesn’t understand. That’s most of what configuration ever needs to be, and you wrote almost none of the machinery.

One thing I’ve skipped: config can also reload while the tool is running, so a long-lived process picks up a changed file without a restart. That’s its own capability with its own moving parts, and I pulled it apart in reloading config without a restart if you need it.

Next part, we give the tool something to do with all this config: we turn it into an AI tool, with a chat command and an MCP server. Until then, go add a couple of validated settings to your own commands. You’ve got the shape of it now.

Building a CLI with go-tool-base, part 1: scaffold and your first command

Wed, 20 May 2026 00:00:00 +0000

Every time I start a new Go CLI, the first hour goes the same way, and none of it is the actual tool. Config loading. A logger. An update command. An error path that prints something a human can act on. A help system. I built go-tool-base so I’d never write that hour again, and I’ve spent a good few posts explaining how the pieces work inside. This series is the other half: how you use it. By the end you’ll have a real CLI with all that wiring for free. This part scaffolds one and gives it its first command.

One note on shape before we start: each part stands on its own. Finish this one and you’ve got a working, buildable tool. Later parts add configuration, AI, self-update and telemetry, one at a time. Where you want to know how a piece works underneath, I’ll link the deep-dive as we go.

Install the gtb CLI

go-tool-base ships an automation CLI called gtb. Install it with the script from the installation docs:

curl -sSL \
 "https://gitlab.com/phpboyscout/go-tool-base/-/raw/main/install.sh" | bash

That fetches a pre-built release, embedded docs and all, and drops gtb in ~/.local/bin, so make sure that’s on your $PATH. Then check it’s there:

gtb version

One thing to get out of the way before we build: versions. As I write this, gtb version prints go-tool-base v0.6.0, and that’s what every command and snippet in this series is verified against. It’s a young tool that’s still moving quickly, and the install script always pulls the latest release, so if you’re reading this later and something doesn’t line up, a newer version is the likeliest reason. When a release changes something that matters to this series, I’ll cover it in a follow-up.

Scaffold a project

One command stands up a whole project, and gtb gives you two ways to drive it.

The direct way, with flags, is good for scripting and for repeating a setup exactly:

gtb generate project \
 --name mytool \
 --repo myorg/mytool \
 --description "My CLI tool" \
 --env-prefix MYTOOL \
 --path ./mytool

--env-prefix is worth setting now: it’s the prefix for the environment variables that can override your config later (so MYTOOL_LOG_LEVEL rather than a bare LOG_LEVEL that would clash with every other tool on the box). The wizard defaults it to your tool’s name in capitals; with flags it’s worth being explicit. We’ll lean on it in part 2.

(gtb generate cli is the same command, if you prefer that name.) Or leave the flags off and gtb walks you through an interactive prompt instead, which is the gentler way the first time:

gtb generate project

Either way, one of the choices is worth calling out now, because it explains something you’ll see in a minute: features. go-tool-base bundles a set of ready-made commands, self-update, embedded docs, a doctor health check, an MCP server, a changelog, OS-keychain storage, and you choose which ones your tool ships with at generation time, either through the --features flag or a checklist in the wizard. The default set is a sensible starting point, and you can add or drop features later. That is why, a moment from now, your brand-new tool already answers --help with commands you never wrote. The full flag list is in the generate reference.

What you just got

cd mytool and look around. It’s a complete, releasable project, not a hello-world:

mytool/
├── cmd/mytool/main.go # entry point
├── pkg/cmd/root/
│ ├── cmd.go # builds Props, wires the root command
│ └── assets/init/config.yaml # embedded default config
├── internal/version/version.go # version info, stamped at release
├── .gtb/manifest.yaml # the generator's record of your command tree
├── .github/workflows/ # lint, test, docs, release pipelines
├── justfile # build / test / lint / docs tasks
├── go.mod # with `go tool` deps pinned
└── ... # .golangci.yaml, .goreleaser.yaml, README, CHANGELOG

One file there is worth understanding before anything else: .gtb/manifest.yaml. It is the generator’s source of truth, a record of every command your tool has, how they nest, and a content hash of each generated file. You won’t edit it by hand, but gtb reads and rewrites it constantly. It is how the generator knows what your command tree looks like, and how it can tell whether you have changed a file it owns. Think of it as the map the generator builds from: it’s committed to git for you, and as long as it’s there, your tool’s structure stays reproducible. We’ll see it earn its keep when we regenerate.

The entry point, by contrast, is tiny, because the framework does the lifting. Here’s the generated cmd/mytool/main.go in full:

// Code generated by gtb. DO NOT EDIT.

package main

import (
	"mytool/internal/version"

	gtbRoot "gitlab.com/phpboyscout/go-tool-base/pkg/cmd/root"
	"mytool/pkg/cmd/root"
)

func main() {
	rootCmd, p := root.NewCmdRoot(version.Get())
	gtbRoot.Execute(rootCmd, p)
}

Two lines of body. root.NewCmdRoot (in your pkg/cmd/root/cmd.go) builds a Props, the container that carries the logger, config, filesystem and version to every command. gtbRoot.Execute runs it and routes any failure through one consistent error handler, so there’s no os.Exit scattered about. Note the DO NOT EDIT header: main.go and the root cmd.go belong to the generator. Your code goes elsewhere, which matters in a minute.

Build it and you already have a working CLI:

just build # or: go build -o bin/mytool ./cmd/mytool
./bin/mytool --help

You’ll see the built-in commands from the features you picked, update, docs, doctor and the rest, with not a line written by you.

There’s one step before those commands will actually run. Try one, say ./bin/mytool docs, and the tool stops with please run init: it has no configuration yet and won’t guess at one. So give it some:

./bin/mytool init

That writes ~/.mytool/config.yaml from the defaults your tool ships with, and now its commands run. (init is itself one of the features. You can switch it off for a tool that should run straight from its built-in defaults with no file at all, but leave it on for now.) Part 2 takes configuration apart properly; for now, init once and carry on.

Add your first command

Don’t hand-roll a command file. gtb generates the boilerplate and leaves you the logic:

gtb generate command --name hello --short "Say hello"

That creates two files (see the command reference):

pkg/cmd/hello/cmd.go (generated, DO NOT EDIT): the options struct, flag wiring, and the NewCmdHello(props *props.Props) constructor.
pkg/cmd/hello/main.go (yours): a RunHello function, where all your real business logic goes.

The split is the whole point. Open pkg/cmd/hello/main.go and write what the command does:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	props.Logger.Info("hello from mytool")
	return nil
}

Rebuild, and the command is wired into the tree:

just build
./bin/mytool hello

You never touched the root command to register it. gtb recorded hello in that .gtb/manifest.yaml and wired it in for you. (If you’d rather wire commands by hand against the library directly, the custom-commands how-to shows that path; the generated route is the one this series follows.)

Regenerate without losing your work

Here’s the bit people are right to be wary of. If the generator owns cmd.go and the root wiring, what happens when it runs again, after you’ve made changes? And it runs often: every gtb generate command rebuilds the wiring.

gtb regenerate project

Your edits survive, and not by luck. Three separate things protect them:

Your logic sits in a file the generator never rewrites. Command logic lives in main.go; only the boilerplate cmd.go is regenerated. The split isn’t cosmetic, it’s the contract.
It notices if you edited a generated file. That manifest stores a content hash of every generated file, so if you’ve changed one, regeneration stops and asks before overwriting rather than silently stamping over you.
You can fence files off entirely. A gitignore-style .gtb/ignore tells the generator to leave specific paths alone, even under --force.

I wrote up how that edit-preserving diff actually works if you want the mechanism; the regenerate reference has the flags. For now, the thing to trust: scaffolding here is not a one-way door. You keep regenerating as the tool grows, and your edits stay put.

Where this leaves you

A few minutes in, you have a real CLI: config, logging, a consistent error path, self-update, embedded docs and a release pipeline, none of it written by you, plus your own hello command and the confidence to regenerate without fear. That’s the head start go-tool-base exists to give.

Next part: configuration. Typed settings, defaults the tool ships with, and how to turn a misspelled config key from a silent shrug into an error that tells you what you got wrong rather than a mystery you debug at 2am. Until then, go add a few more commands. You’ve got the pattern now.

Verifying your own downloads: how I solved it for self-updating CLI tools

Fri, 24 Apr 2026 00:00:00 +0000

Way back in the introduction I promised I’d come back to the self-update integrity checks. Here we are. And the starting point is a slightly uncomfortable admission: for a good long while, go-tool-base’s update command was the most trusting line of code in the entire tool.

The most trusting line of code in the tool

Self-update is a lovely feature. The user runs yourtool update, the tool fetches the latest release, swaps itself out, and they’re current. go-tool-base has had this since early on, wired to GitHub, GitLab, Bitbucket, Gitea and a few others.

But look closely at what that feature actually does. It reaches out to the internet, pulls down a file, and then replaces the executable that’s currently running with that file. The next time the user invokes the tool, they’re running whatever those bytes turned out to be.

The original implementation downloaded the release asset over HTTPS and extracted it. HTTPS gets you transport security: the bytes weren’t tampered with in flight. It tells you nothing about whether the bytes were right when they left, or whether they’re even the bytes you meant to fetch. A truncated download, a CDN cache serving a mangled object, a release asset that got swapped after the fact… HTTPS waves all of those straight through. For the one operation in the whole tool that replaces the binary, “we didn’t check” is an uncomfortable place to be sitting.

GoReleaser already does half the job

The good news is that the build side was already producing exactly what I needed. GoReleaser, which builds go-tool-base’s releases, generates a checksums.txt for every release: one SHA-256 per published artefact, the same format sha256sum emits. It was sitting right there as a release asset and nothing was reading it.

So Phase 1 of the integrity work is exactly that: read it.

When update downloads the platform binary, it now also fetches checksums.txt from the same release, looks up the entry for the asset it just pulled, and compares the SHA-256 of the downloaded bytes against the expected hash before anything gets extracted or installed. Mismatch, and the update aborts before it has so much as touched the installed binary. The hash comparison runs in constant time, which is more defence-in-depth than strictly necessary here, but it costs nothing and means every hash comparison in the codebase is the same and reassuringly audit-boring.

Fail open, or fail closed?

The interesting design question wasn’t the hashing. It was: what do you do when there is no checksums.txt?

Plenty of older releases predate this feature. A release might have been cut by hand without GoReleaser. If go-tool-base flatly refused to update whenever a manifest was missing, the very act of shipping this feature would brick the update path for every existing tool the moment they upgraded into it. That’s a cure worse than the disease.

So the default is fail-open: no manifest, log a clear warning, proceed. It matches how the existing offline-update path already behaved with its optional .sha256 sidecar, and it keeps upgrades working.

Fail-open as a default is not the same as fail-open being right for everyone, though. A security-sensitive tool should be able to say “no manifest, no update, full stop”. Two ways to get there:

Tool authors flip a compile-time switch (setup.DefaultRequireChecksum = true in main()) and their binary ships fail-closed from day one.
End users override either way through config (update.require_checksum) or an environment variable.

go-tool-base itself ships with the strict setting turned on, because a tool whose entire job is being a careful framework should hold itself to the stricter bar.

The caveat

Security features oversell themselves constantly, so here is the limit, stated plainly.

A checksum hosted next to the binary it describes protects you from accidents. Corruption, truncation, a CDN serving stale junk, a release asset that got partially clobbered. It does not protect you from a determined attacker who’s compromised the release platform itself. If someone can replace the binary, they can replace checksums.txt in the same breath, and your tool will cheerfully verify a malicious download against a malicious manifest and pronounce it good.

That’s not a flaw in the implementation. It’s the inherent ceiling of same-origin integrity: the manifest and the artefact share a trust root, so they fall together. Closing that gap needs a signature whose trust root is somewhere the release platform can’t reach, a key the attacker doesn’t have. That’s the next phase of this work, and it’s a bigger piece: GPG-signing the manifest, with the public half both embedded in the binary and published independently so a single platform compromise isn’t enough.

Phase 1 is the floor, not the ceiling. But it’s a floor worth having, because the overwhelming majority of real-world “the download was wrong” incidents are accidents, not attacks, and accidents are exactly what a same-origin checksum catches.

Pulling it together

The update command is the most trusting code in a self-updating tool: it fetches bytes from the internet and then becomes them. go-tool-base now verifies the SHA-256 of every self-update download against the release’s own checksums.txt before installing. It fails open by default so shipping the feature doesn’t strand anyone on an un-updatable version, fails closed for tool authors who ask (go-tool-base itself does), and stays honest that a same-origin checksum stops accidents, not a platform compromise.

Verifying your own downloads is a low bar. The point is that the previous height of that bar was zero.

The blank import that keeps a dependency out of your binary

Wed, 22 Apr 2026 00:00:00 +0000

go-tool-base can stash your credentials in the OS keychain, which most people building on it are perfectly happy about. But some of them ship into regulated and air-gapped environments where the binary isn’t permitted to contain keychain or session-bus code at all… not dormant, not unused, simply not there.

So I had a feature most users want and a minority must be able to provably not have. The way I ended up solving it is one of my favourite little bits of honest Go.

A feature some users have to be able to not have

go-tool-base needs somewhere to keep secrets: AI provider keys, VCS tokens, the occasional app password. The best home for those on a developer’s machine is the operating system’s own keychain. macOS Keychain, GNOME Keyring or KWallet on Linux via the Secret Service, Windows Credential Manager. So I wanted go-tool-base to support all three. (This is the keychain mode I mentioned back in the credentials post, finally getting the explanation I promised it.)

The Go library for that is go-keyring, and it’s good. The catch is what it drags in behind it. On Linux it talks to the Secret Service over D-Bus, which means godbus. On Windows it pulls wincred. Perfectly reasonable dependencies for a desktop tool.

Now here’s the constraint that made this interesting. Some of the people building tools on go-tool-base don’t ship to developer laptops. They ship into regulated sectors and air-gapped deployments where a security review will scan the binary, enumerate every dependency, and ask pointed questions about anything that does inter-process communication. For those builds, “the keychain code is there but we never call it” is not an acceptable answer. The reviewer’s position, and it’s a fair one, is that code which isn’t in the binary cannot be a finding.

So I had a feature that most users want, and a minority of users must be able to provably not have. Same framework, same release.

Why I didn’t reach for a build tag

The obvious Go answer is a build tag. Compile with -tags keychain to get it, leave the tag off to not. I started down that road. I even spent a while on an inverted version, a nokeychain tag, on the theory that the regulated build should be the one that has to ask, so a forgotten flag fails safe.

It works. It also isn’t very nice. Build tags are invisible at the call site. Nothing in the source tells you that a file only exists in some builds. The two worlds drift, because the tagged-out path isn’t compiled in your normal editor session and quietly rots. And the ergonomics for a downstream consumer are poor: every tool built on go-tool-base would have to know the right magic incantation and thread it through their own release pipeline correctly, forever.

I tried a second approach too: pull the keychain backend out into a completely separate Go module. That genuinely solves the dependency question (a module you don’t require can’t contribute to your go.sum). But a separate module for one backend is clunky. Separate versioning, separate release, separate repo, all for a single file’s worth of behaviour. It felt like using a shipping container to post a letter.

The shape that actually fits: a registry and an `init()`

The version I’m happy with leans on two boring, well-worn Go mechanisms and lets them do something quietly clever together.

First, pkg/credentials defines a Backend interface and a registry. By default the registry holds a stub backend that politely returns “unsupported” for everything. The framework only ever talks to the registered backend, whatever that happens to be.

Second, the keychain implementation lives in its own package, pkg/credentials/keychain, still inside the same module, no separate release to manage. That package has an init() that registers its go-keyring-backed backend:

//nolint:gochecknoinits // registration via import is the whole point
func init() {
 credentials.RegisterBackend(Backend{})
}

And go-keyring, godbus, wincred, the whole IPC dependency chain, are only imported by that package.

Now the trick. To switch keychain support on, you import the package. You don’t have to use anything from it. A blank import is enough, because a blank import still runs the package’s init():

// cmd/gtb/keychain.go - the entire file.
package main

import _ "gitlab.com/phpboyscout/go-tool-base/pkg/credentials/keychain"

That single line is the on/off switch for the shipped gtb binary. The blank import means init() runs, the keychain backend registers itself, and credential operations start routing through the OS keychain. No flag, no tag, no config.

The part that makes it provable

Here’s why this beats the build tag, and it comes down to one guarantee in the Go toolchain: the linker only includes packages that are actually imported.

If cmd/gtb/keychain.go exists, the keychain package is in the import graph, so go-keyring, godbus and wincred are linked in. Delete that one file and rebuild, and the keychain package is no longer reachable from main. The linker performs dead-code elimination, and the entire go-keyring chain is gone. Not dormant. Not present-but-unused. Absent from the binary.

That’s the bit a regulated build needs. It isn’t a promise that the code won’t run. It’s a structural fact that the code isn’t there, and you can hand a security reviewer an SBOM that proves it. go-keyring won’t appear, because it genuinely isn’t linked.

For a downstream tool built on go-tool-base the story is the same, and just as cheap. Want keychain support? Add the one-line blank import to your own cmd package. Must ship keychain-free? Don’t. Your binary’s dependency graph follows your import graph, exactly as Go always promised it would. The default (no import) is the locked-down one, which is the right way round for a safety property.

Why I like this more than I expected to

Build tags hide a decision in the compiler invocation. This pattern puts the decision in the source, as an import, where it’s greppable, obvious in code review, and impossible to get subtly wrong. There’s a real file called keychain.go whose entire content is one import, and it reads as exactly what it is: a switch.

It’s also just honest Go. No reflection, no plugin loader, no clever runtime. A registry, an init(), and the linker doing the one job it’s always done. The cleverness, such as it is, is in the arrangement, not in any individual piece.

Stepping back

go-tool-base needed OS keychain support for the many, and a way to provably exclude it for the few. Build tags could express the toggle but hid it in the build invocation and rotted in the dark. A separate module solved the dependency question but was far too much machinery for one backend.

Putting the keychain backend in its own package, activated by a blank import _ that fires its init(), gets you both: a one-line, in-source, code-reviewable switch, and, because the linker only links what’s imported, a build with the import omitted that contains none of the keychain dependency chain. Provable absence, not promised disuse.

If you’re carrying an optional dependency that some of your users need gone rather than merely idle, this is the pattern. Let the import graph be the feature flag.

Where should a CLI keep your API keys?

Mon, 20 Apr 2026 00:00:00 +0000

Your CLI tool needs the user’s API key. It has to come from somewhere, and it has to survive between runs, so the obvious move is to ask once and write it into the config file. One tidy api_key: line. Job done.

It works beautifully on the first afternoon. And then, months later, it’s quietly become a liability nobody actually decided to create.

The config file that quietly becomes a liability

Your CLI tool needs the user’s API key. It has to come from somewhere, and it has to survive between invocations, so the obvious move is to ask once and write it into the tool’s config file. ~/.config/yourtool/config.yaml, a nice api_key: line, done.

It works on the first afternoon. It keeps working. And then, slowly, it becomes a problem nobody decided to create.

The config file gets committed to a dotfiles repo. It gets caught in a tar of someone’s home directory that lands in a backup bucket. It scrolls past in a screen share. It sits, world-readable, on a shared build box. None of these are exotic. They’re just a Tuesday. The plaintext key was fine right up until the file went somewhere the key shouldn’t, and config files go places.

I didn’t want go-tool-base handing every tool built on it that same slow-motion liability by default. So credential handling got rebuilt around a simple idea: the config file should usually hold a reference to the secret, not the secret itself.

Three modes, and which one you get

go-tool-base supports three ways to store a credential.

Environment-variable reference, the default. The config records the name of an environment variable, not its value:

anthropic:
 api:
 env: ANTHROPIC_API_KEY

The secret itself lives in your shell profile, your direnv setup, or your CI platform’s secret store, wherever you already keep that sort of thing. The config file now contains nothing sensitive at all. You can commit it, back it up, paste it into a bug report. The reference is inert on its own.

OS keychain, opt-in. The config holds a <service>/<account> reference and the actual secret goes into the operating system’s keychain: macOS Keychain, GNOME Keyring or KWallet via the Secret Service, Windows Credential Manager.

anthropic:
 api:
 keychain: mytool/anthropic.api

This one is opt-in by design, because the keychain backend carries dependencies that some deployments simply aren’t allowed to ship. (That opt-in mechanism turned out to be an interesting little problem all of its own, and it gets its own post in a couple of days.)

Literal value, legacy and grudging. The old behaviour. The secret sits in the config in plaintext:

anthropic:
 api:
 key: sk-ant-...

It still works, because breaking every existing tool’s config on an upgrade would be its own kind of vandalism. But it’s the last resort, it’s documented as the last resort, and the setup wizard puts a warning in front of you when you pick it.

The one place literal mode is not allowed

There’s a single hard “no” in all of this. If go-tool-base detects it’s running in CI (CI=true, which every major CI platform sets) the setup flow will refuse to write a literal credential, and exits non-zero.

The reasoning is that a plaintext secret written during a CI run is a plaintext secret written onto an ephemeral, often shared, frequently-logged machine, by an automated process that no human is watching. That’s the exact situation where the slow-motion liability becomes a fast one. CI environments inject secrets as environment variables already; there’s no good reason for a tool to be writing one to disk there, so go-tool-base simply won’t.

How it decides at runtime

A credential can be configured more than one way at once. You might have an env reference and an old literal key still lurking. So resolution follows a fixed precedence, highest to lowest:

The *.env reference. If that env var is set, use it.
Otherwise the *.keychain reference. If a keychain entry resolves, use it.
Otherwise the literal *.key / *.value, the legacy path.
Otherwise a well-known fallback env var (ANTHROPIC_API_KEY and friends), so a tool still picks up the ecosystem-standard variable with no config at all.

The useful property here is that adding a more secure mode transparently wins. Drop an env reference next to an old literal key and the next run uses the env var. You can migrate a credential to a better home without first removing it from its worse one, which makes the migration safe to do incrementally instead of as one nervous big-bang edit.

The tool tells on itself

A precedence rule is no use if nobody knows their config still has a plaintext key three layers down. So the built-in doctor command grew a check for exactly that. Run doctor, and if any literal credential is sitting in your config it reports a warning, names the offending keys (the key names, never the values) and points you at how to migrate.

It’s not an error. Literal mode is still legal. But the tool will quietly keep reminding you that you left the campsite messier than you could have, until you go and tidy it. (Old Scout habits die hard, and they’ve leaked all the way into the framework.)

The gist

A CLI tool that writes your API key into a plaintext config file isn’t doing anything wrong, exactly. It’s just handing you a liability that activates later, when the file travels somewhere the key shouldn’t. go-tool-base’s answer is three storage modes: an env-var reference by default, the OS keychain on request, and a plaintext literal only as a documented last resort that CI environments can’t use at all. Runtime resolution runs in a fixed precedence so a more secure mode always wins, which makes migrating a credential safe to do gradually. And doctor keeps an eye on the config so a stray plaintext secret doesn’t get to hide forever.

The secret should live in a secret store. The config file should just know its name.

A configurable AI endpoint is an attack surface

Sun, 19 Apr 2026 00:00:00 +0000

“Let users point at their own AI endpoint” is one of those config options that looks completely harmless on the way in. People want it, for perfectly good reasons. Then you sit with it for a minute and realise you’ve handed every user a loaded gun and pointed it vaguely at their own API key.

Why you offer it at all

There are real reasons to let someone set a custom base URL. They’re running a local model and want localhost:11434. They’re behind a corporate proxy that fronts the real provider. They’re on Azure’s flavour of OpenAI, which lives at a different host. They’ve a self-hosted gateway doing rate-limiting. All reasonable, all things a framework should support rather than fight.

The bit that’s a loaded gun

Here’s what the config option quietly decides: the base URL is where your credential goes. The API key rides along in an Authorization header on every request, to whatever host that URL resolves to. So the moment the endpoint is user-configurable, the destination of your secret is user-configurable too.

And users do user things. They paste a URL from a gist that turned out to be a honeypot. They leave http:// on the front, so the key crosses the wire in plaintext. They copy https://user:token@host/v1 not realising the userinfo changes who they actually authenticate to. They never edit the https://api.example.com/v1 placeholder and wonder why the key’s been posted to a domain they don’t own. None of that is malice. It’s what happens when the destination of a secret is a free-text field.

Validate before the first byte leaves

So every chat.New routes through ValidateBaseURL before the provider is built. The threat model is written at the top of pkg/chat/baseurl.go: an operator who can influence config could “redirect chat-provider traffic to an attacker-controlled HTTPS host and capture the Authorization header.” The checks run cheapest-first: a length cap, no ASCII control characters, must parse, no userinfo, https only, a host must be present, and the host mustn’t be a placeholder.

The userinfo rule is the sharp one:

if parsed.User != nil {
	// Reject any userinfo, with or without password. Never log
	// the URL itself because it contains the credential.
	return errors.WithHint(ErrInvalidBaseURL,
		"base URL must not contain credentials; use the Token field instead")
}

The placeholder check rejects example.com and friends and any subdomain of them, so the unedited https://api.example.com/v1 from a setup wizard never reaches the wire and hits some typosquatted lookalike. And the HTTP escape hatch is test-only by construction: the AllowInsecureBaseURL field that permits plain http is tagged json:"-", so a config file physically cannot set it. This all came out of the 2026-04-17 security audit, finding M-3.

rust-tool-base enforces the same at its own boundary: validate_base_url rejects userinfo, any scheme but https (bar a test-only allow_insecure), and documentation placeholder hosts like example.com.

What it can and can’t do

It won’t stop a user who deliberately points the tool at a malicious HTTPS host they genuinely chose. If someone is set on sending their own key somewhere bad, validation can’t read their mind.

What it stops is the accidents: the plaintext slip, the userinfo confusion, the placeholder nobody changed. Those aren’t theoretical, they’re the ones that happen to careful people on ordinary days. Storing the key well is one job (where a CLI keeps it), stopping it leaking through a log is another, and this is the third side of the triangle: once you’ve stored it and stopped it leaking, make sure you don’t send it somewhere daft.

Redacting the secret you didn't know was in the string

Sat, 18 Apr 2026 00:00:00 +0000

Dammit! How did that get there?

A log line that should never have existed. Not a password I’d carelessly printed, nothing as obvious as that. An upstream API handed me back an error, and it had quoted my own bearer token inside the message, and that error went straight into the logs the way errors do. I didn’t put the secret there. The error did. And I’d never have caught it by being careful, because being careful only protects you from the secrets you know you’re handling.

The easy half of redaction

Hiding the secrets you know about is the part everyone does. You’ve got an API key field, a password flag, so you mask them at the point you print them. key=****. Done, and it feels like you’ve solved redaction, when really you’ve solved the half that was never going to bite you.

The half that bites

The secrets that escape are the ones that arrive inside strings you don’t control. An upstream service echoes your token back in a 401 body. A connection string with the password in the userinfo, https://user:pass@host, lands in a debug line. A library stringifies a whole request, headers and all, for a “helpful” trace. You cannot field-mask a secret you didn’t know was in the string, because you never watched it go in.

You can’t register a value you never had, so match the shape

This is the bit I got wrong in my own head at first. I assumed redaction meant handing it the secrets I was holding so it could watch for them. But the dangerous secrets are exactly the ones I’m not holding a copy of. So pkg/redact doesn’t keep a registry of your values at all. It knows what secrets look like.

pkg/redact/redact.go carries a set of RE2 patterns: a credential in URL userinfo, an Authorization: header sitting in free text, query-string credentials, and the well-known provider prefixes:

prefixPatterns = []*regexp.Regexp{
	regexp.MustCompile(`sk-[A-Za-z0-9_\-]{16,}`), // OpenAI / Anthropic-style
	regexp.MustCompile(`ghp_[A-Za-z0-9]{30,}`), // GitHub PAT classic
	regexp.MustCompile(`github_pat_[A-Za-z0-9_]{30,}`), // GitHub fine-grained PAT
	regexp.MustCompile(`xox[baprs]-[A-Za-z0-9-]{10,}`), // Slack
	regexp.MustCompile(`AIza[A-Za-z0-9_\-]{30,}`), // Google API key
	regexp.MustCompile(`AKIA[A-Z0-9]{16}`), // AWS access key ID
}

Run any string through redact.String and an OpenAI key, a GitHub token or an AWS access key ID gets caught wherever it’s hiding, in an error you didn’t write, in a URL, in a stack trace, because each has a recognisable shape. For the secrets that don’t announce themselves with a prefix there’s a fuzzy fallback: any opaque alphanumeric run of 41 characters or more. The 41 is chosen on purpose, to clear UUIDs (36), MD5 (32) and git SHA-1 (40) without flagging them, while accepting that a SHA-256 (64) will trip it. A deliberate, documented trade rather than a magic number.

Where it runs

At the boundary where a string leaves for somewhere you can’t reach back into. The telemetry backend runs every event argument and error message through redact.String before it emits anything (pkg/telemetry/telemetry.go), and both telemetry and HTTP logging drop the value of any header redact flags as sensitive. It doesn’t matter which code path produced the string, or whether you even wrote that path; everything goes through the same gate and gets the same scrub.

rust-tool-base’s rtb-redact crate takes the same shape-matching approach: regex patterns, the same family of well-known provider prefixes, and an is_sensitive_header check for header values.

A realistic limit

It isn’t a force field. A secret with no recognisable shape, shorter than the fallback threshold, will sail through. You cannot redact what you cannot recognise. But the leak that actually keeps happening isn’t some exotic unknown, it’s a well-known token turning up in a place you didn’t expect, and a shape-matcher sitting at the edge catches exactly that, including secrets you never told it about. Which is the one thing registering your own values could never have done. Storing the key safely is a separate job, where a CLI keeps it; this is about making sure that, having stored it, it doesn’t quietly fall out through a log.

I had the framework audited: every finding was the same shape

Fri, 17 Apr 2026 00:00:00 +0000

When a real security audit lands back in your inbox, the temptation is to read it as a shopping list of unrelated mistakes. Fix one, fix the next, tick them off, move on. I did exactly that the first time. The second time, I noticed something far more useful: the findings weren’t scattered at all. They clustered. Almost every one was the same sentence with the nouns swapped out.

Findings cluster, they don’t scatter

When you get a real security audit back, the instinct is to read it as a list of unrelated mistakes. Finding 1, unrelated to Finding 2, unrelated to Finding 3. Triage each, fix each, move on.

That’s not what the go-tool-base audits looked like once I stopped reading them as a list. The findings clustered. Strip away the specifics and almost every one was the same sentence with the nouns swapped: untrusted input reaches a powerful operation, and nothing checks it in between.

That reframe is worth more than any individual fix, because it turns “we patched some bugs” into “we know where to look next time”. A framework’s attack surface isn’t spread evenly. It’s concentrated at the boundaries: the handful of points where data from outside (a config file, a command-line flag, something typed into a TUI, an HTTP response) flows into machinery that can be made to misbehave. Audit the boundaries and you’ve audited most of the risk. Three examples make the pattern obvious.

Boundary one: a regex compiler

Somewhere in the tool, a user-supplied string gets compiled into a regular expression. A search pattern typed into the docs browser, a filter from a config file. Feeding user input to regexp.Compile feels harmless. It’s just pattern matching, after all.

It isn’t quite harmless. A regular expression is a tiny program, and some tiny programs are catastrophically slow. A pattern with the wrong kind of nested repetition can take exponential time to evaluate against a modestly hostile input. That’s the class of bug known as ReDoS. A user, or something feeding the user’s config, hands you a pathological pattern and your tool wedges, burning a whole core, on what looked for all the world like a search box.

The fix isn’t to ban user-supplied regexes. It’s to stop treating “compile this string” as free. go-tool-base routes any regex whose pattern came from outside the binary through a regexutil.CompileBounded helper. It caps the pattern length and puts a hard timeout on compilation. A pattern known at build time can still use plain regexp.MustCompile, because that isn’t a boundary, it’s a constant. The discipline only applies where the input genuinely crosses in.

Boundary two: a URL opener

The tool needs to open a URL in the user’s browser, a docs link or an OAuth flow. Under the hood that’s the OS handler: xdg-open, or open, or rundll32.

Now ask where the URL came from. If any part of it is influenced by config, by a server response, by user input, then “open this URL” has quietly become “ask the operating system to do something with an attacker-influenced string”. A file:// URL. A javascript: URL. Something with control characters smuggled into it. The browser-open was never the dangerous part. The unvalidated string was.

So go-tool-base funnels every URL-open through one package, pkg/browser, and that package is a gate. It enforces an allowlist of schemes (https, http, mailto, and nothing else), bounds the length, and rejects control characters before the OS ever sees the string. The rule that makes it stick is that nothing else is allowed to call the OS handler directly. One door, and the door has a lock. A scattered capability with no chokepoint can’t be secured; a capability that has a chokepoint can. (You’ll have spotted the “one door out” idea by now… it’s the same instinct as the single error handler, pointed at security instead of consistency.)

Boundary three: a log sink

This one’s the sneakiest, because it runs the wrong way round. The first two boundaries are about dangerous input coming in. This one is about sensitive data leaking out.

The tool handles credentials. It also logs, emits telemetry, and reports errors, and all three of those are exit boundaries: places where strings leave the process for somewhere more persistent and more public, like a log aggregator, an analytics backend, an error tracker. If a token ever ends up in a string that flows to one of those, you haven’t logged an event, you’ve published a secret.

The defence is pkg/redact. Any free-form string heading for an observability surface goes through it first, and it strips the usual suspects: credentials in URL userinfo, sensitive query parameters, Authorization headers, the well-known provider key prefixes (sk-, ghp_, AIza and friends), long opaque tokens. The places most likely to leak, command arguments and error messages in telemetry, get it applied automatically rather than relying on every caller to remember.

Same pattern as the other two. A boundary, and something standing on it checking what goes through.

The grunt work

None of these fixes is clever. There’s no exploit demo, no neat trick to show off. Bound a length. Check a scheme against an allowlist. Run a string through a redactor. The work was almost entirely in noticing the boundary existed, and then making sure everything routes through the one checked path instead of dotting raw calls all over the codebase.

That’s the actual lesson of a security audit, and it’s why the cluster reframe matters. The value wasn’t the dozen-or-so individual fixes. It was learning that the next risk will be at a boundary too, the next place untrusted input meets a powerful operation with nothing in between, and that the job is to find those points and put a single, mandatory, checked door on each.

To sum up

A security audit of a CLI framework reads like a list of unrelated bugs and isn’t one. go-tool-base’s findings nearly all reduced to the same shape: untrusted input reaching a powerful operation unchecked. A regex compiler that needed a length and time bound (regexutil.CompileBounded). A URL opener that needed a scheme allowlist and a single chokepoint (pkg/browser). Log and telemetry sinks that needed credentials redacted on the way out (pkg/redact).

The fixes were structural and dull, which is exactly right. Find your boundaries (config, flags, TUI input, network responses, log and telemetry sinks), give each one a single mandatory checked path, and you’ve spent your audit effort where the risk actually lives.

A mutex on a flag nobody writes twice

Thu, 16 Apr 2026 00:00:00 +0000

“Why is there a mutex around a boolean that only ever gets set once?”

It’s a fair question, and I’d half-asked it of myself before someone asked it of me. The answer turns out to be written, in as many words, in a code comment I’ve grown rather fond of.

The registry and its one-way latch

go-tool-base keeps a feature registry: the initialisers, sub-commands, flags and checks that each feature adds to the CLI. Features register themselves into it at startup, from init(), before main runs. Once everything’s wired, the framework calls SealRegistry() and the registry latches shut. Any Register call after that point panics, on purpose, because a sub-command or flag that turns up after the CLI has parsed its arguments is a bug I want to hear about at once, not discover three releases later.

So there’s a registrySealed bool. It starts false, SealRegistry flips it to true exactly once in normal operation, nothing flips it back outside of tests, and it’s read on every registration attempt. Written once, read many. The textbook shape of “you don’t need a lock for this.”

Except the comment disagrees, on purpose

Here is the actual declaration, in pkg/setup/registry.go:

// registryMu protects globalRegistry and registrySealed. Acquired for write
// by all Register* and Reset/Seal helpers; acquired for read by all Get*
// accessors. The mutex is required for memory visibility of registrySealed
// across goroutines, not only mutual exclusion on the maps.
var (
	registryMu sync.RWMutex
	registrySealed bool
)

That last sentence is the entire post. The mutex has an obvious day job: the registry is a clutch of maps that get appended to during registration, and concurrent appends need genuine mutual exclusion. registrySealed could have just hitched a ride on that lock and nobody would have thought twice. But the comment goes out of its way to say the lock is also required for the flag, for visibility, not only exclusion.

Why a write-once bool still needs the lock

The Go memory model makes no promise that a goroutine reading registrySealed will ever see the write SealRegistry made, unless there is a happens-before relationship between them. No synchronisation, no guarantee. A reader can sit there seeing false long after the seal happened on another goroutine, because the compiler may cache the read and the CPU may serve it from a core-local view. And a concurrent read and write of the same variable, with nothing ordering them, isn’t “probably fine”; it’s a data race, which Go defines as undefined behaviour.

“But registration is single-threaded, it’s all init().” It was, right up until we wanted the tests to run in parallel. This lock exists because of a deliberate campaign to restore t.Parallel() across the codebase after a stack of races forced us to drop it (the same campaign that retired the package-level mocking hooks). Tests build, register, seal and reset this registry from parallel goroutines. The instant that’s true, the seal check has to stay correct while racing, because the very thing it guards against is concurrency. So reads take registryMu.RLock, the write takes registryMu.Lock, and now there’s a happens-before edge: anyone who acquires the lock after SealRegistry released it is guaranteed to see true.

What the lock is actually for

It isn’t there to stop two goroutines both sealing the registry. There’s only ever the one seal. It’s there so that every reader can trust what it reads. A value written exactly once is precisely the case where you’re most tempted to skip the synchronisation, and precisely the case where skipping it can leave a reader legally staring at the stale value for good. The comment spells it out so that the next person to glance at registrySealed, think “that clearly doesn’t need a lock,” and reach for the delete key, reads the sentence first.

(There’s a sibling sealed flag in the middleware registry that follows the identical pattern, for the identical reason.)

Telemetry that asks first

Mon, 30 Mar 2026 00:00:00 +0000

Usage telemetry is genuinely useful. Knowing which commands people actually run, where the errors cluster, whether anyone ever touched the feature you spent a fortnight on… that’s the stuff that makes you a better maintainer. Wanting it is completely legitimate.

The trouble is that the usual way of getting it, on by default and quietly hoovering up everything, is a small betrayal of the people who installed your tool to get a job done. I wasn’t willing to build that, so go-tool-base’s telemetry starts from a different question.

The data you want, and the line you shouldn’t cross

If you maintain a tool, you want to know how it’s actually used. Which commands matter and which are dead weight. Where the error rate spikes. Whether anyone touched the feature you spent that fortnight on. That information makes you a better maintainer, and, to say it again, wanting it is completely legitimate.

The trouble is the standard way of getting it. Telemetry on by default. An opt-out buried three levels down in a settings file nobody reads. And once it’s running, it quietly collects far more than it ever admitted to: the arguments people passed, the paths they were working in, an IP address for good measure.

Every one of those is a small betrayal of someone who installed your tool to get a job done, not to become a data point. And the cost when users notice isn’t a slap on the wrist. It’s trust, and trust in a developer tool does not grow back quickly. A tool that surprises you once with what it was quietly collecting is a tool you uninstall and warn your colleagues about.

So go-tool-base’s telemetry started from a different question. Not “how do we collect the most data” but “how do we collect useful data without ever putting the user in a position they didn’t choose”.

Rule one: it is off until you say otherwise

The foundation is the simplest possible rule, and it’s absolute. Telemetry is never enabled by default. A freshly installed tool built on go-tool-base sends nothing. Not a heartbeat, not a ping, nothing at all.

It only starts collecting when the user makes an explicit, visible choice to let it. Three honest doors: they run telemetry enable, they say yes to a clear prompt during init, or they set TELEMETRY_ENABLED themselves. All three are deliberate acts. None of them is a pre-ticked box or a default they have to discover and then undo.

This is opt-in, and the distinction from a well-hidden opt-out is the entire point. Opt-out telemetry treats consent as something to be assumed and grudgingly reversed. Opt-in treats it as something that has to be given. Only one of those is actually consent.

Rule two: no personally identifiable information, full stop

Consent to “some telemetry” is not consent to “any telemetry”, so the second rule constrains what can ever be collected, even from a user who’s opted in.

No personally identifiable information. The framework does not record command arguments (they routinely contain paths, hostnames, the occasional secret someone’s pasted in). It does not record file contents. It does not record IP addresses.

It does need some notion of “distinct installations” for the numbers to mean anything, so it derives a machine ID from a handful of system signals and runs it through SHA-256. What leaves the machine is a hash. It tells you “this is the same install as last week” and tells you precisely nothing about whose install it is, and the hash can’t be walked backwards into the signals it came from.

The events themselves are deliberately thin. Which command ran, roughly how long it took, whether it errored. The shape of usage, not a transcript of it.

Rule three: the author picks the destination

Even with consent given and PII excluded, there’s a third question: where does the data actually go? go-tool-base doesn’t answer that for you, because it can’t. A corporate internal tool, an open-source CLI and an air-gapped utility have completely different right answers.

So the backend is the tool author’s choice. The framework ships several (a noop backend, stdout, a file, plain HTTP, and OpenTelemetry over OTLP) and supports custom ones. The noop backend matters more than it looks: it lets a tool wire up the whole telemetry surface, commands and all, while sending data precisely nowhere. A perfectly reasonable, fully supported configuration.

Pluggable backends also mean the data never has to touch any infrastructure I run. It goes where the tool’s author decides, on their terms. The framework provides the plumbing and stays well out of the destination.

And a way back out

One last thing, because it’s the part that makes the opt-in real rather than decorative. A user who opted in can opt straight back out, and the package includes a GDPR-aligned deletion path, so “stop, and remove what you have” is an actual supported request rather than a polite fiction.

Consent you can’t withdraw isn’t consent. It’s a one-way door with a friendly sign on it. The deletion path is what keeps the front door an actual door.

The bottom line

Telemetry is genuinely useful to a maintainer and genuinely dangerous to the trust of the people running the tool, and the usual implementation (on by default, opt-out buried, collecting everything) spends that trust recklessly. go-tool-base’s telemetry holds three lines: never enabled without an explicit user action, never collecting personally identifiable information even once enabled, and always sending data to a destination the tool’s author chose, up to and including nowhere. A real deletion path makes the opt-in something you can take back.

You can have your usage numbers. You just have to ask for them, the way you would for anything else that wasn’t yours to begin with.

Nobody reads the manual

Sun, 29 Mar 2026 00:00:00 +0000

Let me describe the actual lifecycle of a user meeting your CLI tool, because it’s a bit humbling. They run it. It doesn’t quite do what they expected. They run it again with --help. They get a wall of monospaced flag descriptions, skim it, don’t find the thing they wanted, and either give up or go and ask a human who already knows.

Your documentation might be magnificent. It doesn’t matter, because the user never reached it.

The manual loses on location, not quality

That’s the lifecycle, and notice exactly where it breaks. The documentation might be excellent. It might answer their precise question in full. It doesn’t matter, because it’s on a website, in another window, behind a search box, and the user is here, in the terminal, mid-task. The docs lost not on quality but on location. They simply weren’t where the work was.

go-tool-base’s answer starts with a decision about location: the documentation gets embedded into the binary itself. Your docs/ folder ships inside the tool, the same way its default config does. Wherever the tool is installed, the docs are right there alongside it, no network, no browser. That embedding is what makes everything else possible, and there are two things built on top of it.

A browser, in the terminal

The first is the docs command, and it’s not --help with extra steps. It launches a proper Terminal User Interface, built on Bubble Tea.

It has a sidebar, structured from the project’s own mkdocs.yml, so the docs are a navigable tree rather than one flat scroll. Markdown renders with real formatting through Glamour (colour, tables, lists, headings) instead of collapsing into monospaced soup. There’s live search across every page, regex included.

Compared with man and --help, the difference isn’t a nicer coat of paint. man gives you linear scrolling and grep; this gives you a structured tree, rich rendering and real search. It’s the documentation experience a modern developer expects, except it followed the tool into the terminal instead of demanding the user leave it.

A documentation assistant that won’t make things up

The second thing built on the embedded docs is the one I find genuinely transformative: docs ask.

The user doesn’t navigate anything. They just ask:

mytool docs ask "how do I point this at a self-hosted server?"

and get a direct, specific answer. Under the hood, the framework collates the tool’s embedded markdown and hands it to the configured AI provider (Claude, OpenAI, Gemini, Claude Local, any OpenAI-compatible endpoint) as the context for the question.

Now, “an AI answers questions about my tool” should immediately make you nervous, and the correct thing to be nervous about is hallucination. An AI that confidently invents a flag that doesn’t exist, or describes behaviour the tool simply doesn’t have, is worse than no assistant at all, because the user trusts it.

This is where embedding the docs pays off a second time, and it’s why I keep stressing that the corpus is closed. The model is instructed to answer only from the tool’s actual documentation, and the context it’s handed is exactly that documentation and nothing else. It isn’t drawing on a vague memory of similar tools from its training data. It’s answering from this tool’s real, shipped, version-matched docs. The corpus is small, closed and authoritative, which is the combination that keeps the answers honest. “Zero hallucination by design” isn’t a slogan about the model. It’s a property of bounding what the model is allowed to look at, which is the same instinct I leaned on with the mcp command: the safety comes from the boundary you drew, not from trusting the AI to behave itself.

There’s a nice second-order effect, too. The answer is always about the version of the tool the user actually has, because the docs were embedded into that build. No mismatch between a website documenting the latest release and the slightly older binary sitting on the user’s machine.

The upshot

Documentation usually loses to --help not on quality but on location: it’s in a browser, and the user is in the terminal. go-tool-base embeds the docs into the binary and surfaces them two ways: a docs command that’s a real TUI browser with a sidebar, rich markdown and search, and docs ask, which answers natural-language questions using the embedded docs as context.

Because that context is the tool’s own closed, shipped documentation and the model is told to use nothing else, the assistant stays grounded, and it’s always describing the exact version the user is holding. The fix for unread documentation was never to write more of it. It was to put it where the work happens and let it answer back.

The config key that quietly did nothing

Fri, 27 Mar 2026 00:00:00 +0000

I once spent the better part of an hour convinced a timeout setting was broken. I’d set it in the config file, the tool ignored it, and the code that read it looked perfectly correct. The setting was tiemout. I’d typed it wrong, and not one thing in the entire stack had thought that worth mentioning.

Config loaders are too polite

Most config loaders have the same agreeable flaw: they’ll read whatever’s in the file and quietly ignore anything they weren’t expecting. Put a key the tool doesn’t know about and it sails straight past. No error, no warning, nothing. The loader assumes you meant it, or assumes some other layer will care, and neither turns out to be true.

That politeness costs you in two directions. A key you misspelled is silently dropped, so the setting you thought you’d changed keeps running on its old value. And a key you forgot leaves the field at its zero value, which you then discover at runtime, usually at the least convenient moment, when something downstream divides by a timeout of zero. The file looked fine. It parsed fine. It was just quietly wrong, and nothing was watching for that.

The struct already knows the answer

The thing is, the program already has a complete description of what valid config looks like. It’s the struct you unmarshal into. The field names, the types, which ones matter. That description exists; it’s just not being used to check anything.

go-tool-base’s config package puts it to work. You hand it a tagged struct and it derives a schema from the tags, in pkg/config/schema.go:

// WithStructSchema derives a schema from a tagged Go struct.
// Supported tags: `config:"key" validate:"required" enum:"a,b,c" default:"value"`.
func WithStructSchema(v any) SchemaOption { ... }

So a feature’s config type carries its own rules inline:

type ServerConfig struct {
 Host string `config:"host" validate:"required"`
 Port int `config:"port" validate:"required"`
 LogMode string `config:"log_mode" enum:"text,json"`
}

There’s no second artefact to keep in sync, which is the same instinct go-tool-base leans on for structured AI output: the type is the schema, and the schema is a projection of the type, so the two can’t drift apart because there’s only one of them. Each package describes its own slice of config on its own struct, and NewSchema composes them into the schema the loaded config gets checked against.

Strict mode turns the typo into an error

Deriving the schema is half of it. The half that actually catches tiemout is this one, also from schema.go:

// WithStrictMode treats unknown keys as errors instead of warnings.
func WithStrictMode() SchemaOption { ... }

By default a key the schema doesn’t recognise is a warning: surfaced, but not fatal, which is the right call when a config file might legitimately carry extra keys for tools other than yours. Turn on strict mode and an unknown key becomes an error. tiemout isn’t in the schema, so the tool refuses to start and tells me which key it didn’t recognise, instead of shrugging and using the default for an hour while I lose my mind. The validator walks every key actually present in the file and checks it against the known set, so a typo has nowhere to hide.

What it deliberately doesn’t do

There’s one decision in here I think is worth calling out, because the obvious feature is conspicuously absent. The schema knows each field’s default value. It would be the easiest thing in the world to have validation fill in missing fields from those defaults.

It doesn’t, on purpose. Validation validates. It tells you what’s wrong and what to do about it, and it stops there. Defaults are a separate job, handled by the embedded default config that every feature ships and merges in before validation ever runs. Keeping the two apart means the validator has exactly one responsibility, and the defaults live in one place rather than being half in an embedded file and half injected by a check. A field’s default tag is there for the documentation and the error hint, not as a sneaky second source of values.

Errors you can act on

The output isn’t a bare boolean. Validation returns a result that separates the fatal from the advisory: the missing required field and the wrong type are errors that stop the tool; the unrecognised-but-harmless key is a warning that informs you without blocking. And because each problem carries the offending key by name and a hint about the fix, the message tells you what to change, in the spirit of errors that tell you what to do next.

The short version

A config loader that silently ignores keys it doesn’t recognise will, sooner or later, ignore one you meant. go-tool-base derives a validation schema straight from your tagged config struct, so there’s no separate schema to maintain, and strict mode promotes an unknown key from a quiet shrug to a real error that names the typo. It validates without injecting defaults, because defaults are the embedded config’s job and a validator with one responsibility is easier to trust. Set tiemout now and the tool tells you, which is roughly fifty-nine minutes sooner than I found out.

One variadic, and I'd already spent it

Thu, 26 Mar 2026 00:00:00 +0000

I had a constructor I was rather pleased with. Hand go-tool-base’s root command its props and as many sub-commands as you like, and off it goes. Then I needed to thread some config file paths through it, reached for the obvious “just add a parameter,” and discovered I’d already spent my one variadic with no second one going spare.

The ergonomics I’d happily bought

NewCmdRoot ends in subcommands ...*cobra.Command. That trailing ... is a small luxury: callers write NewCmdRoot(props, build, deploy, status) and never have to think about slices. Variadics are lovely for exactly this, the “and as many of these as you fancy” argument.

The parameter I couldn’t add

Then config arrived, and the root command needed to know about some extra configuration file paths. The instinct is to add a parameter. The instinct is wrong, because there’s nowhere legal to put it.

You can’t write NewCmdRoot(props, configPaths ...string, subcommands ...*cobra.Command). Go allows a function exactly one variadic, and it must be the final parameter. Two variadics results in a compile error before you’ve finished the line (assuming your IDE does compile time checks for you), and fairly so: at the call site, how would Go ever know where the strings stopped and the commands began? So the variadic I’d spent on sub-commands was spent. There wasn’t another to hand.

The choices, and the one I took

You can demote the variadic. Make it subcommands []*cobra.Command and you’re free to add configPaths []string next to it. Correct, and it breaks every existing call: NewCmdRoot(props, build, deploy) becomes NewCmdRoot(props, []string{}, []*cobra.Command{build, deploy}). Uglier at every site, to solve a problem only some callers have.

You can reach for functional options, and for plenty of go-tool-base’s constructors that is exactly what happened. But the root builder is the one everybody calls first, with the simplest signature in the codebase, and I didn’t want the common case lugging option machinery around for the sake of the rare one.

What I actually did was add a second door. From pkg/cmd/root/root.go:

// NewCmdRoot creates the root command with Props wiring and optional subcommands.
func NewCmdRoot(props *p.Props, subcommands ...*cobra.Command) *cobra.Command {
	return NewCmdRootWithConfig(props, []string{}, subcommands...)
}

func NewCmdRootWithConfig(props *p.Props, configPaths []string, subcommands ...*cobra.Command) *cobra.Command {
	// ...
}

The new argument goes in as a plain []string, sat before the variadic, which is perfectly legal: one variadic, still last. Callers who care about config use NewCmdRootWithConfig explicitly, and NewCmdRoot becomes a one-line wrapper that delegates with an empty slice, so every existing caller compiles untouched and none the wiser. Two doors into the same room, granted, but the original door is exactly where everyone left it.

What it comes down to

A trailing variadic is a slot you fill once. It buys gorgeous ergonomics for the “as many as you like” argument, and in exchange it quietly forecloses on ever appending another parameter, because the next one has nowhere to stand. Once it’s spent, new arguments come in as ordinary parameters before the variadic, and the kind thing to do for your callers is to put that behind a second constructor and let the original keep delegating.

So spend the variadic deliberately. Give it to the argument that genuinely wants to be a loose list, not the first one that happens to be plural, because you don’t get a second.

Half your users don't have eyes

Wed, 25 Mar 2026 00:00:00 +0000

Run a command in your favourite CLI tool and look at what comes back. Colour. Neatly aligned columns. A friendly little summary sentence. Lovely… if you happen to be a human with eyes.

But a good half of any tool’s users aren’t people at all. They’re scripts, CI pipelines, bits of automation. And that pretty output you’re so proud of is, to them, actively hostile.

Your tool has two audiences and only serves one

I made more or less this same point about AI assistants when I argued that your CLI is already an AI tool. The machines are users too. Here it isn’t an AI doing the calling, it’s a humble shell script, but the principle is identical.

Run a CLI command and look at what comes back. Colour. Aligned columns. A friendly summary sentence. It’s designed for a person reading a terminal, and for a person reading a terminal it’s great.

Now picture the other half of your users. A deploy script that needs to know which version is installed. A CI job that runs doctor and wants to fail the build on one specific check. A bit of automation gluing your tool to three others. None of them have eyes. They have parsers.

So what do they do with your beautiful human output? They butcher it. They grep for a keyword, awk out the third field, sed off a prefix. It works in the demo. Then someone rewords a status line, or adds a column, or the colour codes shift, and every script downstream breaks at once. Silently, too, because a broken grep returns nothing rather than an error. You changed a sentence and quietly took out somebody’s pipeline without ever knowing.

The human-readable output was never the contract. It just got used as one, because it was the only output there was.

Give the machines their own channel

The fix is not to make the human output more parseable. That’s a trap. You’d be constraining prose meant for people in order to satisfy programs, and end up serving neither of them well. The fix is to give programs their own output format, declared and stable, kept well away from the prose.

So every command built with go-tool-base gets a --output flag. Leave it alone and you get the friendly human rendering. Pass --output json and you get something a parser can actually rely on.

And not just some JSON. JSON with a fixed shape.

One envelope, every command

The temptation with JSON output is to let each command emit whatever structure happens to suit it. Don’t. A consumer scripting against five of your commands then has to learn five shapes, and “where’s the actual payload?” has a different answer every single time.

go-tool-base wraps every command’s JSON in one standard Response envelope:

{
 "status": "success",
 "command": "deploy",
 "data": {
 "environment": "production",
 "version": "1.4.0",
 "replicas": 3
 }
}

status says how it went. command says what produced it. data holds the command-specific payload, and only the payload. Every built-in command (version, doctor, update, init) emits exactly this shape. So does every command you write, because pkg/output hands you the envelope rather than letting you freelance:

format, _ := cmd.Flags().GetString("output")
w := output.NewWriter(os.Stdout, output.Format(format))

return w.Write(output.Response{
 Status: output.StatusSuccess,
 Command: "deploy",
 Data: result,
})

The consumer-side payoff is the whole point. A script can check .status without ever touching .data. It can pull .data.version and know the field is there because it’s typed, not scraped. It learns the envelope once, and every command in your tool, and every tool built on the framework, honours it. The contract is explicit, versioned, and the same everywhere, which is precisely what the abused human output never was.

The human output gets to relax

There’s a quiet second benefit, and it’s my favourite kind: the sort you get for free. Once programs have their own reliable channel, the human output is freed. It no longer has to stay accidentally parseable. You can reword a status line, add colour, restructure a table, make it genuinely nicer to read, and not break a single script, because no script is reading it any more. They’re all over on --output json, where the real contract lives.

Two audiences, two formats, each one actually suited to its reader. That’s the deal a CLI tool ought to be offering, and most of them don’t.

In short

A CLI tool that only emits human-readable output is only half-built, because half its users are programs that end up grep-ing prose and shattering the moment that prose changes. go-tool-base gives every command a --output json flag and one standard Response envelope (status, command, data) used identically by every built-in command and by anything you write through pkg/output. Machines get a stable, explicit, learn-it-once contract; humans get output that’s now free to be properly readable, because nothing fragile depends on its wording any more.

If your tool will ever be called by another program (and it will), give that program a front door. Don’t make it climb in through the window.

Lifecycle management for when your CLI grows up into a service

Tue, 24 Mar 2026 00:00:00 +0000

There’s a moment in the life of a lot of CLI tools where they stop being a CLI tool. Nobody quite decides it. It just happens. Someone needs the thing to also expose a little HTTP endpoint, or poll a queue, or run a scheduler, so it grows a serve command… and the honest command-line utility you wrote is suddenly a long-running service wearing a CLI as a hat.

And a service needs a whole pile of production plumbing that a one-shot command never did.

The command that stops being a command

go-tool-base is CLI-first. It is not CLI-only, and the reason is a pattern I’ve watched play out more times than I can count.

A tool starts its life as an honest command-line utility. It runs, it does its thing, it exits. Then someone needs it to expose a small HTTP endpoint. Or poll a queue. Or run a scheduler. So it grows a serve command, or a run command, and the moment it does, the thing that was a CLI tool is now a long-running service that happens to have a CLI bolted on the front.

And a long-running service needs a whole category of plumbing a one-shot command never did. It has to start things up in a sensible order. It has to shut them down gracefully when someone sends a SIGTERM, finishing in-flight work rather than dropping it on the floor. It has to tell an orchestrator whether it’s alive, and whether it’s ready. It has to do something sensible when one of its internal services quietly falls over at 3am.

Hand-rolled, that’s a few hundred lines of goroutine choreography, channel-wrangling and signal handling that every such tool reinvents, slightly differently and slightly wrong each time. It’s the first-afternoon problem all over again, just turning up later in the project’s life. So go-tool-base ships it: pkg/controls.

A controller and the things it controls

The model is small. A Controller manages any number of services. You register each with Register(id, opts...) and describe it with functional options: WithStart takes a StartFunc, WithStop a StopFunc. An HTTP server, a background worker, a scheduler, anything with a “begin” and an “end”.

You register your services with the controller and it owns their collective lifecycle. They share a common set of channels (errors, OS signals, health, control messages) so the whole set can react together. A SIGTERM doesn’t get caught by one service off in a corner; it reaches the controller, and the controller takes everything down in order, each StopFunc handed a context with a deadline so that one sulking service can’t wedge the whole shutdown forever.

That ordering and timeout handling is the bit nobody enjoys writing and everybody needs. Centralising it means a tool that adds a second service later inherits correct coordinated shutdown for free, rather than discovering on its first production SIGTERM that it only half shuts down.

Probes, because something is usually watching

If the service ends up in Kubernetes (and a lot of them do) the orchestrator wants to ask two different questions, and they really are different questions.

Liveness: are you alive, or are you wedged and in need of a kill? Readiness: are you alive and able to take traffic right now? A service can quite easily be live but not ready… still warming a cache, still waiting on a dependency. Conflate the two and you get yourself killed during a slow startup, or sent traffic before you can actually serve it.

controls keeps them separate. You attach a WithLiveness probe and a WithReadiness probe to a service, each just a function returning a health report, and the controller exposes them. The tool answers Kubernetes honestly, in Kubernetes’ own terms, without you hand-wiring two more HTTP handlers.

Self-healing, but only if you ask

The last piece is what happens when a service fails. A worker’s StartFunc returns an error. Health checks start failing. In a hand-rolled setup this is where you either crash the whole process or write yourself a bespoke restart loop.

controls has a supervisor that can restart a failed service for you, and the important word in that sentence is can. It’s off by default. A service is only supervised if you hand it a RestartPolicy at registration:

controls.WithRestartPolicy(controls.RestartPolicy{
 MaxRestarts: 5,
 InitialBackoff: time.Second,
 MaxBackoff: 30 * time.Second,
 HealthFailureThreshold: 3,
})

With a policy in place, the controller restarts the service if its StartFunc errors out, or if it racks up more consecutive health-check failures than the threshold allows. Restarts back off exponentially, from InitialBackoff up to a MaxBackoff ceiling, so a service that’s failing because its database is down doesn’t sit there hammering that database flat with a tight restart loop. MaxRestarts caps the attempts, because a service that’s failed five times in a row is not going to be rescued by a sixth go, and at that point honest failure beats a thrashing pretence of health.

Opt-in matters here. Automatic restarts are exactly right for a resilient daemon and exactly wrong for a tool where a failure should stop the line and get a human’s attention. The framework doesn’t make that call for you. It gives you the supervisor and lets you point it at the services that genuinely want it.

The bottom line

A surprising number of CLI tools become long-running services the day they grow a serve command, and the day they do, they need coordinated startup, graceful ordered shutdown, real liveness and readiness probes, and a considered answer to a service falling over. That’s a few hundred lines of fiddly, easy-to-get-wrong plumbing.

pkg/controls provides it: a Controller over Controllable services with shared channels and deadline-bounded graceful shutdown, separate Kubernetes-style liveness and readiness probes, and an opt-in supervisor that restarts failed services with exponential backoff and a restart ceiling. Your tool can start as a command and grow into a daemon without that growth turning into a rewrite.

CLI-first, but not stuck there.

Middleware for CLI commands, not just web servers

Tue, 24 Mar 2026 00:00:00 +0000

Every CLI tool past a certain size grows a category of logic that doesn’t really belong to any one command, and yet has to happen for loads of them. Timing. An auth check. Panic recovery, so a crash becomes a clean error instead of a stack-trace all over someone’s terminal. A log line saying the command started and how it finished.

Web frameworks sorted this out years ago. CLIs, for some reason, mostly still copy-paste it around.

The logic that belongs to no single command

That category of logic doesn’t belong to any one command, yet needs to happen for many of them. Time how long the command took. Check the user is authenticated before a command that needs it. Recover from a panic so a crash becomes a clean error rather than a stack-trace vomited across the screen. Log that the command started and how it ended.

None of that is the command’s job. The deploy command’s job is to deploy. But timing and recovery and auth still have to happen around it, and around build, and around sync.

Put that logic inside each command’s RunE and you’ve copied the same six lines into thirty functions, which means thirty places to fix when the logging format changes and thirty chances to forget one of them. Cross-cutting concerns copied by hand don’t stay consistent. They drift, every time.

Web frameworks already solved this

This is not a new problem. It’s about the oldest problem in web frameworks, and they settled on an answer a long time ago: middleware. Gin has it, Echo has it, every HTTP stack you’ve ever touched has it. A middleware is a wrapper that sits around a handler, runs its cross-cutting logic, and calls through to the handler in the middle.

A CLI command is, structurally, just a handler too. So go-tool-base brings the same pattern to the Cobra command tree, with the same functional Chain shape:

type Middleware func(
 next func(cmd *cobra.Command, args []string) error,
) func(cmd *cobra.Command, args []string) error

A middleware receives the next handler in the chain and returns a new handler that wraps it. You compose a stack of them, and each command’s real RunE runs in the middle of the onion. Write the timing logic once, as one middleware, and every command in the chain is timed. Change the log format once and all thirty commands change with it, because there was only ever one copy. (The “write it once, in a place where everyone inherits it” drum again, which I will keep banging until the series runs out.)

“But Cobra already has PreRun”

It does, and this is the objection worth answering properly, because Cobra ships PersistentPreRun and PreRun hooks and they look, at a glance, like they cover this.

They don’t, and the reason is structural. A PreRun hook is a thing that happens before the command. That’s all it is. It can’t run anything after. It can’t wrap the command in a defer. It can’t catch a panic the command throws. It can’t measure how long the command took, because measuring a duration needs a start point and an end point, and the hook only owns the start.

A middleware wraps the entire execution. Because it’s a function that calls next() in its own body, it straddles the command (with the handler signature abbreviated to HandlerFunc here for readability):

func TimingMiddleware(next HandlerFunc) HandlerFunc {
 return func(cmd *cobra.Command, args []string) error {
 start := time.Now()
 err := next(cmd, args) // the command runs here
 log.Debug("command finished", "took", time.Since(start))
 return err
 }
}

Before, after, and around. A recovery middleware can put a defer recover() in place that a PreRun hook structurally cannot. An auth middleware can check a condition and return an error instead of calling next() at all, refusing to let the command run in the first place. PreRun can’t veto the command; it runs, and then the command runs regardless.

PreRun is a notification that the command is about to happen. Middleware is control over whether and how it happens. For genuine cross-cutting concerns you need the second thing, not the first.

To sum up

Timing, auth, recovery and logging are cross-cutting concerns: necessary for many commands, owned by none. Hand-copied into every RunE, they drift out of sync. Web frameworks fixed this with middleware years ago, and a CLI command is structurally just another handler.

go-tool-base brings the functional Chain middleware pattern to the Cobra command tree. A middleware wraps a command’s whole execution, so it acts before and after and can decide whether the command runs at all… strictly more than Cobra’s PreRun hooks, which only fire beforehand and can’t wrap, recover, time, or veto. Write the concern once, wrap the chain, and every command inherits it consistently.

A logging interface that doesn't leak its backend

Mon, 23 Mar 2026 00:00:00 +0000

The same tool, in two different lives, wants two completely different kinds of log.

On my laptop I want logs I can actually read: colour, alignment, friendly timestamps. The very same tool running as a daemon in a container wants none of that. It wants structured JSON, one object a line, ready for a log aggregator to swallow. And in a test I want the logger to shut up entirely. The interesting question is what it costs you to move between the three.

The same tool wants different logs

On a developer’s machine the tool is a CLI. You want logs that are pleasant to read in a terminal: colour, alignment, human-friendly timestamps. The charmbracelet logger does that beautifully.

Then the very same tool grows a serve command and gets deployed as a daemon in a container. Now coloured terminal output is worse than useless. The log aggregator wants structured JSON, one object per line, machine-parseable. slog does that.

And in tests you want neither. You want the logger to exist, satisfy the interface, and stay completely silent.

That’s three different logging backends, wanted by one tool across three different lives. So what does switching between them actually cost?

What it costs depends on what your packages imported

If your packages import a concrete logger, if pkg/config and pkg/setup and twenty others each have import "github.com/charmbracelet/log" and take a *log.Logger, then the backend is welded into the entire codebase. Switching to JSON for the container build means editing the import and the parameter type in every single one of those packages. The backend has leaked. A detail that should have been one decision has become a property of a hundred files.

go-tool-base doesn’t let it leak. Every package in the framework accepts a logger.Logger, an interface, and nothing else. No package anywhere imports a concrete logging library. A package states, in its types, “I need something I can log through”, and stops right there. It has no idea, and no way to find out, what’s actually on the other end.

// what every package depends on
type Logger interface {
 Debug(msg string, keyvals ...any)
 Info(msg string, keyvals ...any)
 Warn(msg string, keyvals ...any)
 Error(msg string, keyvals ...any)
 // ...
}

The backend gets chosen once, at the top, when the tool builds its Props. It travels down to every package as the interface, through the Props container. The packages underneath never see the concrete type, so the concrete type can change without a single one of them noticing. (There’s that “decide it once, in one place” theme again. I did warn you it runs through everything.)

Three backends, and the swap is one line

go-tool-base ships three implementations of that interface:

charmbracelet (logger.NewCharm(w, opts...)). Coloured, styled, for humans at a terminal. The CLI default.
slog JSON, a slog-backed backend emitting structured JSON, for daemons and containers feeding a log aggregator.
noop, which does precisely nothing, for tests that want a real Logger and total silence.

Switching the tool from a friendly CLI logger to container-ready JSON is a change to the one line in main() that constructs the logger. That’s the lot. pkg/config doesn’t change. pkg/setup doesn’t change. None of the twenty packages change, because none of them ever knew which backend they had. The decision was always one line; the interface is what kept it one line.

The noop backend deserves its own mention, because it’s the one people underrate. A test for a command shouldn’t be spraying log output all over the test run, but the command still needs a non-nil Logger to function. logger.NewNoop() gives you exactly that: interface satisfied, output binned, test quiet. And because it’s just another implementation of the same interface, no test needs any special logging machinery. It passes a different backend, exactly the way the container build does.

The general shape

There’s nothing exotic going on here. It’s “depend on interfaces, not implementations”, which every Go developer has had drilled into them at some point. The bit worth holding onto is where the rule actually pays out, and it’s at the seams between a stable core and a detail you know full well you’ll want to vary.

A logging backend is exactly such a detail. You will want it different in a terminal, in a container, and in a test. So the thing your code depends on has to be the interface, and the concrete backend has to be chosen at one well-known point and nowhere else. Get that boundary right and “we need JSON logs in production” is a one-line change. Get it wrong and it’s a refactor and a bad afternoon.

What it comes down to

One tool legitimately wants three different logging backends across its life: coloured output in a terminal, structured JSON in a container, silence in a test. The cost of moving between them is decided entirely by whether your packages imported a concrete logger or an interface.

go-tool-base’s packages depend only on logger.Logger, never a backend. Three implementations ship (charmbracelet, slog JSON, noop) and the backend is chosen once, in main(), then carried everywhere as the interface through Props. Switching is one line at the top, because the detail was never allowed to leak into the hundred files below it.

Many embedded filesystems, one merged view

Sat, 21 Mar 2026 00:00:00 +0000

Go’s embed package is one of those features that makes you slightly giddy the first time you use it. One //go:embed directive and your default config, your templates, your docs are all baked into the binary. The tool just works the moment it’s installed, with nothing external to lose or forget to ship.

And then you go and build something modular on top of it, and you discover the catch nobody warned you about.

`embed.FS` is an island

An embed.FS has a property that’s easy to miss until it bites: it’s local to the package that declared it. The //go:embed directive can only see files at or below its own source file. So in any project bigger than a toy, you don’t have an embedded filesystem. You have many. The root package embeds one. Each feature, each subcommand that ships its own templates or defaults, embeds another. They’re islands, one per package, and Go gives you no native way to make them behave as a whole.

For most files that’s perfectly fine. A feature’s templates can stay on the feature’s island; nothing else needs them.

It stops being fine the moment features need to contribute to something shared.

The shared-config problem

Here’s the case that forces the issue. A go-tool-base tool has a global config.yaml of defaults, embedded at the root. Now you add a feature, and that feature has its own configuration keys, with their own sensible defaults.

Where do those defaults go?

The naive answer is: edit the root config.yaml and add the feature’s section. And that’s a genuinely bad answer, because it inverts the dependency. The root config now has to know about every feature. Add a feature, edit the centre. Remove one, edit the centre again. The central file becomes a pinch point that every feature has to reach into, and a modular architecture where you can’t add a module without editing the core isn’t really modular at all… it just has more files.

What you actually want is for the feature to ship its own slice of default config, on its own island, and for the global config the tool reads to somehow already contain it. The feature contributes; the centre doesn’t budge.

`props.Assets`: merge the islands

That’s the job of props.Assets. (Yes, it lives on Props, the load-bearing container I keep going on about. Most of the good stuff does.) It’s a layer that implements the standard fs.FS interface, and into it you Register each embed.FS under a name:

// root main.go
Assets: props.NewAssets(props.AssetMap{"root": &assets}),

// a feature's command constructor
//go:embed assets/*
var assets embed.FS

func NewCmdFeature(p *props.Props) *cobra.Command {
 p.Assets.Register("feature", &assets)
 // ...
}

Now Props carries one Assets value that represents all the islands as a single filesystem. The root’s files and every registered feature’s files, addressable through one fs.FS. Each registration is named, so the islands stay individually identifiable, but they read as one.

That alone solves the addressing problem. The genuinely clever part is what happens for structured files.

Opening a file that exists in several places

When you Open a path through props.Assets and that path has a structured extension it recognises (.yaml, .yml, .json, .toml, .csv, and a few more) it doesn’t simply return the first match it stumbles across. It does this:

Discovery. It finds every instance of that path, across every registered filesystem.
Parsing. It unmarshals each one.
Merging. It deep-merges the parsed data, using mergo.
Re-serialisation. It hands you back a single fs.File whose contents are the combined, merged result.

So picture the shared-config problem again, only solved this time. The root ships a config.yaml with the base defaults. Each feature ships a config.yaml on its own island carrying only its own keys. Nobody edits anybody else’s file. When the init command opens config.yaml through props.Assets, it doesn’t get the root’s copy. It gets the deep-merge of the root’s copy and every registered feature’s copy: one config.yaml that contains every default in the tool, assembled at runtime from contributions that never knew about each other.

A feature contributes its defaults simply by existing and registering. The centre never changes. That’s the modular property the naive approach couldn’t give you, and it generalises well beyond config… the same merge applies to a shared commands.csv, or any structured file features want to add rows or keys to.

There’s also a Mount method for attaching an arbitrary fs.FS at a virtual path, which is handy for surfacing something external (a temp directory, say) as part of the same tree. But the structured merge is the feature that really earns Assets its place.

Boiling it down

embed.FS is per-package by design, so a modular CLI ends up with many embedded filesystems, one island per feature. Most of the time that’s fine. It fails specifically when features need to contribute to a shared resource like the global config.yaml, because the naive fix forces every feature to reach in and edit a central file.

props.Assets merges all the registered islands into a single fs.FS, and for structured files it goes further: opening a .yaml, .json or .csv discovers every copy across every island, deep-merges them, and returns the combined whole. A feature drops its own defaults onto its own island, registers, and the merged config the tool reads already includes them. Contribution without coupling, which is rather the whole point of being modular in the first place.

Props: the container that does the heavy lifting

Sat, 21 Mar 2026 00:00:00 +0000

I name-dropped Props back in the introduction and then rather glossed over it, which was a bit unfair of me, because it’s the single most important design decision in the whole framework. So let’s give it the attention it actually deserves.

And the best place to start, oddly enough, is the name.

Start with the name

The container at the centre of go-tool-base is called Props, and the name is doing real work, so we’ll start there.

It is not short for “properties”, though it does hold a few. A prop is the heavy timber or steel beam that stops a structure quietly collapsing in on itself. And for anyone who follows the rugby: a prop is the position in the scrum, the broad-shouldered forward whose entire job is to provide structural support so everyone else can get on with the game.

That’s the design brief, in a single word. Props is not where the clever, flashy work happens. It scores no tries. It’s the thankless, load-bearing thing that holds the framework up so that your actual command logic gets to be the interesting part. Understand the name and you understand what the struct is for.

What it carries

Props is the single object passed to every command constructor in a go-tool-base tool. It holds the dependencies a command might need:

Tool, metadata about the CLI (name, summary, release source).
Logger, the logging abstraction.
Config, the loaded configuration container.
FS, a filesystem abstraction (afero), so a command never touches the real disk directly.
Assets, the embedded-resource manager.
Version, build information.
ErrorHandler, the centralised error reporter.
Collector, the telemetry collector (always present, a no-op when telemetry is off).

A command constructor’s signature is, accordingly, boring on purpose:

func NewCmdExample(p *props.Props) *cobra.Command { ... }

One parameter. Everything the command could possibly need is reachable through it. No globals, no init()-time wiring, no twelve-argument constructor that quietly grows a thirteenth argument next month.

Why a struct, and not `context.Context`

Here’s the design decision I actually want to defend, because it’s the one Go developers tend to raise an eyebrow at. Go already has a well-known way to carry things through a call tree: context.Context. So why not just put the logger and the config in the context and pass that around?

Because context.Context carries its values as interface{}, and that’s the wrong trade for dependencies.

Pull a dependency out of a context and you get this:

l := ctx.Value("logger").(logger.Logger) // a runtime type assertion

That one line has two separate ways to hurt you. The key is a bare string, so a typo compiles perfectly happily and then fails at runtime. The type assertion is unchecked, so if the wrong thing is sitting under that key, your tool panics in front of a user. Neither failure is visible to the compiler. Neither is visible to your IDE. You find out when it breaks, which is to say at the worst possible time.

Pull the same dependency out of Props and you get this:

p.Logger.Info("starting") // a field access

p.Logger is a typed field. If it doesn’t exist, or you’ve used it wrong, the code simply doesn’t compile. Your IDE autocompletes it. Refactor the Logger interface and every misuse lights up at build time. There’s no runtime type assertion, because there’s no interface{} to assert from in the first place.

context.Context is the right tool for what it was designed for: cancellation, deadlines, request-scoped signals that genuinely cross API boundaries. It’s the wrong tool for “here are my program’s services”, because it trades away the compiler’s help for a flexibility you really don’t want here. Dependencies should be declared, somewhere the compiler checks them. Props is that somewhere.

What you get back for it

That one decision pays out in three currencies.

Testability. A command is now a pure function of its Props. To test it, you build a Props with the doubles you want (an in-memory FS instead of the real disk, a no-op Logger, a config you’ve populated by hand) and call the constructor. No global state to reset between tests, no monkey-patching, no init() order to puzzle over. The dependency is an argument, so the test just passes a different one.

Consistency. Cross-cutting changes have exactly one place to happen. When the global --debug flag flips the log level, it does so on the Logger inside Props, and because every command reads its logger from the same Props, every command gets the new level. No command can drift, because none of them owns its own copy.

Extensibility. Adding a new framework-wide service is just adding a field to one struct. Every command can immediately reach it; none of them needed touching to make it reachable.

To sum up

Props is the dependency-injection container at the heart of go-tool-base: one struct, passed to every command, holding the logger, config, filesystem, assets, error handler and tool metadata. It’s a concrete struct rather than a context.Context payload entirely on purpose, because dependencies belong somewhere the compiler can check them, not behind a string key and a hopeful runtime type assertion. That single choice buys you testability, consistency and easy extension.

The name says it best, really. Props doesn’t score the tries. It’s the broad-shouldered thing in the scrum that stops the whole framework folding, so the rest of your code is free to go and play.

Design your whole CLI in one file

Fri, 20 Mar 2026 00:00:00 +0000

Here’s a question that sounds trivial and really isn’t: where, exactly, does a CLI tool’s structure live? Not the logic of each command… the structure. Which commands exist, what they’re called, which flags they take, what’s nested under what.

I’d never properly thought to ask it until go-tool-base forced me to, and the answer turned out to be a little bit embarrassing.

Where does a CLI’s structure actually live?

Picture a CLI tool with twenty commands, some nested under others. In a typical project, where does its structure live? The answer is “smeared across the codebase”. It’s in twenty cmd.go files. It’s in the AddCommand calls that stitch them together. It’s in the flag registrations. To understand the shape of the tool you have to read all of it and assemble the picture in your head, because the picture exists nowhere as a single thing you can point at.

That’s a strange state of affairs for the single most important design fact about a CLI. The command tree is the tool’s interface, it’s the thing users actually touch, and yet it hasn’t got a home.

The manifest gives it one

go-tool-base’s generator gives that structure a home: .gtb/manifest.yaml. The manifest is a single readable file describing the command tree. Every command, its name, its short description, its flags, its place in the hierarchy, whether it carries assets or an initialiser. The shape of the whole tool, in one place you can open and read top to bottom.

And the manifest isn’t documentation about the project. It’s the thing the project’s wiring is generated from. When you run regenerate project, the generator reads the manifest and rebuilds the boilerplate to match it: the command registration, the AddCommand wiring, the flag definitions. The manifest is the source of truth, and the Go wiring is its output.

Design-first, when you want it

This unlocks a way of working that the smeared-across-the-codebase approach simply can’t offer. You can design the interface first, in the manifest, and let the code follow.

Want to rename a command? Edit one line in the manifest, run regenerate, and the rename propagates through every wiring file that ever mentioned it. Want to move a subcommand under a different parent? Change its place in the manifest hierarchy and regenerate. Want to add a flag to three related commands? Add it in the manifest, in three obvious places, and regenerate, instead of going on a little hunting expedition for three flag-registration blocks scattered across the tree.

You’re editing the tool’s interface as a design, in the file whose entire job is to hold that design, and the generator does the mechanical donkey-work of making the code reflect it. The thing you change is the thing that describes the structure. The code is downstream.

If that shape sounds familiar, it should. It’s the same instinct behind spec-driven and test-driven development: write down what the thing should be before you assemble how it works, and keep that statement of intent as a first-class, living artefact rather than a comment that quietly rots in a corner. The manifest is a spec for your command tree, and regenerate is what keeps the implementation honest to it.

It doesn’t trap you

There’s an obvious worry about any generated-from-a-manifest system: am I now locked into editing the manifest? What if I just want to open a Go file and write some Go like a normal person?

You can. The generator is careful not to own everything. It owns the wiring (the registration and the structural boilerplate) and it leaves your command logic well alone. The RunE function where your command actually does its work is yours; the manifest hasn’t got an opinion about it. And the generator tracks the files it produces by content hash, so if you do hand-edit something it generated, regeneration notices and asks before overwriting rather than steamrolling you. That mechanism turned out interesting enough to get its own post.

So the manifest is an option, not a cage. Design-first via the manifest when that suits the change. Drop into Go directly when that suits it better. The two stay in sync because regeneration reconciles them, not because one of them has been forbidden.

Pulling it together

A CLI’s command tree is its most important design surface, and in most projects it has no single home… it gets reconstructed in your head from twenty scattered files every time you need to reason about it. go-tool-base gives it one: .gtb/manifest.yaml, a readable description of the whole tree that the generator rebuilds the wiring code from. Edit the manifest, run regenerate, and the boilerplate follows.

It makes CLI structure something you design in one place, in the spirit of spec-driven development, while still leaving you free to write Go directly when that’s the better tool for the job. The manifest is the spec for your interface. The generator just keeps the code faithful to it.

Scaffolding that respects your edits

Fri, 20 Mar 2026 00:00:00 +0000

When I introduced go-tool-base I made a passing promise to come back to “the generator that won’t clobber your edits”. This is me keeping it, partly because it’s the feature I’m quietly most proud of, and partly because it took the most head-scratching of anything to get right.

The problem it solves is one that every code generator runs into eventually, usually the hard way and usually at the worst possible moment.

The generator’s awkward second act

A project generator has an easy first act. gtb generate project, and you’ve got a complete, wired, idiomatic Go CLI project. Everyone’s happy, me included.

The second act is the hard one. The framework moves on. A convention changes, a new built-in capability appears, the recommended CI shape shifts. Your project, scaffolded three months ago, is now subtly out of date, and you’d quite like the generator to drag it back up to spec.

Except by now it isn’t a fresh scaffold. It’s your project. You tuned the CI workflow. You rewrote the justfile. You added a stanza to the Dockerfile that took an afternoon and a fair bit of swearing to get right. The generated files and your edited files are one and the same files.

A naive generator handles this with breathtaking confidence: it regenerates everything from the template and overwrites the lot. Run it once, lose your afternoon. You learn that lesson exactly once and then never run regeneration again, which means the upkeep feature you were sold is dead on arrival. A scaffold you can’t safely re-run is just a one-shot cp with extra steps.

What the generator needs to know

The thing standing between “safe to overwrite” and “absolutely do not” is a single fact: has this file changed since the generator last wrote it?

If it hasn’t, the file is still pristine boilerplate and the generator owns it. Overwrite away. If it has, a human has been in there, and the generator must not touch it without asking first.

The generator can’t just eyeball that, of course. It needs a record. So every time gtb generate writes a file, it computes a SHA-256 of the content and stores it in the project’s manifest, .gtb/manifest.yaml, as a Hashes map of relative path to hash. The manifest is the generator’s memory of the exact bytes it last produced.

Regeneration becomes a three-way decision

With that record in hand, regeneration stops being “overwrite everything” and becomes a per-file decision with three branches.

The file doesn’t exist. Easy. Write it, store its hash.

The file exists and its current hash matches the manifest. It’s byte-for-byte what the generator last wrote, so nobody has touched it. The generator owns it outright, regenerates from the template and updates the stored hash. No prompt, no fuss. This is the common case, and it’s silent precisely because it’s safe.

The file exists and its hash does not match. Someone has been in there since generation. The generator stops and asks. It will not silently overwrite your hard-won afternoon. You decide: take the new version, or keep yours.

The detail I’m genuinely fond of is what happens when you decline. Declining is non-fatal. Generation carries on with the rest of the files, and the manifest keeps the file’s stored hash rather than dropping it. That matters more than it looks, because it means the file stays tracked. Next time you regenerate, the generator can still tell that file has been modified, and still asks. Skipping a file once doesn’t quietly evict it from the generator’s awareness forever. It stays a known, watched, customised file across every future run.

When you want it to stop asking

Per-file prompting is the right default, but for files you’ve permanently taken ownership of, being asked on every single regeneration is just noise. If you’ve rewritten the CI workflows wholesale and you are never, ever going back to the generated version, you don’t want a prompt. You want the generator to leave them well alone and not bring it up again.

That’s what .gtb/ignore is for. It sits next to the manifest and takes gitignore-style patterns:

# I own the CI workflows now
.github/workflows/**

# ...except the release workflow, keep that managed
!.github/workflows/release.yml

# and my build config
justfile
Dockerfile

Anything matching is skipped during regeneration with no prompt at all. Patterns evaluate top to bottom and later ones win, so the negation (!) behaves the way you’d expect from .gitignore: exclude a whole directory, then claw one file back.

It’s a deliberate escalation ladder. Unmodified files are handled silently. Modified files get a prompt. Files you’ve formally claimed get total silence. Each rung asks for less of your attention than the last, and you choose how far up to climb, file by file.

Stepping back

A generator earns its keep twice: once when it scaffolds your project, and then continuously, every time it drags that project back up to the framework’s current shape. The second job is worth nothing if regeneration flattens your customisations, because you’ll simply stop running it, and who could blame you.

go-tool-base’s generator gets around that by remembering. It hashes every file it writes into .gtb/manifest.yaml, and on regeneration it re-hashes before overwriting: unchanged files it owns and updates silently, changed files it stops and asks about, and .gtb/ignore lets you mark files as permanently yours. Skipped files stay tracked, so the generator never loses sight of what you’ve made your own.

The point of a scaffold isn’t the first five minutes. It’s that you can still run it in month three without holding your breath.

go-tool-base: I got tired of reinventing the wheel

Wed, 18 Mar 2026 00:00:00 +0000

If you’ve written more than two or three command-line tools in Go, you’ll recognise the shape of the first afternoon. I certainly do! You reach for Cobra for the command tree, Viper for config, and then you start the part nobody ever puts in the README… the plumbing.

Where does config live? A file, an env var, an embedded default? In what order do they override each other? How does the tool tell the user there’s a newer version, and how does it actually update itself? What does logging look like, and is it the same logging the next tool will use? And how do you wire all of that into each command without every command reaching into a pile of globals?

None of it is hard. That’s the problem! It’s not hard, it’s just there, every single time, and every single time I’d find myself reinventing it slightly differently to the last time. Different override precedence here. A subtly different update flow there. Logging that didn’t quite match the tool I’d written three months earlier. Each new tool was a fresh re-litigation of decisions I’d already made and then promptly forgotten.

Now, I’ve banged on about the Boy Scout rule for years (leave the codebase better than you found it), but it has an uncomfortable corollary. If you keep turning up to the same campsite and finding it in the same mess, at some point the honest thing to do is to stop tidying it and go and build a better campsite.

First, just packages

So I started pulling the recurring pieces out into their own packages. Nothing grand. A config package that did the hierarchical merge the way I always ended up doing it anyway. A version package that knew how to compare semver and spot a development build. A setup package that handled first-run bootstrap and self-updating from a release. They lived as separate repos, and if you go digging through my GitHub history you can still find the scruffy ancestors of them scattered about.

Separate packages was the right first move. It forced each piece to stand on its own and earn its keep on a real project before I trusted it on the next one. A package that’s only ever been used in the repo it was born in hasn’t really been tested… it’s just been agreed with.

But separate packages come with a tax. Each one has its own release cadence, its own changelog, its own CI. Worse, they have to agree with each other at the seams, and when they’re versioned independently those seams drift. I’d bump the config package, and the setup package that depended on it would quietly need a matching bump, and the tool that used both would need telling about both. I’d traded “reinvent the wheel” for “keep a dozen wheels in sync”, and I’m really not convinced that’s a better deal.

Then, one library

Once the packages had been used enough (used in anger, on real tools, by people who weren’t me) the shape of them stopped moving. The interfaces settled. The arguments about precedence and defaults were over, because the answers had survived contact with reality.

That’s the point where separate packages stop being a virtue and start being friction. So I forged them into one and called it go-tool-base. One module, one version number, one changelog, and one set of seams that are now internal and can’t drift, because they ship together.

The heart of it is a dependency-injection container, a Props struct, that holds the things every command needs: the logger, the config, the embedded assets, the filesystem handle, the error handler, the tool’s own metadata. Commands are handed Props explicitly rather than reaching for globals, which means a command is just a function of its inputs and is therefore trivially testable. That one decision has quietly paid for itself on every tool I’ve built since.

Around that container sits all the stuff I was so tired of rewriting: hierarchical config, structured logging, version checking, self-update from GitHub or GitLab releases, an interactive TUI documentation browser, AI integration, service lifecycle management. A new tool inherits the lot and gets to spend its first afternoon on the thing that’s actually novel… its own logic.

Finally, a generator

A library still leaves you staring at a blank main.go. You still have to know the conventions, wire the container, lay out the directories, register the commands. All knowable, but all boilerplate. And boilerplate is exactly the enemy I set out to kill in the first place.

So go-tool-base ships a generator. gtb generate project scaffolds a complete, working, idiomatic project: directory layout, the wired Props container, the command tree, CI, the whole lot. gtb generate command adds a new command and registers it for you. The generator also handles upkeep: when the framework’s conventions move, it can regenerate the scaffolding of an existing project without trampling all over the code you’ve written on top. (That last bit turned out to be a properly interesting problem in its own right, and a future post.)

The goal is blunt. Creating a CLI tool should be about the tool, not the scaffolding. The first afternoon should be spent on the part that’s actually worth writing.

One thing I was careful about

There’s a nasty failure mode with “batteries-included” frameworks: the day you outgrow them, they hold you hostage. You either stay inside the framework’s worldview forever, or you face a rewrite. I’ve been burned by that before and I had no intention of inflicting it on anyone else.

So go-tool-base generates idiomatic, standard-library-compliant Go. There’s no magic runtime you can’t see, no clever code you couldn’t have written by hand. If you ever outgrow the framework the generated code stands on its own and you walk away with a perfectly normal Go project. A framework should be a starting point you’re glad you took, not a room you can’t get out of.

Where this leaves me

go-tool-base exists because I was spending the first afternoon of every Go CLI tool rebuilding the same plumbing, and rebuilding it slightly wrong relative to last time. It started life as separate packages so each piece could earn its place on real projects; once they’d stopped moving I forged them into a single library so the seams couldn’t drift; and then I wrapped a generator around it so a new tool starts as a working project rather than a blank file.

It’s a framework for the boring 80% (config, versioning, updates, logging, lifecycle) so you can spend your time on the 20% that’s actually yours.

Over the coming posts I’ll dig into the individual pieces… the generator that won’t clobber your edits, the credential handling, the self-update integrity checks, and a few Go techniques I’m rather pleased with along the way. Stay tuned!

Go-Tool-Base on PHP Boy Scout

I filed a feature request into my own framework

The bridge I was about to vendor

A feature request, against myself

Two sessions, one dependency

The one rule that came with it

The truest test of a framework

A stack trace is not an error message

Two different things we both call “the error”

Pick a reader per level

Where the stack belongs

The agent said SUCCESS. The linter disagreed.

What “done” meant to the agent

The instruction that made lint optional

The fix was words, not code

Leave it better, not just building

The cobra hook I was sure I'd enabled

What the start-up hook is for

“Persistent” promises less than it says

A landmine nobody had stepped on

One line, and a note for whoever’s next

Trust, but grep

Sign your own binaries with go-tool-base, part 7: rotation and break-glass

Why there’s no auto-rotate button

Rotate by minting a new key, never by changing the old one

The dual-publish, dual-sign window

The break-glass key, for when there’s no handover

The whole chain, end to end

Sign your own binaries with go-tool-base, part 6: sign every release with GoReleaser

One command wires the signing in

Why only the checksums

No shim, just gtb

Where the credentials come from

Don’t sign when there’s nothing to sign with

Cut a release

A two-person gate, if you want one

Where this leaves you

Sign your own binaries with go-tool-base, part 5: embed the key and require verification

Enable signing with one command

Drop your key in

The WKD cross-check, via --email

Confirm the cross-check is actually firing

Require it, in the right order

Where this leaves you

Sign your own binaries with go-tool-base, part 4: mint and publish your public key

Mint the public key from KMS

Mint the rotation authority while you’re here

Build the WKD tree

Publish it somewhere the platform can’t reach

Where this leaves you

Sign your own binaries with go-tool-base, part 3: keyless CI signing with OIDC

Where the identity provider comes from

The GitLab path

The GitHub path

The gotcha that wastes an afternoon: token audience

Where this leaves you

Sign your own binaries with go-tool-base, part 2: a signing key in AWS KMS

What you’ll need first

The module

What you just built

Four principals, one key policy

The outputs you’ll carry forward

What this costs

Where this leaves you

When you hand the same key to every call

A bug with a two-month alibi

What the key actually was

Changing my mind about cobra

Changing my mind about the tests

What it comes down to

Sign your own binaries with go-tool-base, part 1: sign and verify on your laptop

Make a signing key

Mint the public key you’ll actually publish

Sign something

Verify it, two ways

Where this leaves you

Sign your own binaries with go-tool-base

What you’re protecting against

Two paths through it

Before you start

No shim, just `gtb`

The WKD cross-check, via `--email`

From a script: `csv_stats.py` becomes `csv-stats`

A page from `html/template`