PHP Boy Scout

Why I still write code

Wed, 10 Jun 2026 00:00:00 +0000

By any sensible reading of an org chart, I have no business being in this file. I’m a Head of Software Engineering. My calendar reckons I should be in a room somewhere talking about headcount and roadmaps. Instead it’s late, everyone sensible has logged off, and I’m three retries deep into a release that refuses to tag itself, muttering at a Rust workspace I built with my own hands.

So why am I here? I’ve been asking myself a version of that question for about twenty-five years, and I think I’ve finally got an answer. It’s just not a flattering one.

I’m a builder, and that isn’t really a choice

Strip away the job titles and I’m a builder. I like to make things, I like to solve problems, I like to learn how something works by taking it apart and putting it back together slightly differently. That urge predates every role I’ve ever held and it has survived all of them. In jobs where I wasn’t allowed to scratch it, I went and built in the open instead, which is a polite way of saying open source has spent years absorbing energy my day job wouldn’t take.

I’ll go further, because being coy about it helps no one: it’s closer to an addiction than a hobby. I don’t fully switch off. The current outlet, when I’m not in a terminal, is converting a campervan, which is just software engineering with worse error messages and a real risk of electrocution. The shape of the thing changes. The compulsion doesn’t.

Underneath the building there’s a less charming engine, and I might as well name it: a fairly grim case of impostor syndrome. I wrote about it years ago when I stopped calling myself “Dev in Charge”, and a decade on it hasn’t gone anywhere. The only thing that ever quiets the anxiety is staying genuinely good at the thing, and staying good at the thing means using it. I’m a firm believer in use it or lose it. People say technical skill is like riding a bike, you never forget. Maybe. But step away for a few years and when you climb back on, someone’s bolted a jet engine to the frame and moved the pedals. The bike doesn’t wait for you.

What it actually buys

Here’s the part that justifies the indulgence, because on its own “I enjoy it” isn’t a reason to stay technical as a leader, it’s a reason to have a hobby.

The load-bearing belief is simple, and it’s the one line I’d carve into the desk: I will never ask an engineer to do something I’m not willing to do myself. Everything good about staying hands-on flows from that. Because I’m still in the work, I can give my engineers proper support, the right tools and a clear path, rather than guessing at what they need from a slide. I can steer them through a genuinely hard technical call instead of nodding along. I can sniff out a duff estimate, mine or theirs, because I know what the work actually costs. And I can hold them to account with a straight face, because the accountability runs both ways. They answer to me for what they ship, and they get to hold me to account for what I contribute. That second half is the bit a lot of technical leaders quietly drop, and it’s the half that earns you the right to the first.

The bill, and who paid it

I’d be selling you a fairy tale if I stopped there, so here’s the cost, and some of it is steep.

The obvious one is burnout. I’ve been there more than once over the years, and it’s the single biggest reason I now pitch myself deliberately as a Technical Leader rather than an Engineering Manager. I can do the manager stuff, the HR and the planning and the project-management bollocks, and after enough years in the role I do it well, because it demanded that I did. But competence isn’t appetite. Given the choice I’ll take a technical problem or a bit of mentoring over running the process around either, every time, and spending your days on work you’re good at but don’t much enjoy is its own slow road back to the wall. Sticking to my strengths isn’t ego, and it isn’t an admission I can’t do the rest. It’s self-preservation, learned the hard way.

The steeper bill came due at home. When my kids were small I poured my own time into pushing my skills and chasing the next rung, even starting my own agency. Between that and the burnout, I missed big chunks of their early years, and that is one of the real regrets of my life. I’m not going to dress it up or hide it behind a lesson. It was my decision, I made it, and I own it. I’m immensely proud of the people they’ve grown into, and since their mum and I separated I’ve put everything I have into giving them a stable home, the builder instinct quietly turning into a nest-building one, which is the better use of it. I put this here, plainly, because if you’re reading this with a young family asleep upstairs, I’d sooner you heard it from someone who got the balance wrong than learn it the way I did. The code will still be there next year. They won’t be five next year.

And there’s a smaller, daily cost that I still haven’t fully mastered: knowing when to put the keyboard down. A builder who can’t stop building is exactly the person who becomes the bottleneck, disappears down a rabbit hole, or hoards the interesting problem that would have stretched someone on the team. Stepping back to let them solve it, when every instinct I have is screaming to just fix the bloody thing, is genuinely one of the hardest skills I’ve had to learn, and some days it still feels like walking a knife edge. Open source is a big part of how I manage that. It’s a release valve, somewhere I can let the compulsion run with no brakes on, precisely so I’m not stealing the meaty work off the people I’m meant to be growing.

Does it still count when the robot types?

Fair challenge, given the year. I build solo now with an AI pair, to the point where it’s changed how I branch and release. So when a model writes a good chunk of the actual characters, am I still “writing the code”?

I think I’m doing it more than ever, and I’m certainly learning faster. My typing is genuinely terrible, a quarter-century of practice and still mostly thumbs, so being freed from being the typist is no loss at all. What’s left when you take the keystrokes away is the part that was always the point: reading, reviewing, judging, steering. I can review more code, faster, than I ever could when I was the one hammering it out, and I can run several projects at once by pointing my judgement at each in turn. That is leadership work and engineering work at the same time, which is rather the whole thesis.

It did not come free, mind. I was elbow-deep in AI and ML long before GPT made it fashionable, and I’ve seen the messy version up close. Getting to the point where the tools are good enough and I’ve built the guardrails and habits that make them safe took a long time and a lot of getting it wrong. Owning the judgement when the machine does the typing is harder than it sounds, not easier. The typing was never the hard bit.

What I’d actually put my name to

Not that every leader should write code. Plenty of excellent ones don’t, and they’re brilliant at the parts of the job I’m middling at. The narrower, truer claim is the only one worth making: I lead better when I stay in the work, because it’s the only way I know to support, steer and be held to account without faking any of it, and because I meant that line about never asking for what I won’t do myself.

Staying technical isn’t the job. It’s the thing that lets me do the job honestly. I’m a builder who learned, slowly and at a price I’d rather have not paid, how to keep building without it costing the people around me what it once cost the people closest to me. That’s the balance I’m still working at. I suspect I always will be.

A signature the platform can't forge

Tue, 09 Jun 2026 00:00:00 +0000

A self-updating tool has a chicken-and-egg problem baked into it. The thing doing the updating is the thing being updated, so when it reaches out and pulls down a newer version of itself, it’s the one that has to decide whether to trust what just landed. No human in the loop, nobody to ask. I’ve been closing that gap in go-tool-base’s self-updater in two phases. The first gave it a checksum: download the new binary, hash it, compare it against the release’s checksums.txt. That catches the accidents, the truncated download, the flipped bit on a dodgy mirror. And I said at the time, plainly, that it does nothing about a determined attacker who owns the release platform… the checksums file sits right next to the binary, so whoever can swap one can swap both. I left that as an IOU. This second phase is me paying it.

The thing a checksum can’t do

A checksum is a promise that the bytes you got match the manifest. It says nothing about who wrote the manifest. So if GitLab, or my account, or a leaked CI token gets compromised, the attacker rewrites the binary and the checksums.txt in the same breath, and the hash matches perfectly, because they’re the one who computed it. It’s the same wall I keep walking into whenever I think about supply-chain trust: a checksum is only ever as good as whatever’s standing behind it, and the thing standing behind a checksum is the very platform that just handed you the file. Same hands, both times.

To get past that, you need a signature whose root of trust lives somewhere the platform can’t reach.

The crypto is the easy part

Here’s the bit that caught me slightly off guard while I was building this: the cryptography is the easy part. Verifying a detached OpenPGP signature is a library call, and go-tool-base’s TrustSet wraps it up in one method:

func (t *TrustSet) VerifyManifestSignature(manifest, signature []byte) error {
	// ...
	signer, err := openpgp.CheckArmoredDetachedSignature(
		t.entities, bytes.NewReader(manifest), bytes.NewReader(signature), nil)
	if err != nil {
		return errors.Wrap(ErrSignatureInvalid, err.Error())
	}
	if signer == nil {
		return errors.Wrap(ErrSignatureInvalid, "no signer in trust set matched")
	}
	return nil
}

Hand it the manifest, the detached signature, and a set of trusted public keys (the entities), and it tells you whether any one of them signed it. That’s the whole of the cryptography, and it’s genuinely not where the hard work lives.

The hard work is that set of trusted public keys. Where do they come from? Because if the answer is “we ship them right next to the binary”, well… you’re straight back to the checksum problem. Whoever can swap the binary can swap the key too, sign with their own, and the check waves it through none the wiser.

Pulling the two questions apart

So the design splits along exactly that seam. The verification half is fixed, and deliberately boring (the method above). The trust anchor, the actual keys, comes from a swappable KeyResolver:

// The interface separates "where the trust anchor comes from" from "how a
// signature is verified against it", so SelfUpdater can be wired with
// whichever resolver chain a tool needs without changing verification logic.
type KeyResolver interface {
	Name() string
	Resolve(ctx context.Context) (*TrustSet, error)
}

That little seam is really the whole game. Everything interesting about standing up to a compromised platform comes down to which resolver you hand the updater, and the verification code never has to know the difference.

Three answers to “where does the key live”

The first option is to embed it. Bake the public key straight into the binary at build time (NewEmbeddedResolver), so it rides along inside a release you already trusted enough to run. Tidy and self-contained. The catch is that a future malicious release could embed a different key, so on its own, embedding really just trusts whoever cut the most recent binary.

The second is WKD, the Web Key Directory. Fetch the key over HTTPS from a well-known path on a domain you control (NewWKDResolver), nothing to do with where the release itself is hosted. Now the key isn’t in the binary at all, so poisoning a release doesn’t touch it. You haven’t made the problem disappear, mind… you’ve moved the trust onto your domain’s host and its DNS. A different blast radius, but a blast radius all the same.

The third option is to do both, and make them agree. Run embedded and WKD, and insist they agree:

func (c *CompositeResolver) Resolve(ctx context.Context) (*TrustSet, error) {
	// ... run each child resolver concurrently ...
	if err := checkAgreement(successes); err != nil {
		return nil, err // ErrKeyResolverMismatch
	}
	return successes[0].ts, nil
}

Think of it as the two-key rule on a safe deposit box, or two witnesses who’ve never met telling you the same story. One source on its own you might quietly doubt. But if the key baked into the binary and the key sitting on my domain hand back the same fingerprint, that agreement is worth a great deal more than either of them alone. And if they ever come back different, that’s not a maybe, that’s an alarm: ErrKeyResolverMismatch. Poison one source and the mismatch is the thing that gives the game away.

That composite is the real answer, and it’s why the interface exists at all. There’s nothing a single attacker can get their hands on that holds the whole thing up by itself. The key is baked into a release you trusted, and fetched from a domain well off the release platform, and the two have to match before a single byte of the update is allowed through.

The separation is the whole point

It’s easy to nod along at “two sources” and miss the part that actually does the work. The agreement between the embedded key and the WKD key is only worth something if an attacker can’t reach both of them from the same place. If the key I bake into the binary and the key I serve over WKD both came out of the same release pipeline, whoever owns that pipeline swaps the pair of them, the fingerprints still match, and the cross-check happily waves the forgery through. Same hands, both times. Again.

So they don’t share a pipeline, and that’s the entire design, not an accident of how things ended up. The binary, and the key embedded in it, are built and signed in GitLab CI, which federates into AWS KMS to do the signing itself. The WKD key lives somewhere else completely: a Cloudflare Pages site serving openpgpkey.phpboyscout.uk, deployed by hand at rotation time with the Wrangler CLI and a token allowed to do nothing but edit that one Pages project. No Git integration, no webhook, nothing that lets a push to the repo or a run of the release pipeline so much as touch it. The Cloudflare account is even administered under a different email and a different second factor from the GitLab and AWS ones, so the three anchors really are independent rather than just feeling that way.

Which is what makes them fail independently, and that independence is the only thing that makes the agreement worth checking. To forge a release that survives the cross-check, an attacker doesn’t have to beat one system, they have to beat two unrelated ones, on different platforms, behind different credentials, in the same window, without either of them noticing.

There’s a quieter benefit in the cadence, too. Releases go out constantly and automatically; the WKD key changes rarely, and only ever by hand. So the busy, automated path, the one an attacker is most likely to prise open, is exactly the one with no power to rewrite the key everyone checks against.

Requiring it, without breaking everyone

Now, a check nobody ever switches on is just theatre. But switch it on before the keys are actually out there in people’s installs, and you’ve handed everyone a self-inflicted outage instead. So the default is deliberately timid. The framework ships DefaultRequireSignature = false: a tool built on go-tool-base doesn’t suddenly start rejecting its own updates the day its author bumps the framework version.

The tool author flips it to true in main(), but only after they’ve shipped a release that embeds the key, so every install out there already holds the trust anchor before the first release that insists on one. Ship the key, then turn the lock: the same leave-yourself-a-way-back discipline as any migration you’d like to still have a job after. And the end user still gets an override (update.require_signature, or an env var) for the day it all goes sideways and they need out.

What it actually buys

The first phase stopped accidents. This one stops the platform. And not because the cryptography is clever, OpenPGP checks the signature in a single call, but because the trust anchor is arranged so that nothing the attacker can actually reach holds the whole thing up on its own. A signature only ever proves the sender, never the contents. All of this is really about making “the sender” something a compromised release host can’t quietly fake its way into being.

Which leaves one last thread dangling. The verifying key gets fetched from somewhere, fine… but the signing key, the private half that actually produces these signatures, has to live somewhere the platform can’t reach either, or none of the rest holds up. That’s the capstone, and where this series ends: where that key lives, and why it never leaves the box it’s born in.

Anything under an 8

Mon, 08 Jun 2026 00:00:00 +0000

I read the news about the National Vulnerability Database over a coffee that went cold while I sat there muttering at my phone. The short version: the NVD, the free public catalogue that quietly props up half the security tooling you and I run every day, is going under in slow motion. And the more I dug into why, the worse the taste in my mouth got.

I’m an open-source person. I think of myself as part of that community, and the NVD is one of those public goods the whole community leans on without ever really thinking about it. So my first reaction wasn’t clever or measured. It was a kick in the teeth.

The carcass and the vultures

Here’s where things actually are. In February 2024 the NVD had around 13,000 unprocessed vulnerabilities sitting in a queue waiting to be analysed. By the end of 2025 that backlog had passed 27,000. This April, NIST effectively admitted it can’t dig out: everything published before 1 March 2026 that hadn’t been enriched got swept into a bucket marked “Not Scheduled”, and going forward only the highest-risk entries get the full treatment. The rest you’re on your own with.

The reasons are grimly ordinary. The Cybersecurity and Infrastructure Security Agency stopped funding the programme in 2024. The enrichment contract lapsed that same February, and despite NIST having two years’ notice it needed a replacement, the database limped along understaffed until late November. And the volume kept climbing regardless: 48,185 CVEs in 2025, roughly 131 a day, with forecasts of the annual figure topping 60,000, getting on for ten times what it was a decade ago. No money, a fumbled handover, and a firehose. That’s the whole story.

The bit that turns my stomach is what comes next. When a free public good fails, the gap doesn’t stay empty. It gets filled, and it gets filled by people selling something. There are already commercial vulnerability databases that are better resourced and more current than the NVD, and the moment the free one is visibly on the floor, every one of them sees a market. Plenty of those subscriptions cost more in a year than a small open-source project will see in donations in its lifetime. So the catalogue the little projects relied on most is exactly the one about to be priced out of their reach. Vultures circling a carcass, and the carcass is something we all built on.

The number we never checked

And then I read the part that stopped me blaming everyone else.

A Department of Commerce Inspector General audit went through the NVD’s work and found that NIST’s own severity scores matched independent assessors only 12% of the time. Read that again. Not that NIST was wrong 88% of the time, that’s not quite what it says, but that two competent parties looking at the same vulnerability landed on the same severity barely one time in eight. The score was never an objective fact handed down from on high. It was always an estimate, a judgement call, the kind of thing reasonable people disagree about most of the time.

Which is awkward, because I have spent years treating that number as gospel. And I know I’m not alone, because I’ve watched whole engineering organisations do the same thing in writing. More than one large employer I’ve had bakes the CVSS score straight into policy: anything scored 8 or above blocks the build and gets a meeting, and anything under an 8 goes through at an engineer’s discretion. When time is money, and it always is in those places, “it’s only a 6.4, ship it” is the easiest decision you’ll make all week. I’ve made it. I’ve made it without opening the advisory, without checking whether the vulnerable code path was even reachable in what we’d built, on the strength of a single number that, it turns out, two experts wouldn’t have agreed on anyway.

So before I get cross about the funding, I have to sit with my own part in this. We took a contestable estimate and bolted it to the door as a gatekeeper. We turned “a rough signal worth a closer look” into “the closer look”, and then we stopped looking. The database didn’t promise us a safety net. We just decided it was one and stopped checking underneath.

Don’t blame the robots for this one

There’s an easy villain on offer here, and I want to wave you off it. It would be tidy to say AI did this, that the flood drowning the NVD is a tide of machine-generated slop, the same dynamic I wrote about when curl’s bug bounty buckled under unverifiable reports. It’s tempting, it’s topical, and it’s mostly wrong.

The people who actually crunch the numbers are clear that the surge is largely legitimate growth. There are now more than 484 CVE Numbering Authorities, far more organisations reporting far more bugs far more thoroughly than they did a decade ago. That isn’t a quality collapse, it’s the system working as designed and simply getting bigger than its funding. Pinning it on AI would be scapegoating, and scapegoating the robots for an underfunding-and-mismanagement problem is just a way of letting the people who defunded it off the hook.

None of which means AI gets a free pass. It just isn’t the arsonist. The same machine-assisted discovery tools that found genuine bugs are also forecast to push CVE volumes higher still, and yes, one of the tools named in that forecast is the very one I poked fun at over curl. AI is an accelerant on a fire that was already burning for thoroughly human reasons. It’s a beat in this story, not the spine.

The version I’m betting on

Where does this leave the working engineer? In a harder spot than before, because the easy answer stopped being easy. My usual line, the one I keep ending these pieces on, is that the diligence is the job: pin, lock, audit, and read the actual advisory instead of trusting a number. All of that still holds. But it just got more expensive, because the data underneath the diligence is thinner and, as it turns out, was shakier than we let ourselves believe.

So I’m not going to pretend there’s a clean fix. This problem won’t solve itself, and it won’t be solved by any one of us. It needs all of us to actually support the services we depend on, with money, with contributions, with attention, so the public goods that underpin our craft are still standing in ten years. That’s the unglamorous, grown-up part.

But I’ll end this one looking up rather than down, because for once I can. I think the next few years bend towards safer software almost in spite of us. Modern languages are quietly closing off whole categories of vulnerability at the source: every memory-safety bug that a borrow checker refuses to compile is one that never reaches a database to be mis-scored in the first place, which is rather the point of building a framework that contains no unsafe. Used with proper guidance instead of left to spew slop, AI can be a genuine help finding and triaging the things that do slip through. And the junior engineers we keep sawing off the bottom rung are exactly the people who, mentored by the greybeards before they retire, could build the next generation of vulnerability identification that the current model clearly can’t sustain.

As for the vultures… it’s a coin toss. A lot of firms will look at the NVD on its back and see a land grab. I’d love to be proved an optimist and watch at least one of them stand tall, take all that better-resourced data and open it to open-source projects for nothing, because it’s the right thing to do and because the whole industry drinks from that well. One of them doing the decent thing would be worth more than all the press releases about responsible AI put together.

The catalogue is wobbling. The number was never as solid as we treated it. Neither of those is the end of the world, as long as we stop outsourcing our judgement to a free service we never funded and never checked, and start paying, in every sense, for the foundations we build on. Boring, unfashionable, and the only thing that ever works. I think we’re up to it.

The consent you can't ask for

Sat, 06 Jun 2026 00:00:00 +0000

There’s a comfortable story going round about telemetry, and it goes like this. There are two kinds. There’s the creepy kind, the usage data a vendor harvests to work out who you are and what you do, and that kind needs your permission. And there’s the innocent kind, the operational data a service emits so the people running it can keep it up, and that kind is just plumbing, nobody’s business, no permission required. Two neat boxes, and only one of them has a lock on it.

I don’t think the boxes are that neat. And I think a fair few of the people drawing them that way know it.

Because there’s no clean line where operational data stops being personal. A web service’s logs carry IP addresses. Its traces carry the path you walked through the system, the ids of the things you touched, sometimes the very fields you sent. Point at almost any of it and a GDPR lawyer will cheerfully tell you it can be personal data, and that the law doesn’t much care whether you filed it under “analytics” or “observability”. The word you picked to describe the data was never the thing that decided whether it was personal. The data decided that, and a lot of operational data is personal.

So if you can’t hide behind the box marked “just plumbing”, what do you actually do?

Where I’m coming from

I should say up front that I haven’t always been this relaxed about it. I spent a good few years in righteous fury at every tool that phoned home, every “we collect anonymous telemetry to improve the product” I never agreed to. Then I started building the tools, and I needed the data myself: the kind that tells you which features people actually use and which command falls over on first run, the kind that lets you make the next decision with something better than a hunch. And it softened me. Not into thinking it’s fine to take it without asking. Into understanding why everyone wants to.

What the fury left me with, the one thing I’ve never talked myself out of, is being pro-choice. Not pro-collection, not anti. Pro-choice. Any tool I ask another person to run will never quietly opt them into sending me a thing. It asks. On first run it makes its case, says what it wants and why, and lets them say no and mean it. I’ll try hard to win the yes, because the data is genuinely useful and a tool gets better when people share it. But I won’t presume it. The choice is theirs, and the prompt exists so they actually get to make it.

The trouble with a service

Which is a lovely principle right up until you build a web service. Because who, exactly, do you prompt? An API doesn’t have a first run. It has a thousand callers a second, none of them sat at a terminal waiting to tick a box. You can’t show a consent dialog to a webhook. The answer the industry reaches for is “consent is implied by use”, and… maybe. It’s a grey area, full stop. Implied consent is the same hand-wave that gave us the cookie banner, the thing we all click through without reading. I’m not going to stand here and call it clean.

But there’s a version of the principle that survives the grey, and it’s the one I built the framework around. Consent belongs to whoever can actually give it. For a command-line tool, that’s the person running it, so you ask them. For a web service, the person who can give it was never the end user at all, because you can’t reach them. It’s the engineer who deploys the thing. They know what their service collects, who its users are, which law they sit under, whether they owe anyone a privacy notice. They are the one party in the whole chain who can make the call with any of the facts in front of them. So that’s where the choice goes.

Which is why, in go-tool-base, the web-service telemetry is a switch. On or off, the engineer’s hand on it, collecting only what you need to keep the lights on by default. There’s no consent prompt, not because consent stopped mattering, but because there’s nobody in the loop I could ask. The accountability sits with the person who can hold it.

The part I’ll own

I’m pro-choice on telemetry, which is exactly why I built a way to switch it off and a way to force it on. Because for a web service the person holding the choice was never the end user, it’s the engineer who ships it, and “pro-choice” has to mean putting the switch in their hand, not pretending a popup would have meant anything.

That force-it-on part is the bit I’ll answer for. I built a way for a tool author to bypass the first-run prompt entirely and bake the consent in. There’s a real use case behind it, the enterprise tool deployed under a policy where collection is contractual rather than optional. But I also know I’ve handed someone a way to take the choice away, and I did it deliberately. Rightly or wrongly, I made the framework flexible enough to do the wrong thing, and the line I care about is now only as safe as the judgement of whoever picks it up.

That’s the uncomfortable place this lands, and I’ve come to think it’s the true one. A framework can put the choice in the right hands. It cannot make the right choice. I can build the prompt, build the switch, set the defaults to the modest thing, and after that I have to trust the engineer on the other side to use it justly and with some wisdom, because there is nothing further down the stack that makes them. When the blame gets shared out, and it’s always shared, a piece of it has my name on it, for every escape hatch I left in.

I’m at peace with that, mostly. Not because the grey went away, but because the alternative, pretending there’s a clean line and that “operational” means “not your problem”, is the real dodge. I’d rather say it plainly: this data can be personal, the consent is real even when there’s nobody to ask, and the most a tool can do is hand the decision to the person who can make it, and trust them with it.

Three traps release-plz sets for a Rust workspace

Fri, 05 Jun 2026 00:00:00 +0000

I wrote up the two days I lost releasing a seventeen-crate workspace to crates.io as a war story, wrong turns and all. This is the other half: the field guide, so you don’t have to lose the same two days.

release-plz is a genuinely good tool, and none of what follows is a bug. It’s three behaviours that are entirely within its design and will still ambush you the moment you point it at a Cargo workspace rather than a single crate. Mildest first, because the third is the one that actually ate my release.

First, what release-plz is doing

In one line: it’s release-please for cargo. It keeps a Release MR open, bumps your versions and per-crate changelogs from your Conventional Commits, and when that MR merges it publishes every crate to crates.io and tags the release. On a workspace where N crates all share one version, “the release” is N publishes and N tag operations. Hold on to that N. It’s hiding behind all three traps.

Trap 1: the default tag template is built for one crate, not a workspace

You will reach for one tag per version, and for me it was more than tidiness. I wanted to ship the whole framework as a single release: one v0.5.1 covering all seventeen crates, because that was the compatibility promise I wanted to make. Use the crates that share a version and they’re guaranteed to work together. A single tag felt like the natural way to say “this is one coherent release of the whole thing” (and it didn’t hurt that the repo already had a v0.5.0 tag from before release-plz, so one unified tag also looked like continuity). So you either set this, or, worse, you leave git_tag_name unset assuming the default does something workspace-aware:

git_tag_name = "v{{ version }}"

Here’s the catch. release-plz’s default git_tag_name is v{{ version }}, and release-plz tags per crate. So the first crate publishes and creates the tag v0.5.1. The second crate publishes and tries to create v0.5.1 again:

ERROR failed to create git tag 'v0.5.1'
 "message": "Tag v0.5.1 already exists"

By the time you read that error, the first crate (and on a retry, the next, and the next) is already live on crates.io, and crates.io publishes are forever. Leaving the line out doesn’t save you, because the default is the same single-crate-shaped template. This is the trap I walked straight into on the release commit.

Trap 2: “one release for the whole workspace” isn’t a setting, it’s a category error

The natural next thought is “fine, I’ll keep one tag but configure release-plz to roll the crates into a single release.” There’s no knob for that, and chasing one is a waste of an afternoon. release-plz’s model is per-crate all the way down: per-crate tags, per-crate GitLab/GitHub releases, per-crate changelogs. “One unified release for the whole workspace” isn’t an option it withholds, it’s a shape it doesn’t have.

So you stop fighting it and set the per-crate templates explicitly:

git_tag_name = "{{ package }}-v{{ version }}"
git_release_name = "{{ package }} v{{ version }}"

Now each crate gets its own tag (rtb-assets-v0.5.1, rtb-config-v0.5.1, and so on) and its own release. It’s more objects per version than you wanted, but it’s the grain the tool works in, and once you accept that the collisions stop.

This is where I had to pull apart two things I’d quietly merged in my head: the version and the tag. The compatibility promise I cared about, that crates sharing a version work together, is carried by the version, and release-plz keeps every crate on the one workspace version no matter how it tags them. The tag is just a label pointing at a commit. I’d wanted a single tag to mean “one coherent framework release”, but the coherence was always in the shared version number, not in the tag. Once that landed, seventeen tags stopped feeling like seventeen releases of seventeen different things and started looking like what they are: seventeen labels on one versioned release. The version is not the tag. If you still want one human-facing narrative for the whole thing, keep a hand-written root CHANGELOG.md alongside the generated per-crate ones, rather than trying to make release-plz aggregate.

Trap 3: a release reads its config from the release commit, not HEAD

This is the small one, and the one that cost me the most, because it makes the fix for Trap 1 look like it isn’t working.

When release-plz runs a release, it does not read release-plz.toml from your working tree. It reads it from the release commit, the commit that first introduced the version it’s releasing. So picture the obvious recovery: you hit the tag collision, you realise your template is wrong, you fix it in a follow-up commit and push to main. Your fix is real. It’s committed. It’s on the default branch. And it is completely ignored, because the version hasn’t changed, so the release commit release-plz reads from is still the old one with the old template.

I didn’t take this on faith. With the corrected per-crate template sitting on HEAD, the CI release job still tried to create the unified tag, pinned to the old commit:

ERROR failed to create git tag 'v0.5.1' with ref 'f6de975...'
 "message": "Tag v0.5.1 already exists"

That ref is the release commit, not the HEAD that held my fix. And the cruel part: release-plz release --dry-run on your laptop reads your working-directory config, so it renders the shiny new per-crate tags and tells you you’re sorted. CI runs the real thing against the release commit and does something else entirely. Same config file, two different answers depending on who’s asking, which is why the war story has the title it does.

The operational rule that falls out of this: any release-plz config change that affects how a release behaves has to ride along with a version bump, or it does not apply. A “fix-up” commit on its own is a no-op.

If you set one thing

If you run release-plz on a multi-crate workspace and you change a single line from the defaults, make it the tag template:

git_tag_name = "{{ package }}-v{{ version }}"

And set it before your first release, not during it, so it’s already in the commit that introduces the version, because that’s the only commit a release will ever read it from. Everything else here follows from two facts: the grain is per-crate, and CI reads history while your laptop reads your working tree. Trust the history.

None of this is release-plz misbehaving. Every bit of it is documented and deliberate. It just isn’t where you’ll think to look until it has published six crates you can’t take back, which is roughly how I came to know it so well.

Telemetry that asks, and telemetry that doesn't

Thu, 04 Jun 2026 00:00:00 +0000

go-tool-base has had a thing called telemetry for a long while now. It’s the opt-in kind: the product analytics that asks a user’s permission before it phones a single byte home, sits there as a no-op until they say yes, and can be wiped on request. The whole package is built around consent.

Then the web-service series went and needed telemetry too. Not that telemetry. The other one, the one the rest of the industry means when it says the word: traces, metrics and logs of a running service. And the awkward thing about those two is that they share a name, they want to share a package, and they pull in exactly opposite directions on the one question that matters most.

This is the story of how 0.7.x grew a second telemetry without breaking the first, and where the line between them ended up getting drawn.

Why bother putting it in the framework at all

The starting point is that I could have left observability out. A reader could wire up OpenTelemetry in their own service and go about their day. But the six parts of the web-service series spent a lot of effort making the transports first-class: a gRPC server, an HTTP server, a gateway, TLS across all of them, each one a Register call against the controller. Turning a CLI into a real long-running service and then shrugging “observability is your problem” would have left a hole exactly where it hurts.

Because a service you can’t see into is a liability the moment it leaves your laptop. The series ended with a macguffin service that was typed, fast and served over TLS, and was also a black box: when it got slow, you had nowhere to look. Metrics and traces are how you get the lights on, and they deserved the same first-class treatment as the things they observe.

The other half of the reason is that the framework already had a foot in this world. The analytics package’s preferred backend speaks OTLP, the OpenTelemetry wire protocol. So OpenTelemetry was already in the building. Doing observability any other way would have meant two standards where one would do.

The catch: two telemetries, opposite instincts

Here’s where it gets interesting, and it’s the part worth slowing down on.

The analytics telemetry is about a user. It collects usage data, hashed machine id, which command ran, exit code, and the entire design assumes you have to ask first. It is off by default. The collector you get when it’s disabled is a no-op, so nothing is recorded until the user opts in, and there’s a deletion path for when they change their mind. That’s not an add-on, that’s by design.

The observability telemetry is about a service. It emits operational data, how long a request took, which span was slow, how many errored, to a collector the operator runs. And there is no user in the loop to ask. The operator deploys the service, points it at their collector, and that act is itself the consent. Asking would be nonsensical: whose permission, for data about their own service, on their own infrastructure?

So you have two things called telemetry, wanting to live in one package, with the opposite default on consent. One is off until someone says yes; the other is on the moment it’s configured. Get that wiring wrong and you fail in one of two ugly ways. Gate the operational telemetry behind the user’s analytics opt-in, and a service’s tracing silently does nothing because nobody ticked a box meant for something else. Or loosen the analytics gate to make observability flow, and you start leaking usage data the user never agreed to share. Neither is acceptable, and “just use two packages” throws away everything the two genuinely have in common.

Quite a lot, as it turns out, and all of it below the consent line.

Both ship their data over OTLP to a collector. Both need to describe who is emitting, the service name and version, the resource in OpenTelemetry’s terms. Both parse an endpoint, attach headers, decide whether the connection is plaintext. None of that has the faintest thing to do with consent. It’s just the plumbing of getting bytes to a collector, and the analytics backend already had all of it, written inline.

So the shape of the solution fell out of the problem. Lift the shared plumbing into one place, let both telemetries stand on it, and keep the consent decision firmly out of that shared layer. The structure under pkg/telemetry ended up like this:

pkg/telemetry/
 telemetry.go the analytics Collector (consent-gated)
 backend_otel.go its OTLP backend
 posthog/ datadog/ vendor analytics backends
 otelcore/ shared: OTLP endpoint, resource, telemetry.* config
 tracing/ observability signal
 metrics/ observability signal
 logs/ observability signal
 observability.go Setup: builds the enabled signals (implied consent)

The new otelcore is the keystone. It holds the three things both sides need and nothing they don’t: ParseEndpoint for the OTLP URL, Resource for the service identity, and Resolve for reading the shared telemetry.* config (a base endpoint, plus per-signal overrides, in the same cascade as the TLS config). It imports no signal exporter and knows nothing about traces, metrics, logs or analytics. It is deliberately dumb plumbing.

The refactor: making the old telemetry stand on the new core

This next part is where the old telemetry and the new one become a single thing. The analytics OTLP backend was the first user of OTLP in the framework, and it had grown its own copy of all that plumbing: a function that parsed the endpoint URL, split out the host and path, worked out the insecure flag, and built the resource from a service name. Exactly the code the three new signals were about to need.

So rather than write it a second time and let the two drift, the analytics backend was refactored onto otelcore. Its exporter builder, buildOTelExporterOpts, now calls otelcore.ParseEndpoint, the same function tracing, metrics and logs call, and the resource comes from otelcore.Resource, the same one they use. One implementation of “talk OTLP to a collector”, four callers: the analytics backend and the three observability signals. Change how the framework forms an OTLP endpoint, and every signal moves together.

The reassuring part was that the analytics tests didn’t budge. The refactor moved code without changing behaviour, and the consent machinery, the opt-in, the no-op-when-disabled, the deletion path, never came near otelcore. Which is exactly the point.

Where the line is

Because the shared core is the easy half. The half that earns its keep is the bit that isn’t shared, and it’s a single, deliberate line.

The analytics collector keeps its gate. The constructor, NewCollector, still returns a no-op the moment telemetry is disabled, so a user who hasn’t opted in gets a collector that silently discards everything. Informed consent, untouched.

Observability gets a different door entirely. Setup builds whichever signals the operator has switched on, and it is gated only by telemetry.tracing.enabled and its siblings, which the operator sets. It never consults the analytics opt-in. Turning on tracing doesn’t turn on analytics; disabling analytics doesn’t silence tracing. The two enable flags live under the same telemetry.* config root, sit next to each other, and never read each other.

So that’s the whole architecture in a sentence: one package, one OTLP export core, two consent models that share everything except the answer to “do we need to ask”. The principle underneath, the one that decided every one of these calls, is that the kind of data sets the consent model. Usage data about a person needs informed consent. Operational data about a service runs on implied consent. The CLI and the web service are just where each kind tends to live.

Where this leaves the framework

0.7.x came out the other side with both telemetries: the one that asks first, exactly as it was, and a new one that doesn’t, because it has nobody to ask. They share an export core, a config root and a name, and they part company on the only thing they were ever going to disagree about.

I’ve been careful here to describe how the two consent models are kept apart, not to argue why they have to be. That argument, that “the kind of data decides the consent model” is a line worth holding rather than a convenient bit of engineering, is a piece of its own, and it’s the one I’m writing next.

Same config, two answers

Wed, 03 Jun 2026 00:00:00 +0000

Let me confess a small heresy first, because it’s the reason any of this happened. After a career spent as a branching man, gitflow, gitlabflow, a tidy develop branch and a careful dance of merges, I’ve come round to trunk-based development. I resisted it for years. It felt like working without a net.

What changed my mind was working solo with an AI pair. The branch ceremony that earns its keep on a team of eight is just drag when it’s me and a model at two in the morning. So I’ve softened on “main is always deployable” and let the trunk act as the develop branch, with tagged releases as the actual source of truth. For compiled languages, where the artefact you ship is a built, tagged thing and not whatever’s on a server right now, that finally clicks.

I’d already rolled this out on my Go and Terraform projects with releaser-pleaser, a GitLab-native take on release-please: a bot keeps a Release MR open, and merging it cuts the tag. It’s the same model I wrote about when the infra repo moved to plan-on-merge, apply-on-tag. Lovely. Then I came to do the same for rust-tool-base, and Rust, being Rust, had opinions.

Rust brings its own toolchain

releaser-pleaser is happy to tag a repo and write a release. What it does not do is cargo publish seventeen crates to crates.io in dependency order. Rust’s release story isn’t “push a tag and let a runner build a binary”, it’s a whole publishing pipeline with a public registry at the end of it, and that registry has rules of its own. So for the Rust workspace I reached for the tool built for exactly that job: release-plz. Same Release-MR shape, but it understands cargo, versions every crate, and publishes the lot.

That was the right call. Getting it to actually do it was where I spent two days I’d quite like back.

The gauntlet before the gun

Before I got anywhere near the interesting failure, there was a run of CI papercuts, the sort where every fix politely reveals the next one. GitLab checks out a detached HEAD, and release-plz wants to be on a branch (“HEAD does not point to a branch”), so you re-attach. Then the default CI_JOB_TOKEN can’t push to a protected repo, so you point the remote at a real token. Then release-plz assumes you’re on GitHub and errors that the repo “is not hosted in GitHub”, so you tell it --forge gitlab. Then it refuses to run at all because the pages job left a public/ directory lying about and the working tree is “dirty”, so you stop pulling artefacts into the job.

Five merge requests before the thing would even start doing its actual job. You can read the scar tissue in the before_script; every line in it is a fix for something on that list. None of it was hard. It was just death by a thousand cuts, and I was feeling quite smug by the time it finally reached the publish step.

I should not have been.

“Tag v0.5.1 already exists”

My release-plz.toml asked for one tag per release:

git_tag_name = "v{{ version }}"
git_release_name = "v{{ version }}"

That felt obviously right. It matched the repo’s existing v0.5.0 tag, it’s how a single-crate project tags, and the crates all share one workspace version anyway. One version, one tag. What’s to argue with?

release-plz, that’s what. It tags per crate. So it publishes a crate, creates the tag, publishes the next crate, and tries to create the same tag again:

INFO published rtb-assets 0.5.1
ERROR failed to release package
Caused by:
 0: failed to create git tag 'v0.5.1' with ref 'f6de975a75...'
 1: Response body: { "message": "Tag v0.5.1 already exists" }
 2: HTTP status client error (400 Bad Request) ... /repository/tags

The collision is annoying. What makes it a proper trap is the half-second before it: published rtb-assets 0.5.1. That happened. On crates.io. For keeps. A crates.io publish is forever, there is no unpublish, only a yank that still leaves the name and version burned. So every time my flaky pipeline limped one crate further and then fell over on the tag, it left another crate live on the public registry that I could never take back. By the time the dust settled, six of the seventeen were out there: rtb-assets and rtb-config, then on a later retry rtb-credentials and rtb-error, then rtb-app and rtb-redact. Two more permanent crates per failed run.

I assumed the default

My first fix was the clever one, and it deserves to be on display because it’s the whole lesson in miniature. I deleted the git_tag_name line. My reasoning: per-crate tags are release-plz’s native model, so surely its default does the right thing without me spelling it out. I was confident enough to write it into the commit message: “per-crate tags/releases (release-plz defaults).”

The next run collided on v0.5.1, exactly as before.

Because release-plz’s default git_tag_name is not per-crate. It’s the unified v{{ version }}. I had deleted a line that said the wrong thing and replaced it with a default that said the same wrong thing, then congratulated myself for tidiness. If I’d spent thirty seconds on the configuration reference instead of thirty seconds being clever, I’d have read that in black and white.

Same config, two answers

So I read the manual, and set it explicitly:

git_tag_name = "{{ package }}-v{{ version }}"
git_release_name = "{{ package }} v{{ version }}"

On my laptop, a dry run rendered exactly the per-crate tags I wanted. In CI, the very next run published another crate and then created the tag v0.5.1. The unified one. The wrong one. The one I had just, demonstrably, on main, replaced. Same release-plz.toml, two completely different answers depending on who was asking.

That one took me an embarrassingly long time to see. release-plz does not read your config from the working tree when it runs a release. It reads it from the release commit, the commit that introduced the version it’s releasing. My version was still 0.5.1, set days earlier on a commit that still carried the unified template. You can see it in the failure: the tag it tries to create is pinned to ref 'f6de975...', an old commit, not the HEAD that held my fix. Every edit I made at the tip of main was real, committed, and utterly invisible to the release of 0.5.1, because no version bump had created a fresh release commit for it to read. My fix was correct and inert at the same time. The dry run read my working directory and looked perfect; CI read history and did something else.

There is no config change that rescues an in-flight release. The version was already out, half-published, tagged wrong, and pointed at a commit I couldn’t edit without bumping the version, which I couldn’t cleanly do with six crates already live.

Doing it the way I’d have done it a year ago

So I stopped. Three retries deep, each one a seventy-minute CI cycle thrown at an opaque mismatch, six crates already immovable on crates.io, and a tooling problem I now understood well enough to know the tool was never going to dig me out of this particular hole. The question quietly changed from “why is it doing this?” to “am I going to keep grinding, or finish this the way I would have before I had clever tooling?”

I went manual. cargo publish, the remaining eleven, by hand, in dependency order: the leaf crates first and the rust-tool-base umbrella dead last, because it depends on all of them. crates.io rate-limits new crate names, so after a burst it simply made me wait, a roughly half-hour pause in the middle while the registry caught its breath and I caught mine. Then one v0.5.1 tag, cut by hand, and one GitLab release to match the convention. The next CI run came up green, for the gloriously dull reason that there was nothing left to do: every crate published, the tag already there.

Stop being clever and RTFM

The tool was never broken. Every single thing it did was documented behaviour I hadn’t bothered to read: that the default tag template is unified, that the model is per-crate, that a release reads its config from the release commit and not from HEAD. I assumed my way past the manual three times in a row, and each assumption cost me real, permanent state on a public registry that doesn’t take returns.

And that’s the part that actually stung, because I should have known better than most. I wasn’t a beginner here. I knew the Release-MR pattern cold, I’d shipped it half a dozen times with releaser-pleaser on my Go and Terraform repos. That familiarity was the trap. I trusted the pattern and skipped the tool, on the lazy assumption that something I understood well in one tool would behave the same in the next. release-plz carries the same design, but it’s a different tool, with its own defaults and its own idea of where the config lives. The pattern came across fine. The mechanics didn’t, and I never thought to check.

So here’s the lesson, written down in the hope it sticks this time: no matter how familiar I am with a pattern or a design, the moment I switch the tool that implements it, reading the manual is paramount. The familiarity is exactly what tempts you to skip it, and exactly why you can’t. (The narrower, more practical one, while I’m here: a config change that affects how a release behaves has to travel with a version bump, or it sits there looking applied and doing nothing.)

release-plz is genuinely good, and every release since has gone out clean on the first try, the way the rest of the CI now does. I just had to stop being clever long enough to read how it actually works. RTFM. I’ll get it tattooed eventually.

Building a web service with go-tool-base, part 6: seeing what your service is doing

Tue, 02 Jun 2026 00:00:00 +0000

On paper the macguffin service is finished. Part 5 left it typed, fast, documented and served over TLS. So you deploy it, traffic starts flowing, and a week later someone wanders over to say “it’s slow”.

Slow how? Slow where? Which endpoint, the gateway or the gRPC behind it, the store or the network in between? You open the logs and there’s a wall of “request completed” lines, no way to tell which ones belong to the same request, and not a word about the half-second one of them spent somewhere inside. The service is a black box that happens to be on fire.

That’s the gap this part closes. A long-running service needs to be observable: you need to see what it’s doing, how often, how fast, and where the time actually goes. And we get there without bolting on a separate stack, because go-tool-base speaks OpenTelemetry, and three signals fall out of the same wiring.

This part is written against go-tool-base v0.7.1, the release that brings the observability components in.

The three things you actually want to see

“Observability” is a grand word for three fairly down-to-earth questions. Logs answer what happened (you’ve had those since part 1). Metrics answer how often, and how fast: request rates, error counts, the spread of latencies. And traces answer where did the time go, by following a single request as it crosses from one piece of your system to the next.

For a service shaped like ours, that last one is the prize. A REST call hits the gateway, turns into a gRPC call, runs through the store, and comes back. A trace stitches that whole journey into one timeline, so “it’s slow” stops being a shrug and becomes “it’s slow in CreateMacguffin, in the store, for 400ms”. You can’t get there from a pile of disconnected log lines.

OpenTelemetry (OTel, if you’ve seen it bandied about) is the vendor-neutral standard for all three. You instrument once and ship the data to whatever backend you fancy (Jaeger, Grafana, Honeycomb, a plain collector) over one wire protocol, OTLP. go-tool-base does the instrumenting; you point it at a collector and pick your poison.

Turn it on

One call wires the lot. In the serve command from part 4, right after the controller exists:

// pkg/cmd/serve/main.go

import "gitlab.com/phpboyscout/go-tool-base/pkg/telemetry"

controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

// Build whichever signals are enabled, install them, flush them on shutdown.
if _, err := telemetry.Setup(ctx, p, controller); err != nil {
	return err
}

telemetry.Setup reads your config, builds whichever of the three signals you’ve switched on, and installs them as the process-wide OTel providers. It also hangs their shutdown on the controller, so when a SIGTERM lands the buffered spans and metrics flush before the process exits. That’s the same graceful-shutdown promise from part 1, now extended to your telemetry: a clean stop drops nothing, not even the trace that was mid-flight.

On its own that line shows you nothing. A signal you haven’t enabled is skipped, so an unconfigured service pays exactly nothing. You turn each on in config, which we’ll get to. First, the instrumentation.

Wire it onto the transports

Setup builds the pipeline; two short additions feed it. The gRPC server takes a stats handler:

// pkg/cmd/serve/main.go

grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger,
	gtbgrpc.OTelStatsHandler())

and the HTTP server takes a middleware. This is also where we finally meet go-tool-base’s middleware chain, which we’ve managed without until now. A chain is just an ordered stack of wrappers around your handler. We put the OTel one first so it opens the span, and the request logger second, so it runs inside that span and can see it:

// pkg/cmd/serve/main.go

chain := gtbhttp.NewChain(
	gtbhttp.OTelMiddleware("macguffin"),
	gtbhttp.LoggingMiddleware(p.Logger),
)

if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, mux,
	gtbhttp.WithMiddleware(chain)); err != nil {
	return err
}

That’s the whole code change. Both helpers, OTelMiddleware and OTelStatsHandler, lean on the standard OTel instrumentation libraries underneath (otelhttp and otelgrpc), so the spans and metrics they emit follow the conventions every backend already knows how to read. Every request now opens a span and records its own duration, on both transports, and you wrote none of it per handler.

Go deeper than the plumbing

The transport spans show the request crossing HTTP and gRPC, but they stop at the edge of your handler. To see where the time goes inside it, you open a span yourself, and that needs no go-tool-base API at all, just OpenTelemetry. Here’s the store doing it, opening a span around the work and tagging it with what it’s handling:

// internal/macguffin/store.go

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
)

var tracer = otel.Tracer("macguffinsvc/store")

func (s *Store) Create(ctx context.Context, name string, quantity int32) Macguffin {
	_, span := tracer.Start(ctx, "Store.Create")
	defer span.End()

	span.SetAttributes(
		attribute.String("macguffin.name", name),
		attribute.Int("macguffin.quantity", int(quantity)),
	)

	// ... store the macguffin, generate its id ...

	span.SetAttributes(attribute.String("macguffin.id", id))

	return m
}

The span starts as a child of whatever is already in the context, which is the gRPC server span the handler is running under, so it slots into the trace on its own. The one thing to get right is the context: give the store method a ctx context.Context first parameter and thread it down from the handler, and the span finds its parent. Tag it with the things you’d want when something’s wrong, the macguffin’s name and its id, and they show up on the span in the UI, right next to the timing.

Point it at a collector

The wiring’s on; now tell it where to send. The config lives under one root:

# config.yaml
telemetry:
 endpoint: "http://localhost:4318"
 insecure: true # plaintext: a local collector only
 tracing:
 enabled: true
 sampling: 1.0 # every trace while you're looking
 metrics:
 enabled: true
 logs:
 enabled: true

endpoint is your OTLP collector. Sampling defaults to a level-headed 0.1 in production (you rarely want every trace under real load), so turn it up to 1.0 while you’re developing or you’ll wonder where yours went. Stand up a collector (a local Jaeger handles traces and gives you a UI to poke at), start the service, and push some traffic through the REST side:

$ curl https://localhost:8443/v1/macguffins
$ curl -X POST https://localhost:8443/v1/macguffins -d '{"name":"the-grail","quantity":1}'

One request, the whole way through

Open the collector and there it is, the thing that justifies the exercise: a single trace for that POST, four spans deep. The HTTP server span on top, the gRPC call nested inside it, the gRPC handler inside that, and your Store.Create span at the bottom, with the macguffin’s name and id sat right on it. The REST request became a gRPC call, crossed the gateway we built in part 4, ran the handler and went into the store, and OTel followed it the entire way, because the trace context travels with the request and everything reads from the same providers. “It’s slow” finally has somewhere to point, and “slow where” has an answer.

The metrics ride the same pipeline: request counts, error counts, a latency histogram broken down by route and method. And your logs get an upgrade you didn’t ask for. Every “request completed” line now carries the trace_id and span_id of the request it describes, so a slow trace in the UI and its log line in your terminal share an id, and you can jump straight from one to the other. That correlation lands in your stderr logs and in the records shipped to the collector, both.

A word on the other telemetry

If you read Telemetry that asks first, you’ll know go-tool-base has a telemetry package for product analytics too: the opt-in kind that asks a user’s permission before it phones anything home. This is not that, and the difference is the whole point.

That analytics is about a user, and it’s gated behind informed consent: off until they say yes. This is about your service, emitted by you, to a collector you run, and it works on implied consent. You switched it on; there’s no end user in the loop to ask. Same package, deliberately separate gates, and never the twain shall meet. There’s a fuller post in that one distinction, and it’s the next thing I’m writing.

Where that leaves the service

Six parts in, the macguffin service starts cleanly, reports its health, serves gRPC and REST and interactive docs over TLS, shuts down without dropping a request, and now tells you what it’s doing while it runs, all from one annotated proto and a controller it registers against. Observability was the last missing sense. The thing was fast and correct before, but you couldn’t see inside it. Now you can, and it cost you about a dozen lines.

The security service I had to switch off

Mon, 01 Jun 2026 00:00:00 +0000

A while back I wrote about hardening the account that would hold the signing key, and one line in it has aged badly. “GuardDuty is already looking,” I wrote: the account watched from day one, threat detection on before the key even arrives. Then I went to apply that baseline to a brand-new account, and GuardDuty wasn’t looking at all, because the account wouldn’t let it.

What the baseline switches on

The baseline runs a threat-detection module that turns on three AWS services: GuardDuty, Security Hub, and IAM Access Analyzer. Each sits behind an enable_* toggle, all defaulting to on, which is the right default. An account about to hold something sensitive should be watched, flagged and analysed from the start. The hardening post’s whole argument was that you fit the locks before you move the valuables in, and I stand by it. The gap I hadn’t accounted for is the one between a posture you can design and a posture a fresh account will actually run on its first day.

SubscriptionRequiredException

GuardDuty and Security Hub are first-party AWS services, but they are not on by default. They have to be activated for the account before anything can configure them. Point OpenTofu at a brand-new account that has never had them switched on and you don’t get a security baseline, you get a SubscriptionRequiredException and a failed apply. The locks you carefully specced won’t fit, because the door hasn’t been registered as a door yet.

I’d actually met this exact wall from another direction: an aws-nuke dry-run on a fresh account throws screenfuls of the same exception for every service you’ve never enabled. Same root cause, different tool. A new AWS account is a quieter, emptier place than the console makes it look.

There’s a second, duller reason too, and it belongs in the open: GuardDuty and Security Hub both cost money once their free trial lapses, and the budget for this account wasn’t in place yet. Enabling a service you can’t yet afford to keep on is its own small mistake.

Switching two watchers back off, on purpose

So I did the thing that felt wrong and was right. I switched two of them off, and wrote the reason straight into the module call:

# GuardDuty + Security Hub are deferred. The account's first-time
# service activation is pending (it returns SubscriptionRequiredException)
# and the services carry an ongoing cost beyond their free trial.
# Re-enable by flipping both to true (or deleting these lines) once
# activation clears and the budget is in place.
enable_guardduty = false
enable_securityhub = false

Two things make that a deferral rather than a retreat. First, it’s scoped: Access Analyzer needs no subscription and costs nothing, so it stays on. The account isn’t unwatched, it’s watched by the part that can watch it right now. Second, it carries its own undo. The comment is the re-enable instructions, the toggles sit right there, and flipping them back is a one-line change the day activation clears and the budget lands. A disabled control with a written path back is a deferral. A disabled control with no note is a hole someone finds in a year.

Why this lives in the module, not in a hack

I could only do this cleanly because the module exposes each service as its own toggle, and it only gained that recently; the previous version was all-or-nothing on threat detection. Granular enable_guardduty and enable_securityhub flags are exactly what let you say “this account, these two, not yet” without forking the module or commenting resources out. It’s the difference between a baseline you adapt per account and one you fight.

The honest version of “secure by default”

The hardening post wasn’t wrong. It was idealised. Secure-by-default is the right aim, and on a mature account the baseline goes on clean. On a brand-new one, reality intrudes: services that must be activated before they can be configured, costs that need a budget before they can be incurred. The honest response isn’t to pretend the posture is fully live when it isn’t. It’s to enable what the account will take, defer what it won’t, write down precisely why and how to finish, and leave Access Analyzer watching in the meantime. GuardDuty will be looking soon enough. It just wasn’t looking on day one, and saying so is better than a comment-free enable = true that quietly errored on every apply.

Building a web service with go-tool-base, part 5: docs that write themselves

Sat, 30 May 2026 00:00:00 +0000

The google.api.http annotations we added in part 4 have done one job so far: they told the gateway which REST calls map to which RPCs. But they describe the API precisely, the paths, the verbs, the request and response shapes, and a precise description of an API is most of an OpenAPI document. Feed that document to a viewer and you have an interactive docs site: every endpoint listed, every field typed, and a “try it” button that calls the real service.

So from one .proto we’ve had gRPC, then REST, and now documentation, none of it a separate thing to write or keep in sync.

The annotations, a third time

The pattern of this whole series is one source of truth and several outputs generated from it. The proto defined the gRPC service; the annotations on it generated the REST gateway; and those same annotations generate an OpenAPI document. Add a field to a message, and it shows up in the gRPC API, the REST API, and the docs, all at once, because all three are read from the proto.

Generate the OpenAPI document

This is one more buf plugin. A small wrinkle to know up front: grpc-gateway ships its own OpenAPI generator, but it emits OpenAPI v2 (the old Swagger format). For a v3 document we use kollalabs/protoc-gen-openapi, which reads the very same google.api.http annotations:

go install github.com/kollalabs/protoc-gen-openapi@latest

# buf.gen.yaml
plugins:
 # ... the go, go-grpc and grpc-gateway plugins from before ...
 - local: protoc-gen-openapi
 out: internal/docs/assets
 opt:
 - title=Macguffin API
 - default_response=false

go generate ./... now also writes an openapi.yaml. It’s the REST API described in full, drawn straight from the annotations:

# internal/docs/assets/openapi.yaml (generated, trimmed)
openapi: 3.0.3
info:
 title: Macguffin API
 version: 0.0.1
paths:
 /v1/macguffins:
 get:
 summary: ListMacguffins
 operationId: MacguffinService_ListMacguffins
 parameters:
 - name: pageSize
 in: query
 schema:
 type: integer
 format: int32
 responses:
 "200":
 description: OK
 content:
 application/json:
 schema:
 $ref: '#/components/schemas/ListMacguffinsResponse'

That GET /v1/macguffins operation is the get: "/v1/macguffins" rule from the proto, turned into OpenAPI. You wrote the annotation once; it now feeds three generators.

Serve it, viewer and all

A spec on disk is a means, not an end; people want to read it. The usual move is to bolt on a docs UI like Stoplight Elements or Swagger UI, which means vendoring a couple of megabytes of JavaScript into every project. go-tool-base’s openapi package does that part for you: the Stoplight Elements UI is embedded in the framework, so your project ships only its generated spec.

openapi.Register mounts both the document and the docs site onto a mux:

// internal/docs/docs.go

package docs

import (
	_ "embed"
	"net/http"

	"gitlab.com/phpboyscout/go-tool-base/pkg/openapi"
)

//go:embed assets/openapi.yaml
var spec []byte

// Register mounts /openapi.yaml and the Stoplight docs site (/docs/) onto mux.
func Register(mux *http.ServeMux) error {
	return openapi.Register(mux, spec, openapi.WithTitle("Macguffin API"))
}

The //go:embed bakes the generated spec into the binary (the same trick as part 3.5), and openapi.Register serves it at /openapi.yaml and the Stoplight site at /docs/. Wiring it into serve is one line, on the same mux the gateway is mounted on:

// pkg/cmd/serve/main.go (the existing HTTP wiring)

mux := stdhttp.NewServeMux()
mux.Handle("/v1/", gw) // REST, from part 4

if err := docs.Register(mux); err != nil {
	return err
}

Why it goes on the same server

That last point is doing more work than it looks. The docs, the spec, and the live REST API are all on the one HTTP server, so they share an origin. That’s what makes the “try it” console actually work: when you fill in a request in the docs and hit send, the browser calls /v1/macguffins on the same host it loaded the page from, with no cross-origin dance to configure. And because the certificate is your mkcert one, the page and its requests are all clean HTTPS, no warnings to click past, which is exactly why we set the local CA up back in part 2.

$ curl https://localhost:8443/openapi.yaml | head -4
# Generated with protoc-gen-openapi
openapi: 3.0.3
info:
 title: Macguffin API

Open https://localhost:8443/docs/ in a browser and there’s the service: every endpoint, every field, and a working console that calls the real thing.

One proto, three faces

Step back and look at what a single annotated .proto is now producing: a gRPC service for the things that speak it, a REST API for the things that don’t, and an interactive docs site for the people who have to consume either. One source, three faces, and nothing hand-maintained between them. That’s the whole argument for building it this way, and it’s why the annotations were worth the small ceremony.

The service is, by any reasonable measure, done: typed, fast, documented, and served over TLS. The last part is about what happens once it’s out there and taking traffic. In part 6 we add telemetry and logging, so you can see how it’s being used and why it’s slow, without bolting on a separate observability stack.

From allow_failure to blocking

Sat, 30 May 2026 00:00:00 +0000

There’s a special kind of CI job that everyone on a team quietly learns to ignore: the one marked allow_failure: true. It runs, it goes red, the pipeline goes green anyway, and after the third time you stop looking at it. I inherited six of those when I moved rust-tool-base’s CI to GitLab. Over a few days I turned three of them into real gates, and the interesting part was never the YAML. It was working out which ones had earned the right to block, and which hadn’t.

What allow_failure actually buys you

allow_failure: true is genuinely useful, and quietly corrosive. It lets a job report a problem without stopping the pipeline, which is exactly right for a check that’s noisy, not yet stable, or guarding against something you can’t fix this minute. The trouble is that a warning nobody is forced to act on is a warning nobody acts on. Leave a job advisory long enough and it becomes scenery: red, ignored, pointless. So an advisory check is really a promise, “I’ll make this blocking once it’s trustworthy”, and a promise you only ever mean to keep is just a lie you haven’t noticed yet.

When I migrated rust-tool-base from GitHub Actions to GitLab CI, the move landed six jobs as allow_failure: true: the macOS and Windows tests, the integration tests, cargo-audit, trivy, and coverage. That wasn’t laziness. A migration is the wrong moment to also be fighting flaky gates. But it left me holding six promises to either keep or admit I wasn’t going to.

A check earns the right to block

Here’s the rule I settled on. A check earns the right to fail your build when two things are true: it’s meaningful (a red result is a real problem, not noise) and it’s reliable (it goes red only when there genuinely is a problem, and it can actually run to completion). Flip a check to blocking before both hold and you haven’t raised the bar, you’ve taught the team to force-merge past red, which is worse than no gate at all, because now the red means nothing.

Three of my six crossed that line within a few days. Three deliberately didn’t. The reasons are the whole story.

trivy: blocked once there was nothing to block on

trivy scans the dependency tree for HIGH and CRITICAL advisories. It went across as advisory for an honest reason: the Cargo.lock at migration time already carried two known HIGH/CRITICAL advisories I hadn’t cleared yet, a path-traversal in gix-validate and a DNS-rebinding issue in rmcp. Make trivy blocking with those sitting there and the pipeline is red from day one, over problems I already knew about and was already fixing. So it stayed advisory until the dependency bumps cleared both, and then the allow_failure line came out. The gate never changed. The tree underneath it got clean enough to stand on.

integration-tests: blocked once it could actually run

The integration tests stand up a real Gitea in a Docker-in-Docker service and talk to it. They were advisory for a different reason: they couldn’t reliably run. dind needs a privileged runner, and the suite was resolving the container host with a hardcoded 127.0.0.1 that didn’t hold everywhere. Blocking a job that fails for infrastructure reasons rather than code reasons is the fastest way to make people distrust the entire pipeline. So the fix wasn’t in the YAML, it was making the thing dependable: privileged set on the runner, and the host resolved through the test library’s own get_host() instead of a hardcoded address. Once it ran the same way every time, it earned the gate.

coverage: blocked once it could run at all, then once it cleared the bar

Coverage is the two-step one, and my favourite, because it nearly didn’t make it for a thoroughly undramatic reason: it ran out of memory. cargo llvm-cov instruments every test binary, and linking hundreds of instrumented object files needs more RAM than the shared medium runner had, so the job bus-errored on the link. I tagged it onto a larger runner, and then the shared SaaS runners were switched off entirely, so the tag matched nothing and the job sat pending forever.

The fix was a self-hosted homelab runner with the RAM the instrumented link actually needs. I moved coverage there but kept it advisory for one run, to confirm the box could finish the build before I trusted it. It did, at 73.22% line coverage, so I set the gate to fail under 70% and made it blocking. Three points of headroom: enough that ordinary churn won’t trip it, tight enough that a real drop will. A coverage gate pinned to the current number is a tripwire that fires on the very next commit; set it a touch below and it catches regressions instead of normal life.

The three I left advisory, on purpose

The point was never “block everything”. Three jobs are still allow_failure in the current pipeline, deliberately. The macOS and Windows tests run on SaaS runners that bill by the minute; they’re worth running, not worth blocking every merge of a Linux-first project over a quota I’m choosing to ration. And cargo-audit stays advisory because cargo-deny already does the blocking advisory check: cargo-audit is a second opinion from a different database, and a second opinion that can veto isn’t a second opinion, it’s a duplicate gate that will eventually disagree with the first and block you on the difference.

That’s the same rule from the other side. Those three haven’t earned the right to block, because blocking them would cost more than it ever caught.

The upshot

allow_failure: true is fine as a waiting room and corrosive as a destination. Every advisory check is a promise to make it blocking once it’s both meaningful and reliable, and the job is to keep the promise or admit you won’t. trivy earned its gate when the advisories cleared, the integration tests when they ran the same way every time, coverage when it had a runner with enough memory and a threshold set just below the current mark. The three I left advisory earned that standing too, by costing more to block than they’d catch. The YAML is one deleted line per job. Knowing which line to delete, and when, is the whole skill.

Building a web service with go-tool-base, part 4: REST for free, with the gateway

Fri, 29 May 2026 00:00:00 +0000

A quick tally of where part 3 left us. One domain, the Store. One gRPC service over it, mapping the domain to proto with toProto. And then a whole second transport, the REST layer, with its own routing and its own toDTO mapping the very same domain into the very same shape, by hand. Two encodings of one thing, drifting apart the moment anyone adds a field and forgets the other side.

I promised that doubling would go away. This is the part where it does, and the thing that does it is the grpc-gateway.

What the gateway actually is

The grpc-gateway is a reverse proxy, generated from your .proto, that speaks REST on the front and gRPC on the back. A JSON request comes in, the gateway turns it into the matching gRPC call, hands it to your gRPC server, and turns the gRPC response back into JSON on the way out.

Read that again with part 3 in mind, because it’s the whole point. The gateway does the JSON-to-proto-and-back encoding for you, using the proto types your gRPC server already produces. You wrote domain → proto once, in the gRPC adapter. The gateway supplies proto → JSON. There is no second hand-written encoding to keep in step, because there is no second implementation: REST becomes a generated front door onto the gRPC service you already have.

So the plan is short. Tell the proto which HTTP calls map to which RPCs, regenerate, wire the gateway in, and delete the part-3 REST layer entirely.

Map HTTP onto the proto

gRPC has no opinion about URLs and verbs; REST is all URLs and verbs. The bridge is an annotation, google.api.http, that you attach to each RPC to say “this one is GET /v1/macguffins/{id}”. Here’s the service with those rules added:

// proto/macguffin/v1/macguffin.proto

import "google/api/annotations.proto";

service MacguffinService {
 rpc GetMacguffin(GetMacguffinRequest) returns (Macguffin) {
 option (google.api.http) = {get: "/v1/macguffins/{id}"};
 }
 rpc ListMacguffins(ListMacguffinsRequest) returns (ListMacguffinsResponse) {
 option (google.api.http) = {get: "/v1/macguffins"};
 }
 rpc CreateMacguffin(CreateMacguffinRequest) returns (Macguffin) {
 option (google.api.http) = {
 post: "/v1/macguffins"
 body: "*"
 };
 }
}

Each rule is small but exact. {id} in the path binds to the id field of the request message. body: "*" on the create says the whole JSON body maps onto the request. The list takes no body and no path parameter, just the verb and path. This is the same information part 3’s hand-written routes carried, except now it lives next to the RPC it describes, and a generator reads it instead of you.

These rules go a good deal further than the three cases we need: query-string parameters, several URL bindings for a single RPC, custom verbs, choosing which field becomes the response body. When you reach for those, the grpc-gateway docs walk through the mapping, and the canonical reference is the HttpRule message that google.api.http comes from, its comments document every option.

The annotations.proto import comes from Google’s common protos, so tell buf where to find them by adding a dependency, then fetch it:

# buf.yaml
deps:
 - buf.build/googleapis/googleapis

buf dep update

Generate the gateway

This is another buf plugin, exactly like part 2’s. Install it:

go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest

and add it to the generators:

# buf.gen.yaml
version: v2
plugins:
 - local: protoc-gen-go
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-go-grpc
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-grpc-gateway
 out: internal/gen
 opt: paths=source_relative

go generate ./... now also writes macguffin.pb.gw.go, the gateway: a RegisterMacguffinServiceHandler function that, given a connection to your gRPC server, mounts the REST routes the annotations described.

Wire it on

The gateway needs to call your gRPC server, which means dialling it like any other client, over the same TLS, with credentials that trust its certificate. That’s fiddly to get right by hand, so go-tool-base’s gateway package does it for you. gateway.New opens the connection (matching your server’s transport security) and hands you a mux to register the generated handlers on:

// pkg/cmd/serve/main.go

package serve

import (
	"context"
	stdhttp "net/http"

	"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
	"google.golang.org/grpc"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	"gitlab.com/phpboyscout/go-tool-base/pkg/gateway"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	gtbhttp "gitlab.com/phpboyscout/go-tool-base/pkg/http"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	store := macguffin.NewStore()

	// gRPC: the one implementation, mapping the domain to proto.
	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	// REST, for free: the gateway proxies JSON/HTTP to the gRPC server above.
	gw, err := gateway.New(ctx, p.Config,
		func(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error {
			return macguffinv1.RegisterMacguffinServiceHandler(ctx, mux, conn)
		})
	if err != nil {
		return err
	}

	mux := stdhttp.NewServeMux()
	mux.Handle("/v1/", gw)

	if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, mux); err != nil {
		return err
	}

	controller.Start()
	controller.Wait()

	return nil
}

The only macguffin-specific line is the one inside the callback, RegisterMacguffinServiceHandler. Everything around it, the dial, the credentials, the mux, is the framework’s. Mount the result under /v1/, register it on the same controller and HTTP server as before, and you’re done.

Delete the duplication

Here’s the satisfying bit. The hand-written REST adapter from part 3, the resthand package, the routes, the toDTO, all of it, comes out. You don’t need it: the gateway serves the same REST surface, backed by the gRPC service, from the proto. The serve command shrinks to one gRPC server and one gateway, and your codebase now has a single place where a macguffin becomes JSON.

See it work

The gateway answers REST, and it’s the same store the gRPC service uses:

$ curl https://localhost:8443/v1/macguffins
{"macguffins":[{"id":"m-1","name":"maltese-falcon","quantity":1}]}

$ curl -X POST https://localhost:8443/v1/macguffins -d '{"name":"the-grail","quantity":1}'
{"id":"m-2","name":"the-grail","quantity":1}

Create over REST, and the macguffin is there over gRPC a moment later, because both are the same implementation over the same Store:

$ grpcurl localhost:50051 macguffin.v1.MacguffinService/ListMacguffins
{ "macguffins": [ { "id": "m-1", ... }, { "id": "m-2", "name": "the-grail", ... } ] }

Errors, and changing them

Error handling comes across too. When a gRPC handler returns a status code, the gateway maps it to the matching HTTP status. The codes.NotFound we returned back in part 2 arrives as a 404, with a JSON error body, and we wrote none of it:

$ curl -s -o /dev/null -w '%{http_code}\n' https://localhost:8443/v1/macguffins/nope
404

That default mapping is the sensible one you’d reach for anyway. A few of the common codes:

gRPC code	HTTP
`InvalidArgument`	400
`Unauthenticated`	401
`PermissionDenied`	403
`NotFound`	404
`AlreadyExists`	409
`Internal`	500
`Unavailable`	503

So the rule of thumb is simply to return the right codes.* from your gRPC handlers, and the REST side gets the right status for free.

When the default shape isn’t what your clients expect, a {"error": {…}} envelope, a trace id header, a tweak to one particular status, you supply your own error handler. The grpc-gateway takes one as a runtime.ServeMuxOption, and gateway.New passes those straight through:

gw, err := gateway.New(ctx, p.Config, register,
	gateway.WithMuxOptions(runtime.WithErrorHandler(myErrorHandler)),
)

myErrorHandler receives the error and the http.ResponseWriter and writes whatever your API promises. That same WithMuxOptions hatch takes the gateway’s other knobs, header matchers, custom marshalers, and the rest; they’re all in the grpc-gateway docs.

Where this leaves us

This is the shape the series was building towards. One domain, one gRPC implementation, one domain → proto mapping you wrote by hand, and a REST API generated from the same proto that needed no second implementation and no second encoding. The things that speak gRPC get gRPC; the browser, the webhook and the curl get JSON; and there’s exactly one place to change when a macguffin grows a field.

Those google.api.http annotations have one more trick in them. They describe your REST API precisely enough to generate an OpenAPI document, and in part 5 we serve that as a live, clickable docs site, from the very same proto.

Nobody's coming to clean your supply chain

Fri, 29 May 2026 00:00:00 +0000

Pick a week in May 2026 and there’s a supply-chain attack in it. On the 11th someone owned TanStack’s CI and pushed 84 poisoned package versions in six minutes. On the 14th, three malicious versions of node-ipc, a library with ten million weekly downloads, shipped an identical credential-stealer. Days later it was @antv, cascading down into a charting library a million projects depend on. Each one runs its payload the moment you install it, then quietly tries to publish itself from your machine.

You’ve heard this story so many times the outrage has worn smooth. So let me point at the one detail that should still make you sit up: the TanStack packages carried valid signing provenance. Real attestation, pointing at the real pipeline. The seal was genuine. The contents were poison.

A signature proves the sender, not the contents

I’ve spent a fair while building integrity and signing into my own tools, so this one stings a little. Signing is a trust mechanism, and a good one. It’s how I prove a binary you downloaded was built and published by me and nobody else, and in a world with this many ways to be impersonated, that matters more than ever.

But TanStack shows the limit in neon. If the pipeline doing the signing is itself compromised, the signature is still perfectly valid. It just now certifies a lie. Provenance answers “did this come from where it claims?” It does not answer “is what’s inside safe?”, and we have spent a few years quietly letting people treat those as the same question.

They aren’t. A signature is a promise about the sender. The thing we actually need is a promise about the contents: that whoever signed has done the diligence, the testing, the vetting, to vouch for what’s in the tin. A signature without that behind it isn’t a safety certificate. It’s a tamper-proof seal on a poisoned jar.

It was never just npm

It’s tempting to file all this under “npm being npm”. Resist it, because it’s a category error. The thing that makes these attacks work, a stranger’s code running on your machine as a side effect of installing or building, is not an npm bug. It’s a near-universal design choice.

Ecosystem	Untrusted code on install/build?	Mechanism
npm (JS)	Yes, at install (dependencies too)	`pre`/`postinstall` scripts
PyPI (Python)	sdist yes, wheel no	`setup.py`; wheels forbid hooks
RubyGems	Yes, at install	native-extension build (`extconf.rb`)
cargo (Rust)	Yes, at build	`build.rs` and proc-macros
Composer (PHP)	Dependencies: no	only the root project’s scripts run, by design
Maven/Gradle (JVM)	Yes, at build	build scripts and plugins
NuGet (.NET)	Modern: no	`install.ps1`, legacy format only
Go (modules)	No	no install or build hooks

(Lifecycle hooks across ecosystems are catalogued at ecosyste.ms if you want the receipts.)

Read that and the lesson isn’t “npm is uniquely bad”, it’s “this was a choice, and several ecosystems chose differently”. Go runs no install or build hooks at all. PHP’s Composer flatly refuses to run a dependency’s scripts, only your own project’s. Python’s wheel format forbids install hooks. The hook was never inevitable.

And yes, that includes my own back yard. cargo’s build.rs is the same gun fired at build time instead of install time, and the TrapDoor campaign used exactly that to rifle through keystores on crates.io this year. Rust isn’t safe here. It’s a smaller, better-policed target, which is a different thing, and I’d rather say so than pretend one of my favourite languages is above it.

No registry can hand you a clean package

Here’s the uncomfortable core. Not one of these registries can guarantee the package you pull is clean. They can sign it, scan it, attest its origin and mandate 2FA on maintainers, and they should do all of that. But none of it is a guarantee, because the failure modes are endless and attackers keep finding new ones. A maintainer account gets phished. A CI token leaks. A trusted contributor turns. A dependency four levels down quietly changes hands.

So the onus lands, and will keep landing for a good while yet, on the consuming engineer. That isn’t a comfortable answer or a clever one. It’s the true one.

And it’s a genuinely rotten spot to stand in, because the advice contradicts itself. Patch slowly and you’re scolded for running known-vulnerable dependencies. Patch the instant a release drops and you’ve skipped the bedding-in that might have caught a poisoned one. There’s no setting on that dial that’s safe, only trade-offs you have to actually think about. Add CI that leaks credentials it never needed, and a dependency tree thousands of strangers deep, and you can see why there’s no single villain to blame and no single switch to flip.

The boring discipline that actually helps

What’s left isn’t heroic, it’s hygiene, and it’s the unglamorous stuff I keep banging on about. Pin your CI actions to commit SHAs so a moved tag can’t swap code under you. Commit your lockfiles. Run the auditors, cargo-audit, pip-audit, govulncheck, npm audit, or Google’s cross-ecosystem OSV-Scanner, on every build. Gate the dependency tree and give every exception an expiry date so “we’ll deal with it later” can’t quietly become “never”. Keep the tree small: every crate you don’t add is a stranger you don’t have to trust.

None of that is a solution. All of it is diligence, and diligence is the only thing that was ever going to stand behind the signature. When I sign a release, the cryptography is the easy part. The promise underneath it, that I pinned, locked, audited, vetted and tested before I put my name on it, is the part worth anything. That’s the contract. The signature is just how I countersign it.

The encouraging note is that the structural defences exist and they work. Go’s checksum database and its refusal to run hooks. Composer declining to trust a dependency’s scripts. Python’s wheels. cargo-vet and cargo-deny giving you somewhere to record human judgement at scale. More ecosystems should steal these shamelessly, because a registry that makes the safe path the default does the working engineer a far bigger favour than one that leaves it all to discipline.

The same shape, a third time

If this feels familiar, it should. I wrote recently about a bug bounty that collapsed because the cost of slop was deferred, and about a junior pipeline being cut because the bill lands years later. Supply-chain security is the same shape a third time. The convenience is now, the catastrophe is later, and the only thing standing in the gap is an engineer paying attention, doing the dull work, refusing to be rushed into trusting something they haven’t checked.

There is no clean package waiting to be found, no registry about to solve this for us, no signature that means “safe” all on its own. There’s the diligence you do before you put your name to something, and the judgement to know when an install is asking you to trust more than you should. For a good while yet, that is the whole job. Boring, unfashionable, and the only thing that works.

Building a web service with go-tool-base, part 3.5: the same server, now a website

Thu, 28 May 2026 00:00:00 +0000

The HTTP server from part 3 serves JSON. But net/http doesn’t care what you hand it: HTML, an image, a stylesheet, a whole little site, it’s all just bytes with a content type. So before we get back to the API in part 4, a short detour to prove the point and pick up a couple of genuinely useful tools: we’ll turn the macguffin service into a tiny website.

This is a bonus, off to the side of the API arc, but it earns its place. Real services nearly always grow a bit of HTML eventually: a status page, a landing page, a small admin view, an embedded docs site (we’ll do exactly that in part 5). The mechanics are the same every time, and worth having in hand.

A page from `html/template`

Go’s html/template renders HTML from a template and your data, and it escapes that data on the way out, so a macguffin called <script> becomes text rather than a problem. Here’s a page that lists the catalogue:

<!-- internal/site/templates/index.html -->
<!doctype html>
<html lang="en">
 <head>
 <meta charset="utf-8" />
 <title>Macguffins</title>
 <link rel="stylesheet" href="/static/style.css" />
 </head>
 <body>
 <h1>The macguffin catalogue</h1>
 <ul>
 {{range .}}
 <li>{{.Name}} <span class="qty">&times;{{.Quantity}}</span></li>
 {{end}}
 </ul>
 </body>
</html>

{{range .}} walks the slice we pass in, and {{.Name}} / {{.Quantity}} read each macguffin’s fields. The data is the same Store from part 2, so the page is a view onto the very same domain the gRPC and JSON APIs serve.

Shipping the files inside the binary

A template and a stylesheet are files, and you do not want to deploy a folder of loose assets next to your binary and hope they line up. Go’s embed package bakes them into the binary at build time, so the whole thing ships as one file.

// internal/site/site.go

package site

import (
	"embed"
	"html/template"
	"io/fs"
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

//go:embed templates static
var content embed.FS

var tmpl = template.Must(template.ParseFS(content, "templates/index.html"))

type Site struct {
	store *macguffin.Store
}

func New(store *macguffin.Store) *Site {
	return &Site{store: store}
}

func (s *Site) Routes() *http.ServeMux {
	static, err := fs.Sub(content, "static")
	if err != nil {
		panic(err) // the embedded path is a compile-time constant
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /{$}", s.index)
	mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.FS(static))))

	return mux
}

func (s *Site) index(w http.ResponseWriter, _ *http.Request) {
	if err := tmpl.Execute(w, s.store.List()); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Three things are doing the work. //go:embed templates static pulls both folders into the content filesystem. template.ParseFS parses the page from it once at startup. And http.FileServer(http.FS(static)) serves the stylesheet (and anything else under static/) straight from the embedded files, with content types set for you, so GET /static/style.css comes back as text/css.

The GET /{$} pattern is worth a note: the {$} anchors it to the exact root path, so / renders the page but /anything-else doesn’t accidentally match it.

If you’d rather edit templates without rebuilding during development, swap the embedded filesystem for the real one: http.FileServer(http.Dir("static")), and template.ParseGlob instead of ParseFS. Embed for release, disk for the edit-refresh loop; the handlers don’t change.

On the same server

Routes() hands back a *http.ServeMux, which is an http.Handler, so it registers exactly like the JSON API did, on the same controller, with the same TLS:

// pkg/cmd/serve/main.go (or a dedicated command)

if _, err := gtbhttp.Register(ctx, "site", controller, p.Config, p.Logger,
	site.New(store).Routes()); err != nil {
	return err
}

Because the certificate is the mkcert one from part 2, opening https://localhost:8443/ renders the page, stylesheet and all, with a clean padlock on any machine that trusts your local CA (where you ran mkcert -install); anywhere else, the browser flags the cert, exactly as it should.

The same hardened server, the same graceful shutdown, the same /healthz, now serving a website instead of (or alongside) JSON.

Back to the API

That’s the whole trick: the HTTP server is just net/http, and it will serve whatever you point it at, escaped and content-typed properly, shipped inside the binary. We’ll use exactly this in part 5 to serve interactive API docs.

Detour over. In part 4 we get back to the API and finally deal with that duplicated REST layer, the one we wrote twice and promised to delete.

Building a web service with go-tool-base, part 3: a REST service, by hand and by spec

Wed, 27 May 2026 00:00:00 +0000

The gRPC service from part 2 is the right core for service-to-service traffic. It is also useless to a great many of the things that might want to call it: a browser, a webhook from some SaaS, a partner who will not touch protobuf, a curl in a 2am runbook. They all want the same thing, plain JSON over HTTP.

So we need a REST face as well. The only way we know how so far is to build one, a second implementation of the macguffin service, this time over HTTP. We’ll do it two ways, by hand and from a spec, and wire it onto the very same controller from part 1. And yes, we’ll feel the duplication, because the next part is about making it disappear.

Why not just `net/http`?

A fair question before we register anything: go-tool-base’s HTTP server is net/http underneath, so why not stand up a http.Server{} yourself? You can, and nothing here hides the standard library from you. What gtbhttp.Register adds is the dull, easy-to-botch scaffolding that goes around the handler:

It’s still net/http. You write ordinary http.Handlers. There’s no bespoke router and no framework to learn underneath, just the standard library doing the serving.
Timeouts and limits you’d otherwise forget. A bare http.Server{} has no ReadTimeout or WriteTimeout, which is a Slowloris waiting to happen. You get sane read/write/idle timeouts and caps on header and body sizes.
The hardened TLS from part 2, the same shared certificate and the same 1.2-minimum AEAD config, with no extra setup.
Lifecycle for free. It registers on the same controller as your gRPC server, so a SIGTERM drains in-flight requests and stops both together, instead of you re-writing the signal-and-Shutdown dance for every service.
Health endpoints, /healthz, /livez and /readyz, backed by the controller and ready for an orchestrator to probe, with nothing to write.

You write the handlers; it owns the lifecycle and the hardening. With that settled, let’s write some handlers.

By hand, on the standard library

net/http’s ServeMux acquired method and path patterns in Go 1.22, so a small REST surface needs no router and no dependencies at all. And the hard part is already behind us: the Store from part 2 is our domain, and the HTTP handlers are just another thin adapter over it, exactly as the gRPC server was. A small type holds the store, and the routes hang off it:

// internal/resthand/server.go

package resthand

import (
	"encoding/json"
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

type API struct {
	store *macguffin.Store
}

func (a *API) Routes() *http.ServeMux {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /macguffins", a.list)
	mux.HandleFunc("GET /macguffins/{id}", a.get)
	mux.HandleFunc("POST /macguffins", a.create)

	return mux
}

Each pattern names a method and a path, and {id} is a named wildcard a handler reads back with r.PathValue("id"), no third-party router required. And because the domain Macguffin already carries JSON tags, this adapter can encode it straight to the response, with no separate type to map to:

// internal/resthand/server.go (same file)

func (a *API) get(w http.ResponseWriter, r *http.Request) {
	m, ok := a.store.Get(r.PathValue("id"))
	if !ok {
		http.Error(w, "macguffin not found", http.StatusNotFound)
		return
	}

	writeJSON(w, http.StatusOK, m)
}

list and create are the same shape: ask the store, encode the result with a small writeJSON helper. Routes() hands back a *http.ServeMux, which is itself an http.Handler, so it slots straight into the controller in a moment.

There’s nothing clever in any of it, and that’s the appeal: total control, no tooling, and because we serve the domain type directly, nothing to map. The cost is the kind that creeps up on you. Every route, every bit of marshalling, every status code is yours to write and keep correct, and as the surface grows, so does the area for small mistakes.

By spec, with oapi-codegen

The other road is the OpenAPI mirror of part 2’s proto. You describe the API in an OpenAPI document, and a generator turns it into a Go interface for you to implement. The tool is oapi-codegen.

First the contract, api/macguffin.openapi.yaml (trimmed to one path here):

# api/macguffin.openapi.yaml
paths:
 /macguffins/{id}:
 get:
 operationId: getMacguffin
 parameters:
 - name: id
 in: path
 required: true
 schema: { type: string }
 responses:
 "200":
 description: OK
 content:
 application/json:
 schema:
 $ref: "#/components/schemas/Macguffin"
 "404":
 description: Not found
components:
 schemas:
 Macguffin:
 type: object
 required: [id, name, quantity]
 properties:
 id: { type: string }
 name: { type: string }
 quantity: { type: integer, format: int32 }

Install the tool and tell it what to emit with a small config. We want the standard-library server, so std-http-server:

go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@latest

# api/oapi-codegen.yaml
package: restapi
output: internal/restapi/macguffin.gen.go
generate:
 std-http-server: true
 models: true

As in part 2, wire it into go generate so it can’t drift, this time sitting right alongside the buf directive:

// gen.go (at your module root): add the second line alongside part 2's

//go:generate buf generate
//go:generate oapi-codegen -config api/oapi-codegen.yaml api/macguffin.openapi.yaml

go generate ./... now regenerates both the gRPC and the REST code in one go. What oapi-codegen writes is the message models and, the important part, a ServerInterface, one method per operation:

// internal/restapi/macguffin.gen.go (generated)

type ServerInterface interface {
	ListMacguffins(w http.ResponseWriter, r *http.Request)
	CreateMacguffin(w http.ResponseWriter, r *http.Request)
	GetMacguffin(w http.ResponseWriter, r *http.Request, id string)
}

If that feels familiar, it should: it’s the exact shape from part 2. The generator hands you an interface and your job is a type that honours it, an adapter over the same Store. But here’s the wrinkle that earns this part its keep. oapi-codegen generated its own Macguffin type, a DTO, so the adapter has to map the domain into it. (The path parameter, at least, arrives already typed as id string rather than fished out by hand.)

// internal/restapi/server.go

package restapi

import (
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func (a *API) GetMacguffin(w http.ResponseWriter, _ *http.Request, id string) {
	m, ok := a.store.Get(id)
	if !ok {
		http.Error(w, "macguffin not found", http.StatusNotFound)
		return
	}

	writeJSON(w, http.StatusOK, toDTO(m))
}

// toDTO maps the domain type to oapi-codegen's generated Macguffin DTO.
func toDTO(m macguffin.Macguffin) Macguffin {
	return Macguffin{Id: m.ID, Name: m.Name, Quantity: m.Quantity}
}

That toDTO is the second mapping of this kind we’ve written. Part 2’s gRPC adapter had toProto; this one has toDTO. The same domain data, encoded twice, into two generated shapes, kept in step by hand. Hold that thought: part 4 is where that second mapping stops being something you write.

A generated helper turns your implementation into an http.Handler:

handler := restapi.HandlerFromMux(restapi.NewAPI(store), http.NewServeMux())

So which road? By hand when it’s a handful of endpoints you fully control and you’d rather not own a generator. Spec-first when the contract carries weight, several teams consume the API, the shape changes often, or you simply want the OpenAPI document to exist, which (spoiler) we’ll be serving as live, clickable docs in part 5.

Wire it onto the controller

Whichever road you took, you’re holding an http.Handler. We don’t replace the gRPC server from part 2; we add the HTTP one beside it, on the same controller, both reading from one shared store. This is the controller from part 1 doing exactly what it exists for: two transports, one lifecycle.

// pkg/cmd/serve/main.go

package serve

import (
	"context"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	gtbhttp "gitlab.com/phpboyscout/go-tool-base/pkg/http"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
	"gitlab.com/myorg/macguffinsvc/internal/resthand"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	// One shared domain behind both transports.
	store := macguffin.NewStore()

	// gRPC, from part 2.
	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	// HTTP, new this part. (Or the oapi-codegen handler; either is an http.Handler.)
	handler := resthand.New(store).Routes()
	if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, handler); err != nil {
		return err
	}

	controller.Start()
	controller.Wait()

	return nil
}

The HTTP Register is the counterpart of part 2’s gRPC one: same controller, so a single SIGTERM drains and stops both together and /healthz reports on both. The HTTP server reads server.http.port, the gRPC server server.grpc.port from part 2, and both take their TLS from the shared server.tls block, the one certificate:

server:
 grpc:
 port: 50051
 reflection: true
 http:
 port: 8443
 tls:
 enabled: true
 cert: ./localhost+2.pem
 key: ./localhost+2-key.pem

A browser away

Here’s where the mkcert groundwork from part 2 pays off. Because that certificate is signed by a CA your machine already trusts, the service answers over HTTPS with no --cacert flag and, more to the point, no browser warning:

$ curl https://localhost:8443/macguffins
[{"id":"m-1","name":"maltese-falcon","quantity":1}]

$ curl https://localhost:8443/macguffins/m-1
{"id":"m-1","name":"maltese-falcon","quantity":1}

And /healthz now reports both transports, since they registered against the one controller:

$ curl https://localhost:8443/healthz
{"overall_healthy":true,"services":[{"name":"grpc","status":"OK"},{"name":"http","status":"OK"}]}

Open https://localhost:8443/macguffins in an actual browser and it just loads, green padlock and all. That matters more than it sounds, and it’s the reason we set the local CA up early: in part 5 the API docs are a web page, and a docs page behind a cert warning is a docs page nobody trusts.

Where this leaves us

Step back and count what we actually built. There’s one domain, the Store, and it didn’t change at all this part. What we added was a second delivery adapter over it: routing, marshalling, and, on the spec-first road, a toDTO mapping sitting beside part 2’s toProto. The same data, encoded into two generated shapes, kept in step by hand. That’s the real duplication, not the logic, but the transport scaffolding wrapped around it.

And it’s the itch the next part scratches. We built a second transport and a second encoding of the same domain, and kept them in step by hand. Part 4 is where that whole second layer, adapter and encoding both, stops being something you write at all. We felt the cost first on purpose; now we get to remove it.

The greybeards' edge was never typing

Wed, 27 May 2026 00:00:00 +0000

I have a retirement plan, and it is gloriously low-tech. A cabin, some trees, a woodstove, and a firm rule that no wifi symbol ever appears within a mile of me again. I think about it more than is probably healthy.

There’s a snag, though, and it’s the same one the whole industry is currently pretending it can’t see. For me to vanish into the woods, somebody has to be able to do my job after I’ve gone. And right now, collectively, we are working very hard to make sure nobody can.

The boost, and the drag

I wrote the other day about how AI made producing plausible work nearly free while verifying it stays expensive and human. Point that same lens at a team and something uncomfortable falls out. It isn’t mine; it belongs to Mark Russinovich and Scott Hanselman of Microsoft, who laid it out in Communications of the ACM: agentic coding tools give a senior engineer an AI boost, multiplying what they ship, because a senior has the judgement to steer and verify the output. The same tools give an early-career engineer an AI drag, because they don’t have that judgement yet, and the machine hands them far more rope than they can hold.

The cold incentive writes itself, and they name it: hire seniors, automate juniors. It isn’t hypothetical, either. Meta cut 8,000 roles last week, in a round the Times filed under mounting AI casualties. For any single quarter you care to look at, the maths is impeccable.

The bill is just deferred

Here’s the line the spreadsheet leaves off. The grindy, unglamorous work a junior used to cut their teeth on, the small fixes, the boring migrations, the read-the-stack-trace-and-figure-it-out, is exactly the work AI now does. So the proving ground is gone. And the entry-level seats where they’d have stood on it are the ones being cut. Squeezed from both ends at once: no reps, and nowhere to take them.

Russinovich and Hanselman put the consequence plainly. Without early-career hiring the talent pipeline collapses, and you arrive at a future with no next generation of experienced engineers. The seniors you’ll be desperate for in 2032 are the juniors you declined to train in 2026. The bill doesn’t vanish. It just falls due long after the people who cut the cheque have moved on.

How to manufacture a world of AI slop

I named the last piece for its villain; let me name this one’s too. Raise a generation that can produce with AI but was never taught to validate, and here is what you get: people shipping machine-built products at speed with no instinct for where the output is quietly wrong, because they never had to be wrong the slow way first. Software nobody genuinely understands, human-written and AI-written alike, and a steady leak of trust out of all of it.

That isn’t a productivity problem. That’s a world of AI slop, and not in one project’s inbox this time but everywhere at once. We’d have automated our way clean out of the one job AI cannot do for us: knowing when not to trust the machine.

It’s a choice, and it’s yours

Andrew Murphy put it with more bite than I’d quite dare: AI didn’t kill your junior pipeline, you did. He’s right. This isn’t weather. Nobody is making you do it. It’s a decision, taken quarter by quarter, and a decision is a thing you can take differently.

The fix isn’t complicated, it’s just unfashionable. Keep hiring early-career engineers. Say out loud that they cost you capacity at first, and treat their growth as an actual goal rather than something meant to happen by osmosis. Russinovich and Hanselman call it preceptorship at scale: senior mentorship, deliberately structured, turning the ordinary day’s work into teachable moments.

And the proving ground can be rebuilt, just not where it stood. If AI does the writing now, the apprenticeship moves to the reviewing. Put juniors in the loop on the machine’s output and have them hunt for the subtle wrongness, the way a scanner is an argument, not an order. That’s how judgement gets built now: not by grinding out the work, but by verifying it. Which, as luck would have it, is the single most valuable thing anyone on your team can learn to do.

The part that’s on the greybeards

This is where I stop letting the companies wear all the blame, because some of it is mine, and yours. Verification is a craft, and crafts pass from person to person or not at all. I know where every one of my own AI misfires comes from: I gave it too little context, or too much rope, and didn’t check the result closely enough. The tool rarely went rogue. The gap was always my diligence. That’s not a confession, it’s the curriculum, and it’s precisely the judgement a junior can only earn by sitting in the loop beside someone who has already made those mistakes.

So the senior engineer’s job has quietly changed underneath us. It was never really the typing. It was knowing when something is off, and what the customer actually needs, and now it is also handing that on, deliberately, while there’s still time to. Mentor and guardian first; fastest prompt in the room a distant second.

The ladder you’re standing on

There will always be something AI can’t do well enough, and for a good while yet it’s the thing that matters most: being the accountable human who genuinely understands what’s needed and can be held to it when it goes wrong. A simulation can be enormously convincing. It cannot be responsible.

Which brings me back to my cabin. I do want it one day, the trees and the woodstove and the blissful disconnection. But I only get to go if the work outlives me, and the work only outlives me if the people do. So the last useful thing my generation does, before we shuffle off to find our trees, isn’t shipping a little more code. It’s making sure there’s somebody left who can tell when the machine is wrong. Pull the ladder up behind us and there’ll be nobody to notice the rot, and no cabin quiet enough to make that sit right.

AI didn't kill curl's bug bounty. The bounty did.

Tue, 26 May 2026 00:00:00 +0000

In January, Daniel Stenberg shut down curl’s bug bounty. The headlines wrote themselves, and they all said the same thing: AI killed it. A flood of machine-generated slop drowned the maintainers, so they pulled the plug.

That’s true, as far as it goes. It’s also the wrong lesson, and the right one is sitting in plain sight in the same project, in the same few months.

Volume without validation is the attack

curl had run its bounty since April 2019. Over its life it paid out more than $100,000 for 87 genuine vulnerabilities, a thoroughly good return for one of the most depended-on pieces of software on the planet. Then the reports stopped being reports. The confirmation rate, the share of submissions that turned out to be a real bug, had historically sat north of 15%. By 2025 it was below 5%. Fewer than one in twenty submissions were worth anything, and the rest still had to be read.

That last part is the whole problem. A bogus report doesn’t announce itself. Someone has to open it, take it seriously, try to reproduce it, and work out that it’s nonsense, and that someone is a human being with a finite number of hours and a project to run. Stenberg put it plainly: the slop “take[s] a serious mental toll to manage and sometimes also a long time to debunk.” The submitter spends seconds. The maintainer spends an afternoon. Do that at volume and it stops being noise and becomes an attack, a denial-of-service aimed not at curl’s servers but at its maintainers’ attention. No exploit required. Just plausibility, in bulk.

The bounty was the accelerant, not the AI

So far this is the story everyone tells. Here’s where I get off the bus.

The instinct is to blame the AI for the slop. But look at what a bounty actually is. It’s a cash prize, and curl’s was priced for the thing it wanted: the hours and the judgement a skilled human pours into finding a real flaw. That pricing made complete sense right up until the cost of producing something that looked like a finding collapsed to nearly nothing.

That’s what AI changed. Not the supply of bugs. The supply of plausible-looking bug reports. Put a cash prize on “looks like a finding”, then make “looks like a finding” free to generate, and you haven’t got a bug bounty any more. You’ve got a slot machine. Stenberg said he’d started to sense “a bad faith attitude” in the reports, and of course he had. The incentive was openly inviting it.

So the death spiral was structural, not bad luck. The moment generating plausible reports went free, any cash bounty became a magnet for spray-and-pray, and the only open questions were how fast it would rot and whether you’d close the programme or just let the rewards quietly wither. The AI was the match. The bounty was the petrol. We have been pointing at the wrong one.

The proof: curl turned around and hired the AI

If AI were really the villain here, you’d expect curl to have slammed the door on it. It did the opposite.

In the same stretch, by AISLE’s own account, an AI security platform contributed 24 pull requests to curl, five of which earned CVEs, and the project now runs it internally for continuous review. The same tooling reportedly found all twelve zero-days in an OpenSSL release in late January. (Both of those are the tool-makers’ and a third party’s numbers rather than curl’s audited figures, so weigh them as such. But curl adopting the thing isn’t a claim. It’s a decision.)

Sit with the shape of that. curl shut down strangers being paid for AI-shaped noise, and in the same breath put AI to work as a tool its own maintainers drive. The two moves look contradictory only if you think “AI” is a single thing with a single verdict attached. It isn’t. Pointed at the problem by people accountable for the result, with no prize to farm, it found real bugs. Dangled in front of anonymous strangers chasing a payout, it produced sand.

The tell is which AI curl kept, and which it mocked

Stenberg drew that line about as sharply as a person can. When Anthropic put its security model, Mythos, in front of curl this spring, it scanned 176,000 lines of C and surfaced a single flaw, and Stenberg called the surrounding fanfare the greatest marketing stunt he’d seen. Same maintainer. Adopts one AI, rubbishes another.

The deciding factor was never whether the thing was AI. Both were. It was whether the output survived a human checking it, and whether you could check it at all. AISLE handed over pull requests and CVEs you could read and merge. Mythos arrived as a closed model and a press release, which is to say a claim the community has no way to independently test.

My bias, up front, because it runs the opposite way to what you’d expect from someone writing this: I’m a paying Claude subscriber and I lean on Anthropic’s models every working day, the one behind the spadework for this post included. I’m an advocate, not a sceptic, and AI genuinely has its place. That is exactly why the Mythos fanfare grates. Overselling a closed model to get out ahead of the competition, when the one test the public got to see turned up a single bug, is the sort of thing that chips away at trust in all of it. A result you can’t verify is marketing until proven otherwise, whoever’s logo is on the slide, and I’d rather the tools I depend on didn’t stoop to it.

The cheap half and the expensive half

Pull back from curl for a moment, because the lesson isn’t really about bounties at all. Anyone who works with these tools every day knows the same thing: when they go wrong, it’s rarely the model running off on its own. It’s the context it wasn’t given, the rope it was handed, the output nobody checked closely enough. The failure sits on the human side of the keyboard, at the one step that’s easiest to skip, which is verification.

That’s the pattern curl hit at the scale of an ecosystem. AI made one thing nearly free: producing work that looks right. It did not make the other thing a penny cheaper: confirming the work is right. That cost still falls, in full, on a person. (A scanner, I’ve argued before, is an argument, not an order; the same goes double for a model.) The bounty’s fatal mistake was paying for the cheap half and quietly assuming it had bought the expensive one. The same trap waits in code review, in hiring, in CVs read by machines, but that’s a bigger argument for another post.

Pouring sand into the machine

curl didn’t capitulate to AI, whatever the headlines decided. It stopped paying for the worthless half and started using the valuable half, and it had the discernment to tell a useful tool from a press release while it did so.

The bounty wasn’t a casualty of artificial intelligence. It was a structure that, the instant plausible output became free, could only fill with sand. Stenberg said he hopes closing it stops “more people pouring sand into the machine.” Reading the last year of his inbox, I think he’ll get his wish. The sand was only ever there because somebody left a bucket of money beside the funnel.

Building a web service with go-tool-base, part 2: a gRPC service, with TLS

Tue, 26 May 2026 00:00:00 +0000

The heartbeat from part 1 runs, ticks along, and shuts down politely when you ask it to. It also talks to absolutely no one. A service people can actually call needs an API, and for a typed, fast, streaming-capable one, gRPC is the obvious first move.

The catch is that a production-grade gRPC server is rather more than grpc.NewServer(). You want health checks an orchestrator understands, reflection so you can poke at it without the .proto file in hand, a graceful shutdown that doesn’t guillotine calls that are still in flight, and TLS, which is where most people’s first attempt quietly goes wrong. The good news: part 1 already gave us the thing that carries all of it. A gRPC server is just another service to register on the controller.

Why gRPC, and not just REST

Worth a moment on why we’re reaching for gRPC at all, because for plenty of services a plain JSON-over-HTTP API is the right call and less faff. gRPC earns its place when a few of these matter:

A contract that’s enforced, not hoped for. The .proto is the single source of truth, and both ends are generated from it. You don’t hand-write JSON marshalling, and you don’t find out at runtime that the client and server disagree about a field’s type. Evolve the schema carefully (add fields by number) and old clients keep working.
Clients in any language, for free. The same .proto generates a Go server and a Python, TypeScript, Rust or Java client with nobody writing an SDK by hand. For an internal service that several teams call, that one point can decide it.
It’s built for service-to-service traffic. Binary protobuf is smaller and quicker to encode than JSON, calls multiplex down a single HTTP/2 connection, and streaming (from the client, the server, or both at once) is a first-class thing rather than something you bolt onto REST with websockets.
Deadlines, cancellation and a health protocol come built in, rather than conventions you reinvent for every service.

The trade-offs are real. A browser doesn’t speak gRPC natively, and a binary protocol is fiddlier to poke at than a JSON endpoint you can curl (which is exactly why reflection and grpcurl exist). That’s not a reason to avoid it; it’s the reason this series doesn’t stop at gRPC. In part 4 we put a REST/JSON face on this very service, so the things that call it get the typed, fast core and the things that can’t speak gRPC still get a friendly HTTP surface. You don’t have to pick a side.

Define the contract

gRPC starts with a schema. Here’s a small macguffin service, macguffin.proto:

// proto/macguffin/v1/macguffin.proto

syntax = "proto3";

package macguffin.v1;

option go_package = "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1;macguffinv1";

service MacguffinService {
 rpc GetMacguffin(GetMacguffinRequest) returns (Macguffin);
 rpc ListMacguffins(ListMacguffinsRequest) returns (ListMacguffinsResponse);
 rpc CreateMacguffin(CreateMacguffinRequest) returns (Macguffin);
}

message Macguffin {
 string id = 1;
 string name = 2;
 int32 quantity = 3;
}

message GetMacguffinRequest { string id = 1; }
message ListMacguffinsRequest { int32 page_size = 1; }
message ListMacguffinsResponse { repeated Macguffin macguffins = 1; }
message CreateMacguffinRequest { string name = 1; int32 quantity = 2; }

From proto to Go

If gRPC in Go is new to you, this is the part that catches people out: you don’t write the Go for those messages and that service interface, you generate it from the .proto. The proto is the source of truth; a compiler turns it into Go you import and build against. Same goes for a client in any other language, all from the same file.

That compiler is protoc, and on its own it’s a faff. You install it, then a separate plugin for each output you want (protoc-gen-go for the message types, protoc-gen-go-grpc for the client and server stubs), keep their versions in step, and drive the lot with a command line of -I include paths and --*_out flags that’s easy to get subtly wrong.

buf is the friendlier way to run exactly that. It wraps protoc and its plugins behind a couple of small config files, handles the plugin versions, and turns that gnarly invocation into a single word. It’s become the usual way to work with protobuf in Go, and it’s what we’ll use here.

At a minimum you need three binaries on your PATH: buf itself, and the two plugins it drives. go install is the quickest way to get them:

go install github.com/bufbuild/buf/cmd/buf@latest
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Those land in $(go env GOPATH)/bin, so make sure that’s on your $PATH. Then describe what you want generated in a buf.gen.yaml:

# buf.gen.yaml
version: v2
plugins:
 - local: protoc-gen-go
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-go-grpc
 out: internal/gen
 opt: paths=source_relative

Each part of that earns its place. version: v2 is buf’s config format. The plugins list names the generators to run, and we run two, because gRPC in Go arrives in two halves:

protoc-gen-go turns the messages into Go structs, the Macguffin type and the request and response types, in a macguffin.pb.go file.
protoc-gen-go-grpc turns the service into the client and server scaffolding, in a macguffin_grpc.pb.go file, including the MacguffinServiceServer interface you’re about to implement.

out: internal/gen is where the files land, and paths=source_relative lays them out mirroring the proto’s own folders (so proto/macguffin/v1/... becomes internal/gen/macguffin/v1/...) rather than deriving the path from the go_package line. Then run it:

buf generate

Both files appear under internal/gen/macguffin/v1, and we’re ready to write the implementation.

Running that by hand once is fine; remembering to run it every time the .proto changes is where it goes wrong, and the generated code quietly drifts out of step, usually right before a demo. Wire it into go generate instead. Drop a one-line directive in a file at your module root, say gen.go:

// gen.go (at your module root)

package macguffinsvc

//go:generate buf generate

Now go generate ./... regenerates everything from the proto, and it’s the same one command for any other generator you add later. Run it whenever the schema changes, and in CI if you want to catch a stale checkout.

If OpenAPI is your map of the territory

If your mental model of an API contract is an OpenAPI (Swagger) document, a .proto is the same idea wearing fewer clothes: a typed, language-neutral description of a service that both ends generate from. The thing you notice first is how much less of it there is. Here’s that Macguffin message again, and the same shape written as an OpenAPI schema:

message Macguffin {
 string id = 1;
 string name = 2;
 int32 quantity = 3;
}

Macguffin:
 type: object
 properties:
 id:
 type: string
 name:
 type: string
 quantity:
 type: integer
 format: int32

And that pattern holds across the whole service. The proto above, three calls and five messages, is about twenty lines. Describe the same surface in OpenAPI and you’re closer to a hundred, because OpenAPI also pins down the HTTP verbs, paths, status codes and content types: the transport details a proto leaves out on purpose. That isn’t OpenAPI being bloated; it’s describing more. But when the contract is the thing you care about, the proto says it with less ceremony, and it doesn’t wed your API to HTTP, which is exactly what lets us serve this same service over gRPC now and REST later. (We’ll generate a real OpenAPI document from this proto in part 5, for the readers who still want one.)

Implement it

Generating the code gave you the message types and, more to the point, an interface to satisfy. Open macguffin_grpc.pb.go and you’ll find MacguffinServiceServer, one method per RPC in the proto:

// internal/gen/macguffin/v1/macguffin_grpc.pb.go (generated)

type MacguffinServiceServer interface {
	GetMacguffin(context.Context, *GetMacguffinRequest) (*Macguffin, error)
	ListMacguffins(context.Context, *ListMacguffinsRequest) (*ListMacguffinsResponse, error)
	CreateMacguffin(context.Context, *CreateMacguffinRequest) (*Macguffin, error)
	mustEmbedUnimplementedMacguffinServiceServer()
}

That interface is the server-side contract. Each method takes the request message you defined and hands back the response message, plus an error. Writing the type that honours it, the actual logic behind each call, is the part that’s yours: the proto pins down the shape of the conversation, and this is what the service actually does when one happens.

The one curious line is mustEmbedUnimplementedMacguffinServiceServer(). Alongside the interface, buf generated an UnimplementedMacguffinServiceServer struct with a do-nothing stub for every method, and you embed it in your own type. It earns its keep twice over. It satisfies that unexported method, so your type counts as a real implementation. And it future-proofs you: add a new RPC to the proto later and your existing server still compiles, falling back to the stub (which returns a clean “unimplemented” error) until you write the real method.

Before we satisfy that interface, one separation worth making up front. The gRPC server is a delivery mechanism, not the place the data lives. If we stuff the map of macguffins straight inside it and then build an HTTP server next part, we’d have two servers each hoarding their own copy of the same data. So keep the domain, the macguffins and what you can do with them, in its own type, and let each transport be a thin layer over it.

Here’s that domain: an in-memory store standing in for the repository a real service would have. Nothing in it knows about gRPC, HTTP, or any wire format.

// internal/macguffin/store.go

package macguffin

import "sync"

// Macguffin is the domain type. The JSON tags will let a hand-written HTTP
// handler serve it directly in part 3.
type Macguffin struct {
	ID string `json:"id"`
	Name string `json:"name"`
	Quantity int32 `json:"quantity"`
}

type Store struct {
	mu sync.Mutex
	items map[string]Macguffin
	seq int
}

func (s *Store) Get(id string) (Macguffin, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()

	m, ok := s.items[id]

	return m, ok
}

List and Create are the same shape, and NewStore seeds it with a single maltese-falcon. Now the gRPC server is thin: it embeds the stub, holds a store, and each method calls the store and translates the result into the generated protobuf type.

// internal/grpcsvc/server.go

package grpcsvc

import (
	"context"

	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

type Server struct {
	macguffinv1.UnimplementedMacguffinServiceServer

	store *macguffin.Store
}

func New(store *macguffin.Store) *Server {
	return &Server{store: store}
}

func (s *Server) GetMacguffin(_ context.Context, req *macguffinv1.GetMacguffinRequest) (*macguffinv1.Macguffin, error) {
	m, ok := s.store.Get(req.GetId())
	if !ok {
		return nil, status.Errorf(codes.NotFound, "macguffin %q not found", req.GetId())
	}

	return toProto(m), nil
}

// toProto maps the domain type to the generated protobuf DTO.
func toProto(m macguffin.Macguffin) *macguffinv1.Macguffin {
	return &macguffinv1.Macguffin{Id: m.ID, Name: m.Name, Quantity: m.Quantity}
}

ListMacguffins and CreateMacguffin are the same: call the store, map the result. The one habit worth keeping is to return real gRPC status codes (codes.NotFound here) rather than bare errors, so callers get something they can branch on.

That toProto step is worth a second look, because it comes back round later. The domain has one shape, the proto has its own generated Macguffin, so the adapter maps between them. It’s a small price for a single transport. In part 3 we add a second transport with its own generated type and pay that price again, and part 4 is where we stop paying it twice.

Wire it onto the controller

This is the part that earns its keep. First, generate a serve command, the same way the CLI series generated its commands:

gtb generate command \
 --name serve \
 --short "Run the macguffin service"

That scaffolds two files: pkg/cmd/serve/cmd.go (generated, and wired into your command tree for you) and pkg/cmd/serve/main.go, which holds a RunServe function for your logic. Fill it in:

// pkg/cmd/serve/main.go

package serve

import (
	"context"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	store := macguffin.NewStore()

	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	controller.Start()
	controller.Wait()

	return nil
}

That’s the whole server. gtbgrpc.Register does four things in one call: it builds a *grpc.Server, wires the standard gRPC health service to the controller’s health reports (the ones we met in part 1), registers Start, Stop and Status against the controller so the lifecycle is handled, and hands you back the server to register your own service on, which is the RegisterMacguffinServiceServer line. After that it’s the same Start() / Wait() we used for the heartbeat.

It reads its port from config (server.grpc.port, falling back to server.port), so a minimal config is:

server:
 grpc:
 port: 50051
 reflection: true

Poke it

Build, run mytool serve, and reach for grpcurl. Reflection is on, so you don’t need the .proto to hand:

$ grpcurl -plaintext localhost:50051 list
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection
macguffin.v1.MacguffinService

$ grpcurl -plaintext localhost:50051 macguffin.v1.MacguffinService/ListMacguffins
{
 "macguffins": [
 { "id": "m-1", "name": "maltese-falcon", "quantity": 1 }
 ]
}

And the health service is already answering, wired straight to the controller, without you registering a thing:

$ grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
{ "status": "SERVING" }

That’s the lifecycle work from part 1 paying out: the controller’s health is the gRPC health, and a SIGTERM still drains and stops the server cleanly.

Now turn on TLS

Here’s the bit people brace for. Plaintext gRPC is fine on a laptop and unacceptable the moment it leaves one. With go-tool-base it’s a config change, not a code change.

The fiddly part of local TLS is usually the certificate. A hand-rolled self-signed one means passing a -cacert to every client and clicking past browser warnings. mkcert makes that go away: it creates a local certificate authority and installs it into your system’s (and your browser’s) trust stores, so anything it signs is simply trusted. Set the CA up once:

mkcert -install

Then mint a certificate for the names the service answers on:

mkcert localhost 127.0.0.1 ::1

That writes localhost+2.pem (the certificate) and localhost+2-key.pem (the key), signed by your now-trusted local CA. Doing this properly now pays off later: in part 3 the HTTP server, and in part 5 the API docs in a browser, both lean on that certificate being trusted with no warning.

Point the tool’s config at the pair, under the shared server.tls block:

server:
 grpc:
 port: 50051
 reflection: true
 tls:
 enabled: true
 cert: ./localhost+2.pem
 key: ./localhost+2-key.pem

No code changes. Run mytool serve again and it comes up over TLS:

INFO starting gRPC server tls=true addr=:50051

Because the certificate is signed by a CA your machine already trusts, the client needs no extra flags:

$ grpcurl localhost:50051 macguffin.v1.MacguffinService/GetMacguffin -d '{"id":"m-1"}'
{ "id": "m-1", "name": "maltese-falcon", "quantity": 1 }

A plaintext client is now refused, as it should be. (In production you’d point those same two config keys at whatever your real CA issues; the wiring doesn’t change.) Two details are worth knowing about what just happened, because both are easy to get wrong by hand. The server uses a hardened TLS config (1.2 minimum, AEAD cipher suites, X25519), so you’re not accidentally shipping a weak handshake. And the listener advertises HTTP/2 over ALPN, the h2 protocol gRPC rides on, which sounds like a footnote until you discover that recent gRPC clients refuse a TLS connection that doesn’t offer it. The framework sets it for you; it’s the single most common reason a hand-rolled gRPC-over-TLS server works with old tooling and mysteriously rejects a current client. All of that lives in the shared pkg/tls package.

I put the certificate under server.tls rather than server.grpc.tls deliberately. That shared block is the cert every transport falls back to, so the HTTP server in the next part and the transports after it can use the same one, with a per-transport override only where you actually need it.

The short version

A few files in, you have a real gRPC API: a typed contract, an implementation, health an orchestrator understands, reflection for poking, a clean shutdown, and TLS, and the only part that was actually yours to write was the service logic. The reference for the server helpers is the gRPC component doc, and the add-a-gRPC-service how-to has the manual-wiring path if you ever want it.

Next part puts an HTTP face on the very same controller, REST handlers and the same health endpoints an orchestrator probes, sharing that one certificate.

Building a web service with go-tool-base, part 1: lifecycle and graceful shutdown

Mon, 25 May 2026 00:00:00 +0000

A CLI does its job and gets out of the way. You run it, it prints something or writes a file, the process exits, done. Then one day you want the opposite: a thing that stays running. A server answering requests, a worker chewing through a queue, something that sits there until you tell it to stop. And the moment a process is long-lived, a pile of unglamorous problems lands on your desk that a short-lived command never had to think about.

How does it shut down when Kubernetes sends it a SIGTERM, without dropping the requests it’s halfway through? How does anything outside it know it’s alive, or ready for traffic? When one part falls over at 3am, does the whole thing come down, or pick itself back up? None of that is your actual service. It’s the plumbing around it, and it’s the sort of plumbing that’s easy to write almost right and only notice the gap in during an incident.

This is a new series, a companion to the one on building a CLI. That series gets you a working command-line tool; this one turns it into a web service, a piece at a time: gRPC, HTTP, a gateway that bridges the two, TLS across all of them, and live API docs. But every one of those is a long-running process, so we start with the part they all stand on.

Same shape as before, each part stands alone. By the end of this one you’ll have a process that starts cleanly, reports its own health, and shuts down without dropping anything, and you won’t have written the lifecycle code yourself. The series is written against go-tool-base v0.6.0, the release that brings the web-service components in.

What every long-running process needs

go-tool-base’s answer to all of the above is the controls package, and its centrepiece is the Controller. You hand it a set of services, things with a way to start, a way to stop, and a way to report health, and it owns their lifecycle. It starts them, watches for the operating system asking the process to quit, drives a graceful shutdown in the right order, and keeps a running picture of whether everything is alright.

A “service” here is deliberately loose. An HTTP server is one. A gRPC server is one. So is a background worker that wakes every few seconds, or a queue consumer. The controller doesn’t care what’s inside; it cares that it can start it, stop it, and ask after its health. That’s the whole trick: get those three verbs right once, in one place, and everything you bolt on later inherits them.

A service in thirty lines

Let’s build the smallest useful one: a heartbeat that logs a tick every second. It isn’t exciting, but it’s a real long-running service, and it shows every moving part without a transport getting in the way.

// main.go

package main

import (
	"context"
	"os"
	"sync/atomic"
	"time"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	"gitlab.com/phpboyscout/go-tool-base/pkg/logger"
)

func main() {
	log := logger.NewCharm(os.Stderr, logger.WithTimestamp(true))

	// The controller owns the process lifecycle: it starts registered services,
	// watches for SIGINT/SIGTERM, and drives a graceful shutdown.
	controller := controls.NewController(context.Background(), controls.WithLogger(log))

	var beats atomic.Int64

	controller.Register("heartbeat",
		// Start launches the work. The context is cancelled when the controller
		// shuts down, so the goroutine just watches ctx.Done().
		controls.WithStart(func(ctx context.Context) error {
			go func() {
				ticker := time.NewTicker(time.Second)
				defer ticker.Stop()

				for {
					select {
					case <-ctx.Done():
						return
					case <-ticker.C:
						log.Info("beat", "count", beats.Add(1))
					}
				}
			}()

			return nil
		}),
		// Stop runs during shutdown for any explicit cleanup.
		controls.WithStop(func(_ context.Context) {
			log.Info("heartbeat stopping", "total_beats", beats.Load())
		}),
		// Status reports health. Here we're healthy as long as we're ticking.
		controls.WithStatus(func() error { return nil }),
	)

	controller.Start()
	controller.Wait()
}

A few things are earning their keep there. NewController takes a context and some options, here just a logger. Register names a service and gives it its three verbs through functional options: WithStart launches it (and is handed a context that gets cancelled when the controller shuts down, which is the hook the goroutine watches), WithStop is called during shutdown for cleanup, and WithStatus answers “are you alright?”. Then controller.Start() launches everything and controller.Wait() blocks until the whole thing has stopped. The three options are all in pkg/controls.

Build it and run it:

go run .

INFO beat count=1
INFO beat count=2
INFO beat count=3

A service, running. Now for the half that’s easy to get wrong.

Shutting down on purpose

Press Ctrl-C, or send the process a SIGTERM the way an orchestrator would, and watch what it does:

WARN [Controller] : received signal signal=terminated
WARN [Controller] : Stopping Services
INFO heartbeat stopping total_beats=3
INFO [Controller] : Stopped

Nothing in our thirty lines handled a signal. The controller registered its own handlers for SIGINT and SIGTERM, and when one arrived it cancelled the context that every service’s Start is watching, gave them a window to finish, ran each Stop, and exited cleanly. That cancel-the-context-then-Stop order is the thing: it’s exactly what stops an HTTP server from dropping requests it’s mid-way through when the pod rolls. We’ve got it here for a heartbeat that does nothing on the way out, and we’ll get the same order for free for every real transport we add later.

You can tune the window with WithShutdownTimeout, and turn the signal handling off entirely (handy in tests) with WithoutSignals. The defaults are the right ones for a service in a container.

Health, before anything’s asking

That third verb, WithStatus, is the start of the health story, and it’s worth seeing now even though nothing’s calling it yet. The controller can report three separate things: overall status, liveness, and readiness. Those aren’t the same question. Liveness is “is this process wedged and in need of a restart”; readiness is “should traffic come here yet”. An orchestrator uses them differently, which is why they’re kept apart, and a service can answer them separately by adding WithLiveness and WithReadiness alongside WithStatus.

Right now nothing asks, because we’ve no transport. But this is the quiet payoff of putting lifecycle first: when we add an HTTP server in part 3, these reports are what back its /healthz, /livez and /readyz endpoints, and when we add gRPC in part 2 they back the standard gRPC health service, with no re-plumbing on our side. The controller also carries a restart policy for services that should pick themselves back up, and standalone health checks for things like “can I still reach the database”, but those earn their place once we’ve something worth checking.

Where this leaves us

A few lines in, we’ve a process that starts, ticks along, answers for its own health, and stops cleanly when the platform asks it to, on a controller that the real transports will register against unchanged. The heartbeat is a stand-in. Next part we swap it for a proper gRPC service, give it TLS, and the controller barely notices the difference, which is the entire point of it.

If you want to read ahead, the controls component has the full interface, and the service-orchestration deep-dive covers how the startup ordering and shutdown actually work underneath.

Building a CLI with go-tool-base, part 5: a CLI that updates itself

Sun, 24 May 2026 00:00:00 +0000

You ship version one. A week later someone finds a bug, you fix it, you cut version two. Now for the awkward part: how does the person who installed version one ever get version two? Email them? Hope they wander back to the install page? For a CLI that lives on people’s machines, “go and re-download it” is the answer that quietly strands half your users on old, broken builds. This part closes that gap, and like most of this series, the work is already done for you: your tool has shipped with an update command since part 1.

As before, this is written against go-tool-base v0.6.0 (gtb version).

The command is already there

update is one of the default features, so it’s been in your binary all along. Your users run:

mytool update

and the tool fetches the newest release, checks it, and replaces itself in place. No package manager, no re-download, no instructions. The rest of this part is about what that one command actually does, and how to make sure the binary it pulls down is the one you shipped.

Where it looks for releases

A tool can’t update itself without knowing where its releases live. That’s the --repo you passed back in part 1: it filled in your tool’s release source, the platform, owner and repository it checks. For --repo myorg/mytool that’s github.com/myorg/mytool, and mytool update looks at that project’s releases.

go-tool-base speaks more than one platform here, GitHub, GitLab, Gitea, Codeberg, Bitbucket, or a plain HTTP server, so the same command works whether you publish on github.com or your own GitLab. If you ever need to point somewhere else (a mirror, a private host), the custom release source how-to covers it; for a private repository it reads a token the same way the rest of the tool does.

What `update` does, step by step

When a user runs it, the command walks a short, careful path:

Resolve the latest release from your release source.
Compare versions. It reads the version baked into the running binary and compares it, as semver, against the latest. If you’re already current, it says so and stops: already running latest version, v1.2.0. (If your build somehow reports a version ahead of the latest published, it tells you off in character: your tardis travelled too far into the future...)
Download the right archive for the user’s OS and architecture.
Verify it before trusting it (the next section).
Replace the running binary with the new one, in place.
Bring the config along. If your tool has the init feature (it does by default), the update then runs the new binary’s init over the user’s config directory to fold in anything the release added.

That last step is easy to miss and matters more than it looks. A new version often ships new config: a key for a feature you just added, a changed default. Rather than leave the user a version behind, with code that expects settings their config file has never heard of, update re-runs init against their existing config once the swap is done, non-interactively (it passes --skip-login --skip-key, so nobody gets re-prompted for a token). It’s the same initialiser system from part 2, reused: the merge keeps what the user set and adds what the new version introduced, so the binary and its config move forward together. Turn the init feature off and this step is simply skipped, there’s no config to keep in step with.

There are two flags worth knowing. --version v1.3.0 targets a specific release instead of the latest, handy for pinning or rolling back. And --force updates even when the version check thinks you don’t need to. Most of the time, a bare mytool update is the whole story.

Downloaded isn’t the same as trusted

A binary that arrives over the network is a binary you didn’t build on the machine it’s running on, and a self-updater that swaps itself for whatever the server sent is a lovely way to ship a corrupted or tampered build straight into your users' hands. So before the swap, update verifies what it downloaded against a checksum manifest, the checksums.txt GoReleaser produces alongside your binaries. If the hash of the downloaded archive doesn’t match the one in the manifest, the update aborts and nothing gets replaced.

By default this is best-effort: a release that ships a checksums.txt is verified, but a release without one is updated with a warning rather than a hard stop. When you want the guarantee, make it mandatory:

# in your tool's config
update:
 require_checksum: true

Now a missing or mismatched checksum is a refusal, not a shrug. I wrote up why this matters, and exactly what it does and doesn’t buy you, in verifying your own downloads.

The limit is worth stating plainly, because it’s the whole reason there’s a “part two” to this story. A checksum proves the binary matches the manifest on the same release page. It catches a corrupted download or a botched upload cold. What it cannot catch is an attacker who owns the release platform and swaps both the binary and its checksum in the same breath, because then the two still agree. Closing that gap needs a signature whose trust root the release host can’t reach, which is a different piece of machinery (and a post of its own). go-tool-base now does exactly that: self-update signature verification has shipped, the binary checking a detached signature against a key it both embeds and fetches over WKD (how it works). Until you turn signing on for your own tool, checksums are the floor, and a worthwhile one.

Seeing it work without publishing anything

Here’s the catch with writing about self-update: you can’t update from a release you haven’t published, and your tutorial tool isn’t on anyone’s GitHub. There’s a flag for exactly this, meant for offline and air-gapped installs but perfect for a look under the hood: --from-file installs from a local release archive instead of the network.

Build a snapshot of your tool the way your release pipeline would (GoReleaser’s --snapshot builds the archives without publishing), then point update at one:

goreleaser release --snapshot --clean
mytool update --from-file ./dist/mytool_Linux_x86_64.tar.gz

You’ll watch the same extract-and-swap the network path uses, with nothing published and no release source involved. It’s also genuinely useful in its own right, for shipping into environments that can’t reach the internet.

The real loop

In production the cycle is the one part 1 already set you up for. The project gtb scaffolds ships a GoReleaser config and a release pipeline, so the flow is:

Tag a version and push the tag.
CI builds the binaries for every OS and architecture, generates checksums.txt, and publishes them as a release on your source.
Your users run mytool update and get it, verified.

You write git tag v1.3.0 && git push --tags; everyone who installed v1.2.0 is one command away from the fix. That’s the whole point of putting the update channel inside the tool: shipping a fix becomes tagging a release, and nothing else.

What this buys you

A tool that updates itself turns “please go and reinstall” into mytool update, and a tool that verifies what it updates to turns “I hope that download was clean” into a checked guarantee. Both came with the scaffold; the only work was understanding them. The full reference, including the config keys and the per-platform release sources, is in the update command docs and the auto-update concepts page.

Next part is the last one, and it’s about what happens after your tool is out there doing its job: telemetry and logging, so you can see how it’s actually being used without spying on the people using it. Until then, tag a release and watch your tool catch up to itself.

Building a CLI with go-tool-base, part 4: an AI dungeon master

Sat, 23 May 2026 00:00:00 +0000

I run a Dungeons & Dragons game on the odd weekend, so when I sat down to put an AI feature inside a CLI, my first instinct wasn’t a chatbot. It was: could the tool run a little adventure, with an AI as the dungeon master? It turns out that’s a near-perfect way to learn the chat client, because the thing that makes a game trustworthy, rules the players can’t break, is exactly the thing that makes any AI feature trustworthy. So this part builds mytool adventure: a tiny dungeon you play in your terminal, narrated by an AI that is firmly on a leash.

Part 3 pointed AI at your CLI from the outside (an agent driving your commands over MCP). This part goes the other way: AI inside your tool, as a feature you write. The worry everyone has about that is fair, AI output is unpredictable, and a CLI is meant to be dependable. The whole lesson here is how you square those two: you don’t hope the model behaves, you box it in with rules it can’t escape and mechanics it doesn’t get to invent.

As before, this is written against go-tool-base v0.6.0 (gtb version).

Behind the DM screen

A turn of our game looks like this: the player types what they want to do, the AI dungeon master narrates what happens and offers a few choices, and round it goes until the adventure reaches an end. The trick is where the truth lives. The model’s job is the prose, and only the prose. Everything else is yours:

The rules live in the system prompt: what the DM may and may not do.
The mechanics live in Go functions the model calls as tools (dice, combat). It never makes a number up.
The state lives in a Go struct you hand the model fresh every turn, so it never has to remember, and can’t quietly rewrite history.
The shape of each turn is a typed Go struct the model fills in, so your code always gets back something it can render, never a wall of prose to parse.

Two go-tool-base capabilities do the heavy lifting: the AI calling your Go functions, and the AI handing back a typed struct instead of text you have to regex. The game is just a fun excuse to use both at once.

Wiring a provider

The chat client (pkg/chat) is a library you import; you don’t need any special feature flag for it. It does need an API key, and it’ll find one from a few places. The simplest, for now, is the well-known environment variable for your provider:

export ANTHROPIC_API_KEY="sk-ant-..." # or GEMINI_API_KEY, OPENAI_API_KEY

That’s the bottom of the client’s lookup chain, which is fine for playing locally. For a tool you actually ship, go-tool-base has the ai feature and its mytool init wizard (the same initialiser system from part 2) to store the key properly, and there’s a whole post on where a CLI should keep your keys. For learning the client, an env var is plenty.

Scaffold the command

You know this step from part 1:

gtb generate command --name adventure --short "Play a dungeon adventure"

Everything below goes in the RunAdventure function the generator left you in pkg/cmd/adventure/main.go, plus a couple of types and helpers in the same package.

The state is yours, not the model’s

Start with the truth. The game state is a plain Go struct that you own. The model never holds it; instead you hand it the current state at the top of every turn (more on that in the loop). This is the part to grow: start small, then add rooms, items, NPCs, quest flags, whatever your adventure needs. Nothing else in the design has to change when you do.

// GameState is the single source of truth for the game. Extend it freely.
type GameState struct {
	PlayerHP int
	Location string
	Inventory []string
	Foes map[string]int // foe name -> remaining hit points
}

// summary renders the state into a line the model is given each turn.
func (g *GameState) summary() string {
	foes := make([]string, 0, len(g.Foes))
	for name, hp := range g.Foes {
		foes = append(foes, fmt.Sprintf("%s (%d HP)", name, hp))
	}
	return fmt.Sprintf("You have %d HP, at %s, carrying %s. Foes: %s.",
		g.PlayerHP, g.Location, strings.Join(g.Inventory, ", "), strings.Join(foes, ", "))
}

And the shape of a turn, the thing the model has to produce:

type Turn struct {
	Narration string `json:"narration"`
	Choices []string `json:"choices"`
	GameOver bool `json:"game_over"`
}

The dungeon master’s tools

A tool in pkg/chat is a chat.Tool: a name, a description the model reads to decide when to use it, a parameter schema, and a handler. The handler gets the model’s arguments as raw JSON and returns any value (which the framework JSON-encodes back to the model) or an error.

The simplest possible one is a die roll. This is the canonical “give the model something it’s bad at” tool, because language models cannot be trusted to roll fairly or even add up:

func rollTool() chat.Tool {
	return chat.Tool{
		Name: "roll",
		Description: "Roll a die with the given number of sides; returns 1..sides.",
		// Use an anonymous struct so the schema's properties sit at the top level,
		// which is where SetTools looks. A named type would hide them behind a $ref.
		Parameters: jsonschema.Reflect(struct {
			Sides int `json:"sides" jsonschema:"description=number of sides on the die"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			var a struct {
				Sides int `json:"sides"`
			}
			if err := json.Unmarshal(args, &a); err != nil {
				return nil, err
			}
			if a.Sides <= 0 {
				a.Sides = 20
			}
			return rand.Intn(a.Sides) + 1, nil
		},
	}
}

That comment about the anonymous struct matters, by the way. Reflect a named type and jsonschema emits a top-level reference with the real fields tucked inside, and the tool ships with no parameters at all. An anonymous struct inlines them where the framework expects. It’s the one sharp edge in the whole exercise.

Combat is where state actually changes, so combat is a tool too. Note it takes the foe by name and looks it up in Foes, so it works for the goblin and for any creature you add later, without touching this function:

func attackTool(game *GameState) chat.Tool {
	return chat.Tool{
		Name: "attack",
		Description: "Resolve the player's attack on a named foe. Rolls to hit, applies damage.",
		Parameters: jsonschema.Reflect(struct {
			Target string `json:"target" jsonschema:"description=the name of the foe being attacked"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			var a struct {
				Target string `json:"target"`
			}
			if err := json.Unmarshal(args, &a); err != nil {
				return nil, err
			}
			hp, ok := game.Foes[a.Target]
			if !ok {
				return map[string]any{"error": "no such foe: " + a.Target}, nil
			}
			if rand.Intn(20)+1 < 10 {
				return map[string]any{"hit": false, "foe": a.Target}, nil
			}
			dmg := rand.Intn(6) + 1
			hp -= dmg
			if hp < 0 {
				hp = 0
			}
			game.Foes[a.Target] = hp
			return map[string]any{
				"hit": true, "foe": a.Target, "damage": dmg,
				"foe_hp": hp, "defeated": hp == 0,
			}, nil
		},
	}
}

A bad target comes back as a plain error string, which the framework hands to the model so it can recover (apologise, pick a real foe) rather than crash.

That’s the whole tool set, and there’s deliberately nothing here for reading the state. The model never fetches it. Instead the loop hands it the current state at the top of every turn, which we wire up shortly. A language model has no memory you can rely on, so rather than trust it to remember the fight, you give it the truth each time.

The turn is a tool too

Here’s the neat part. The chat client won’t let a single call both run tools and return a typed struct, they’re separate modes. So instead of asking for the struct afterwards, we make submitting the turn into a tool of its own. The dungeon master ends its turn by calling submit_turn, and its handler captures the typed Turn into a variable we hold:

func submitTurnTool(out *Turn) chat.Tool {
	return chat.Tool{
		Name: "submit_turn",
		Description: "End your turn. Call this exactly once, last, with the turn's outcome.",
		Parameters: jsonschema.Reflect(struct {
			Narration string `json:"narration" jsonschema:"description=two-sentence narration of what just happened"`
			Choices []string `json:"choices" jsonschema:"description=the actions the player may take next"`
			GameOver bool `json:"game_over" jsonschema:"description=true only if the game has ended"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			if err := json.Unmarshal(args, out); err != nil {
				return nil, err
			}
			return "turn recorded", nil
		},
	}
}

So the turn’s structure is enforced by a schema, same as any other tool’s parameters. Your loop gets a populated Turn every round, never prose.

The rules

This is where you bound the model. The system prompt is the rulebook, and it leans hard on the tools so the DM has no room to freelance the mechanics:

const dmRules = `You are the dungeon master of a short fantasy adventure. Each turn
you are given the current game state and the player's action.

Resolve the action and end the turn:
- If the player attacks, you MUST call the attack tool with the foe's name to
 resolve it. Do not decide the hit or the damage yourself.
- For any other chance event, call the roll tool and use its result.
- For simple actions, just narrate them.
- Then call submit_turn exactly once: a two-sentence narration, two or three
 choices, and game_over.

Trust the state you are given; never contradict it. A foe at 0 hit points is dead
and stays dead. The game ends when the player's hit points reach 0 (they lose), or
when the player reaches a satisfying ending. When it ends, set game_over and narrate
the finish.

Keep the tone light and quick.`

Two of those lines carry the weight. Trusting the state you are given, and never contradicting it, is what keeps the world consistent: the state is handed in fresh every turn (the next section), so the model works from the truth instead of from a memory it does not reliably have. And you MUST call the attack tool is what stops it quietly deciding hits and damage itself when it would rather just narrate. Those two are the difference between a game with rules and a model telling a story.

The loop

Now stitch it together. Create the client with the rules as its system prompt, register the tools once, and run a turn each time the player acts:

func RunAdventure(ctx context.Context, props *props.Props, opts *AdventureOptions, args []string) error {
	game := &GameState{
		PlayerHP: 20,
		Location: "the mouth of a damp cave",
		Inventory: []string{"a short sword", "a guttering torch"},
		Foes: map[string]int{"goblin": 12},
	}
	var turn Turn

	client, err := chat.New(ctx, props, chat.Config{
		SystemPrompt: dmRules,
		MaxSteps: 8, // roll/attack, then submit_turn
	})
	if err != nil {
		return err
	}

	if err := client.SetTools([]chat.Tool{
		rollTool(),
		attackTool(game),
		submitTurnTool(&turn),
	}); err != nil {
		return err
	}

	action := "I step into the cave."
	for {
		turn = Turn{}
		// Hand the model the current truth, then the player's action.
		input := fmt.Sprintf("State: %s\nThe player: %s", game.summary(), action)
		if _, err := client.Chat(ctx, input); err != nil {
			return err
		}

		fmt.Println("\n" + turn.Narration)
		if turn.GameOver {
			return nil
		}
		action, err = chooseAction(turn.Choices)
		if err != nil {
			return err
		}
	}
}

The same client runs every turn, so the conversation and the tools carry through the whole game; and the State: line you prepend is always current, because the attack tool mutated game last turn. The model is never trusted to remember, only to narrate.

The one helper I glossed is chooseAction. A bare fmt.Scanln would do, but we can do much better with almost no effort, and make a point while we’re at it. The framework already leans on Charm’s huh for its init wizard, you met it in part 2, so we’ll use the same library for a proper menu, with one deliberate addition:

func chooseAction(choices []string) (string, error) {
	const other = "__other__"

	opts := make([]huh.Option[string], 0, len(choices)+1)
	for _, c := range choices {
		opts = append(opts, huh.NewOption(c, c))
	}
	opts = append(opts, huh.NewOption("Something else...", other))

	var pick, custom string
	form := huh.NewForm(
		huh.NewGroup(
			huh.NewSelect[string]().
				Title("What do you do?").
				Options(opts...).
				Value(&pick),
		),
		// A second step that only appears when the player chose "Something else".
		huh.NewGroup(
			huh.NewInput().
				Title("Describe your action").
				Value(&custom),
		).WithHideFunc(func() bool { return pick != other }),
	)
	if err := form.Run(); err != nil {
		return "", err
	}

	if pick == other {
		return custom, nil
	}
	return pick, nil
}

The select gives the player a tidy arrow-key menu instead of typing a number, but the addition that earns its keep is the last option. “Something else…” is always there, and choosing it unfolds a second step (huh shows or hides a group with WithHideFunc) where the player types whatever they actually want to do. That free text goes straight to the dungeon master as the next turn’s input, and because the DM is an AI bound by the rules rather than a switch statement over three fixed choices, it just copes. Bargain with the goblin, search your pockets, set the cave alight: the model narrates it within the rules you gave it, rolling and applying damage through the same tools. That is the agency a scripted game can’t offer, and it’s the natural place to start building your own richer interactivity on top.

Play it

Set your key, build, and go:

export ANTHROPIC_API_KEY="sk-ant-..."
just build
./bin/mytool adventure

A turn looks like this (your wording will differ every time; the mechanics won’t):

You swing your short sword at the goblin, the blade whistling through the damp cave
air. The creature snarls as it tries to dodge your blow.

What do you do?
> Attack the goblin again
 Try to push deeper into the cave
 Retreat to the entrance
 Something else...

Your blade whistles through the air, but the nimble goblin dances back just in
time. It lunges forward with a rusty dagger in return, yet its clumsy strike only
finds empty air.

What do you do?
> Swing your sword again!
 Try to intimidate the creature
 Retreat from the cave
 Something else...

Behind that, the dungeon master called attack each turn (a hit, then a miss), the goblin’s hit points changed in the GameState you own, and the next turn handed that updated state straight back to the model. The prose is the model’s; every number is yours.

The pattern under the game

Strip the dungeon away and you’re left with the thing worth keeping. An AI feature you can ship is one where you’ve kept the model away from everything that has to be right: the rules live in the system prompt, the mechanics in typed Go tools the model must call, the state in a struct you hand it fresh each turn rather than trust it to remember, and the output in a struct it fills in rather than free text. Do that and the model’s unpredictability is confined to exactly where you want it, the wording, and walled out of everywhere you don’t, the maths, the state, the shape of the result.

Two honest limits worth knowing. There’s no temperature dial on the client (the setting that would let you turn the model’s randomness down), so you can’t make the prose reproducible; you make the mechanics reproducible instead, which for most features is what you actually needed. And a tool calling loop is several round-trips to the model per turn, so it’s not free, keep MaxSteps tight for anything interactive.

That’s the foundation, and the state struct is already sized for more than one fight: it carries a location, an inventory and a map of foes you’ve barely touched. Add a move tool that updates Location, a use_item tool that reaches into Inventory, a second creature in Foes, even a give_quest flag, and the adventure grows without the architecture changing. The model just gets more tools to call and more truth to read. Saved games come nearly free, too: the client can snapshot and resume a conversation. Next part leaves AI behind and gets the tool ready to look after itself: shipping signed self-updates, so a new release reaches your users safely. Until then, go explore the cave.

Building a CLI with go-tool-base, part 3: expose your CLI to AI agents

Fri, 22 May 2026 00:00:00 +0000

“Make it work with AI” is the request that lands on your desk with a thud and no further detail. The first time it landed on mine I braced for a treadmill of integration work: an adapter for this assistant, a wrapper for that one, one per client, forever. Then I looked at the hello command we built back in part 1. It has a name, a one-line description, and (once you give it some) typed, documented flags. That is exactly the shape an AI agent needs to call a tool. You already did the hard part.

This part wires that up: turning the CLI you’ve been building into something an AI assistant can drive, with no AI code of your own. The how-it-works behind it is in your CLI is already an AI tool; here we just use it.

A version note, as in the earlier parts: this is written against go-tool-base v0.6.0 (gtb version). The tool is young and moving, so if you’re on a newer release and a command or its output has shifted, check there first.

The translator you already have

An AI agent that wants to call local tools needs three things: a list of named operations, a description of each so it knows when to reach for them, and a typed parameter schema for each so it knows how to call them. A good CLI is already all three. The only missing piece is a translator between “this is a CLI” and “this is a set of tools an AI can call”, and that translator is the Model Context Protocol (MCP), an open standard every serious assistant now speaks.

Your tool already ships it. mcp is one of the default features, so it’s been in your binary since you scaffolded in part 1, no flag required. Check:

./bin/mytool mcp --help

You’ll see subcommands you never wrote. The rest of this part is just three of them.

See what the agent sees

Before you connect anything, look at what your tool would expose. This writes the tool definitions to a file:

./bin/mytool mcp tools

{
 "tools": [
 {
 "name": "mytool_hello",
 "description": "Say hello",
 "inputSchema": {
 "type": "object",
 "properties": {}
 }
 }
 ]
}

That’s your hello command, seen from an agent’s side of the glass. The name is your tool’s name and the command path joined with an underscore; the description is the Short you gave it in part 1; the inputSchema is empty because hello has no flags yet. Add a flag and it shows up here as a property, with the type and help text you already wrote. There’s no second schema to keep in sync, because the command tree is the schema.

A few things are deliberately left off this list: hidden and deprecated commands, pure command groups that don’t do anything themselves, and the mcp, help and completion plumbing. So mcp tools doubles as an audit: it’s exactly what an agent can reach, and nothing else.

Run the server

One command turns the whole thing on:

./bin/mytool mcp start

It doesn’t print a banner and march off doing things. It sits quietly, speaking MCP as JSON-RPC over standard input and output, waiting for an assistant to talk to it. You won’t run this by hand much; the assistant launches it for you. But it’s worth knowing what happens when the agent calls one of your tools: the server re-runs your own binary with the arguments the agent supplied, captures the output, and hands it back. The agent isn’t poking at your internals. It’s running mytool hello, the same command a human would, and getting the same result.

Point an assistant at it

The quickest way is to let the tool write the client config for you. For Claude Desktop:

./bin/mytool mcp claude enable

There are cursor and vscode variants too. Restart the assistant and your CLI is in its toolbox.

If you’d rather wire it by hand (or your client isn’t one of those three), the config is small. Point the client at your binary with mcp start as its arguments, using the absolute path:

{
 "mcpServers": {
 "mytool": {
 "command": "/absolute/path/to/bin/mytool",
 "args": ["mcp", "start"]
 }
 }
}

Claude Desktop keeps that in claude_desktop_config.json (under ~/Library/Application Support/Claude/ on macOS, %APPDATA%\Claude\ on Windows); Cursor uses ~/.cursor/mcp.json; VS Code’s Copilot reads github.copilot.mcpServers in your settings. The shape is the same everywhere. From here, ask the assistant to say hello and watch it call mytool_hello.

The agent’s reach is exactly your CLI’s reach

This is the part worth being calm about. Exposing your CLI over MCP doesn’t widen its surface by an inch. The agent can call the commands you shipped, with the parameters you defined, and nothing else. It can’t invent a command or pass a flag you never wrote. The boundary of what it can do is the boundary you drew when you built the tool, and mcp tools shows you that boundary in full. If there’s a command you don’t want an agent touching, mark it hidden and it drops off the list.

For a long-running or remote setup there’s also ./bin/mytool mcp stream, which serves the same tools over HTTP instead of stdio; the mcp reference has the details. For most desktop assistants, start over stdio is all you need.

What it comes down to

You turned the CLI you’ve been building into an agent-callable tool with one command and zero lines of AI code, because the real work, naming your operations and documenting their inputs, you finished the moment your --help was any good. Every command you add from here is a new tool the agent gets for free.

Next part goes the other way: instead of letting an assistant drive your tool from outside, we put AI inside it, wiring up a provider and building a feature against go-tool-base’s chat SDK. Until then, add a command or two and watch them appear in mcp tools. The agent’s toolbox grows as your CLI does.

Technical CV writing is still hard, and now a robot reads it first

Fri, 22 May 2026 00:00:00 +0000

Seven years ago I wrote a post called Technical CV writing is hard, pulled my own CV apart, and explained every choice in it. I even bragged that it converted to a first interview about eighty per cent of the time, then added “watch me now jinx myself for the future”. Reader, I jinxed myself. I’m back on the market, the same CV that served me for two decades went out into the world, and what came back was a sort of stunned silence. Not even rejections. Just nothing.

The CV that suddenly stopped working

The thing about that silence is how specific it was. Some applications behaved exactly as they always had: a human read the CV, liked it or didn’t, and replied like a person. Others went into a void. And the void had a pattern to it. It was the bigger, more process-heavy outfits, the ones you’d bet good money have an Applicant Tracking System and an “AI-assisted screening” line item in some HR budget.

That’s when the penny dropped. My CV wasn’t failing to impress anyone. It wasn’t reaching anyone. The first thing reading it wasn’t a person at all.

The reader changed, and I hadn’t noticed

I’ve made this exact point on this blog before, only about software: half your users don’t have eyes. A CLI tool’s output has two audiences, the human at the terminal and the script parsing the output, and they want completely different things. It turns out a CV is now precisely the same. It has two readers, and the first one is a machine.

A human recruiter reads a CV the way I designed mine to be read: narrative, personality, a sense of the person. An ATS or an AI screen does nothing of the sort. It parses for structure, for keyword density, for recency, for numbers it can latch onto. My CV was a beautifully tailored sales pitch aimed squarely at a human who, increasingly, never gets to see it, because a parser in front of them scored it and quietly binned it first.

Everything that made it a good human document was, to the machine, either invisible or actively confusing.

So I asked an AI what the AI hated

There’s an irony here I’m choosing to enjoy rather than resent. The way I worked out what the filters object to was to sit down with Gemini, hand it my CV, and ask it to read the thing the way a recruitment AI would and tell me where it tripped. Using one AI to get past another. Fight fire with fire.

The one instruction I was firm about, and I’ll come back to it, was that the CV had to stay recognisably me. I wasn’t asking Gemini to launder my career into something generic and machine-shaped. I was asking it to help me keep as much of my own voice and judgement as possible, while making the thing easier for an AI to approve and a human to enjoy. There’s a practical edge to that, too: the screening tools are increasingly tuned to spot the patterns of generated text and weight them down, so a CV that reads as though a model wrote it can trip the very filter you were trying to please, quite apart from leaving the human at the end of it cold.

With that ground rule set, the hurdles it surfaced were genuinely illuminating, and a bit humbling given I’d written a whole confident blog post about how to do this.

The skills tables are worse than useless. My CV led with two lovely tables: Management Skills and Technical Skills, each with a level and years of experience. Clean and scannable for a human. To a lot of parsers, a table is a trap: they flatten it into a jumble and lose the structure entirely. Worse, listing “20+ years” against nearly everything triggers what I can only call the recency trap. Modern screening looks for skills that show up in your recent job descriptions, not in a header table. A language sitting in my skills table but not in my last two roles reads as stale or unverified, no matter how many years I claimed next to it. Gemini put it plainly: “if a tool sees Golang in a top table but doesn’t see it explicitly mentioned in your last two job descriptions, it assumes the skill is stale or unverified.”

“I have a passion for what I do” is noise. My opening profile statement, which I was rather proud of, is exactly the sort of thing a screening tool discards wholesale. As Gemini noted, these tools “completely ignore subjective self-assessments … because they cannot be measured or verified.” It wants a dense, factual summary full of the nouns it’s searching for, right at the top.

My numbers thin out the further back you go. My recent roles are full of the data these tools love: a 75% reduction in deployment times, three thousand-odd Kubernetes clusters, a GitLab instance with four hundred thousand repositories. My older roles, written years ago in a more narrative style, are all “oversaw the delivery of solutions” with not a metric in sight. The machine reads that as a career that got vaguer over time, which is the opposite of true.

Four pages is at least two too many. Parsers weight the first page or two most heavily. My education and the foundational stuff sat on pages three and four, where the algorithm barely bothers to look.

It couldn’t work out what I am. This was the sharp one. With “pre-sales”, “client management” and “Managing Director” sitting next to deep technical keywords, the classifier genuinely can’t decide whether I’m a commercial manager who used to code or a hands-on engineer who drifted into management. As Gemini described it: “the algorithm gets confused … It struggles to classify you: Are you a commercial manager who used to code, or a hands-on techie who got pushed into management?” So it does the safe thing and matches me to neither.

What I’m actually changing

Knowing the hurdles, here’s what the rebuild looks like. This is the part I want to be useful, so it’s concrete.

The tables are gone. In their place is a “Core Expertise” section, plain text the parser can read, grouped so my leadership sits next to my technical stack. And I’ve done the thing 2019-me was too much of a show-off to do: tiered it honestly. Instead of “Expert+” against everything, there’s a primary tier of what I actually do day to day, a proficient tier I can deploy without blinking, and a frank “familiar, not current” tier for the languages I last touched in anger a decade ago. That honesty isn’t just decency. A wall of “expert at everything” reads as noise to a machine and as bluster to a human, and I’d been doing both.

The subjective profile is replaced with a keyword-rich professional summary that says, in the first two lines, exactly what I am and at what scale.

The keywords that mattered have been woven down into the recent role bullets, so the parser sees them where it trusts them. And I’ve reframed the people-management and pre-sales language toward technical enablement and architectural advisory, because what I’m actually chasing is the technical-leader sweet spot: the person who owns the architecture and mentors the engineers, without the HR admin and the sales pitches. The CV now points at that, deliberately, so the classifier stops dithering.

There’s also a more personal beat in here. A previous employer handed me a role with a “VP” title, sold to me as exactly the technical-leadership job I’d been chasing. It wasn’t. The title turned out to be a pay-grade bracket rather than a description of the work, the work itself was hands-on firefighting with little of the leadership or empowerment I’d been promised, and I moved on within a few months. To a screening AI, that pairing is doubly awkward. A “VP” title files me as a meeting-heavy executive and rules me out of the hands-on Principal and Lead roles I actually want, and a sub-six-month stint trips the flight-risk flag that some trackers quietly score you down for. So the fix is to stop letting the inflated label do the talking: describe the functional reality of the work, retitle it to the technical track it actually was, and let the scale of what I wrestled with speak instead of the job title. Titles, it turns out, are for the pay band. The bullets are for the truth.

Keeping myself in it

Back to that ground rule. Every one of these changes is in service of getting past the machine to the human behind it, and neither reader is well served by a CV with the person scrubbed out of it. The screen, increasingly, is trained to notice generic generated phrasing and mark it down; the human, always, would rather read something with a pulse. So the keywords go in, the structure gets fixed, the metrics come forward, and the voice stays mine. No “results-driven synergistic leveraging of cross-functional paradigms” that nobody would ever say out loud. That was the whole point of doing it this way: let the AI help reshape the structure a parser cares about, while the words stay mine, so what comes out is easier for a machine to approve, easier for a human to enjoy, and still unmistakably written by me. Optimising for the filter and sounding like myself turned out not to be in conflict at all.

I genuinely don’t know if this works yet

Here’s the part that makes this a post and not a victory lap. I don’t know if any of this lands. The old CV converted at around eighty per cent, on my own possibly-generous reckoning, right up until it abruptly didn’t. The new one is going out now, into the same market and the same filters that were stonewalling me a fortnight ago.

So this is a promise as much as a post. I’m going to keep count, the way I should have all along, and come back with the actual numbers: did reshaping my CV for a reader with no eyes genuinely move the needle, or did I just make it uglier and learn nothing? Either way you’ll get the truth, because a follow-up that only reports good news isn’t worth writing. Watch this space, and if you’re sending CVs into the same silence, maybe try reading yours the way a machine would first. It’s a deeply odd exercise, and I suspect it’s now an essential one.

Building a CLI with go-tool-base, part 2: configuration your tool can trust

Thu, 21 May 2026 00:00:00 +0000

In part 1 you scaffolded a tool and gave it a hello command. It says the same thing every time, which is fine for a first command and useless for a real one. The moment a tool does anything worth doing it needs settings: an endpoint, a default, a token, a log level. And the moment you have settings, you have the problem nobody warns you about. You set one in a file, the tool ignores it, the code that reads it looks perfectly correct, and an hour later you find you’d typed tiemout. Nothing in the whole stack thought that worth a word.

The good news is you don’t have to build any of this. Your scaffold already wired up a config system in part 1, the same one the rest of go-tool-base uses. This part puts it to work: where a setting’s value actually comes from, how to ship sensible defaults alongside the command they belong to, how to layer files so a team and a laptop can disagree politely, and how to turn a fat-fingered key from a silent shrug into an error that tells you exactly what you got wrong.

The same version note as part 1, since each of these stands on its own: everything here is written against go-tool-base v0.6.0 (gtb version will tell you what you’re on). The tool is young and still changing shape, so if you’re on a newer release and a detail has drifted, that’s the first thing to check. I’ll flag anything that breaks across versions as it comes up.

You already have a config system

The root command loads configuration for you before any of your command code runs, merges every source together, and hands the result to each command through Props. By the time your RunHello runs, props.Config is populated and ready.

A value can arrive from several places at once, so there’s an order. Highest wins:

Command-line flags
Environment variables (your tool’s prefix plus the key, so hello.greeting reads MYTOOL_HELLO_GREETING, with the dots turned into underscores)
Config files (on disk, in the order they were loaded)

That ladder is the mental model for what beats what: a flag beats an env var, an env var beats a file. The files are worth pinning down, though, because there’s more than one and they don’t all come from the same place. This is the bit that’s easy to trip over:

Embedded defaults are baked into the binary, one slice per command. You don’t read these at runtime directly. The init command (coming up) bakes them into your config file for you.
The file init writes, ~/.mytool/config.yaml, is the default the tool reads, along with a machine-wide /etc/mytool/config.yaml if one exists.
Files passed with --config replace those defaults for that run rather than adding to them. Name one or more and the tool reads exactly those.

We’ll set each of these up in turn. The full reference lives in the config docs.

Reading a value is one call, and it’s typed:

greeting := props.Config.GetString("hello.greeting")
timeout := props.Config.GetDuration("server.timeout")
debug := props.Config.GetBool("verbose")

Give a command a setting

Let’s make hello configurable. Open pkg/cmd/hello/main.go (your file, the one the generator leaves alone) and read the greeting from config instead of hard-coding it:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	greeting := props.Config.GetString("hello.greeting")
	props.Logger.Info(greeting)
	return nil
}

Build and run it:

just build
./bin/mytool hello

ERRO failed to load config: no configuration files found
please run init, or provide a config file using the --config flag

Not what you expected, maybe, but it’s the right instinct from the tool. It has no configuration to read yet, and rather than guess, it stops and says so. Which brings us neatly to where settings actually come from.

Defaults belong to the command

You could drop a default into the project’s central config, and for something truly global like the log level that’s the right home. But a setting that belongs to hello should live with hello, not in a shared file you have to remember to edit every time you add a command. The generator does this for you, you just have to ask. Back in part 1 you generated hello without config support, so run the same command again with --assets:

gtb generate command --name hello --short "Say hello" --assets

This is safe to re-run. The generator honours the code you’ve already written: it refreshes the boilerplate cmd.go, adds the asset scaffolding, and leaves your main.go, and the RunHello you’ve been editing, completely alone. One thing to hold off on here: don’t reach for --force. Force rewrites everything, including that main.go, which is exactly the work you want to keep.

You now have pkg/cmd/hello/assets/init/config.yaml, and the generator has already opened it under the command’s own namespace:

hello:

Fill in your defaults under it:

hello:
 greeting: Hello
 style: plain

Those values are embedded into the binary as an asset, and the generated cmd.go registers them with Props for you (props.Assets.Register("hello", &assets)), so the config system knows where your command’s defaults live. A quick word on style, since we’ll lean on it shortly: it’s a second setting I’m giving a default now so it’s ready when we need it. Plain says the greeting as written; loud will shout it.

That per-command home comes with one rule worth taking seriously: namespace your keys. Notice the generator opened the file under a hello: key rather than at the top level. Copy that. Every command ships its defaults in its own embedded file, and those files are all merged together to build the config, but the order they merge in is not guaranteed. If two commands both defined a top-level timeout, which one won would be a toss-up that could flip between builds. Keep each command’s settings under its own name (hello.greeting, report.timeout) and the clash can’t happen in the first place. The generator namespacing the file for you is a hint worth taking.

One thing the defaults file does not do is set values through struct tags. If you later add a default:"info" tag to a config field, that’s documentation for the error messages, nothing more. Real defaults live here, in the embedded YAML. It’s an easy thing to assume otherwise and then wonder why your default never applied.

First run: init

So your defaults are baked into the binary. The tool still needs an actual config file to read, and that’s what init is for. It’s one of the features your tool shipped with, so it’s already there:

./bin/mytool init

INFO Configuration initialised in /home/you/.mytool/config.yaml

Open that file and you’ll find your command’s defaults waiting in it, merged with the framework’s own:

hello:
 greeting: Hello
 style: plain
log:
 level: info

That’s the missing piece. init gathers every command’s embedded defaults through the Assets layer, writes them to ~/.mytool/config.yaml, locks the file down to 0600 (it may hold secrets later), and drops in a .gitignore so nobody commits it by accident. Now hello has something to read:

Prefer no init step? init is a feature, and you can leave it out of your tool’s feature set. With it off, the tool loads its embedded defaults directly and runs with no config file at all, you’d only add one to override something. That suits a small, self-contained tool. This tutorial keeps init on, which is the default and the right call while a tool is finding its feet, so the rest of the article assumes it.

./bin/mytool hello

INFO Hello

Setup that needs a human: initialisers

Static defaults cover the values you can decide for the user. Some you can’t: a token, an API key, an endpoint that differs per person. Writing a blank or guessed value for those is worse than useless. This is where go-tool-base does something I’ve not seen many CLI frameworks bother with: it lets a command bring its own first-run setup, and wires it in for you. It’s one of the genuine reasons to build on the framework rather than roll your own, so it’s worth a proper look.

Generate a command with --with-initializer:

gtb generate command --name greet --short "Greet someone" --with-initializer

Alongside the usual files you get an init.go. It’s generated and marked DO NOT EDIT, and it does all the wiring. Here’s the heart of it:

// Code generated by gtb. DO NOT EDIT.
package greet

func init() {
	setup.Register(props.FeatureCmd("greet"),
		[]setup.InitialiserProvider{func(p *props.Props) setup.Initialiser {
			if skipGreet {
				return nil
			}
			return &GreetInitialiser{}
		}},
		[]setup.SubcommandProvider{func(p *props.Props) []*cobra.Command {
			return []*cobra.Command{NewCmdInitGreet(p)}
		}},
		[]setup.FeatureFlag{func(cmd *cobra.Command) {
			cmd.Flags().BoolVar(&skipGreet, "skip-greet", false, "skip initializing greet configuration")
		}},
	)
}

type GreetInitialiser struct{}

func (i *GreetInitialiser) Name() string { return "greet" }

func (i *GreetInitialiser) IsConfigured(cfg config.Containable) bool {
	return cfg.IsSet("greet")
}

func (i *GreetInitialiser) Configure(p *props.Props, cfg config.Containable) error {
	return InitGreet(p, cfg)
}

That package init() registers three things with the framework the moment your command is imported, with no central setup file for you to edit: the initialiser itself, an init greet subcommand so the user can reconfigure just this command later, and a --skip-greet flag on the main init. IsConfigured is how the framework avoids nagging: if the greet key is already in the config, init leaves it be and moves on.

All of that is generated for you. The one piece that’s yours is the InitGreet function in main.go, which starts as a stub:

func InitGreet(p *props.Props, cfg config.Containable) error {
	// TODO: Implement custom initialization logic for greet
	return nil
}

Fill it in with whatever the setup needs. go-tool-base leans on huh for prompts, the same library its own GitHub and AI setup use, so a one-question form looks like this:

func InitGreet(p *props.Props, cfg config.Containable) error {
	var greeting string

	form := huh.NewForm(
		huh.NewGroup(
			huh.NewInput().
				Title("What greeting should greet use?").
				Value(&greeting),
		),
	)
	if err := form.Run(); err != nil {
		return err
	}

	cfg.Set("greet.greeting", greeting)
	return nil
}

Set the value on cfg and you’re done. After the initialisers run, init writes the whole config out to disk, so the answer persists into ~/.mytool/config.yaml with everything else. Run mytool init on a fresh machine now and it stops to ask for the greeting; run it again and it sails past, because IsConfigured sees the key is already there. Need to redo just this one command’s setup? mytool init greet. The framework hands each command its own setup step, its own subcommand and its own skip flag, and asks you for a single function in return. That’s the trade worth making: static defaults in your embedded YAML, anything that needs a human in an initialiser.

Overriding: the environment and layered files

With a config file in place, the other sources come into their own. The quickest override is an environment variable. Remember the prefix you set when scaffolding in part 1: hello.greeting maps to MYTOOL_HELLO_GREETING, the prefix and key joined up, uppercased, dots turned to underscores:

MYTOOL_HELLO_GREETING="Hello from mytool" ./bin/mytool hello

INFO Hello from mytool

You didn’t register that variable anywhere; the config system binds it for you. The prefix is what keeps it from colliding with some other tool’s LOG_LEVEL on the same machine, which is exactly why it’s worth having.

Files are the other half, and they’re where that precedence list earns a closer look. A single config file is fine until two people, or two machines, want slightly different settings, and then you’re copying files around by hand. The --config flag fixes that: pass it more than once and the tool merges the files in order.

./bin/mytool hello \
 --config ./config.yaml \
 --config ./config.local.yaml

Between the files you name, the rule is later wins on a clash, and every key that doesn’t clash is kept. If config.yaml sets hello.greeting: Hello and config.local.yaml sets hello.greeting: Oi, you get Oi, but keys that appear in only one file survive untouched. It’s a merge between them, not a replacement.

The edge to remember is what --config does to the default locations: it replaces them. The moment you name a file, ~/.mytool/config.yaml drops out of the picture unless you name it too. So you pass the whole stack you want, a shared base and a local override together, and let precedence settle it. Commit a config.yaml with the team’s settings, keep an untracked config.local.yaml for your own, run with both, and your local tweaks win without anyone editing a shared file. Leave --config off and you’re back on the defaults init wrote: ~/.mytool/config.yaml plus that machine-wide /etc/mytool/config.yaml if it’s there. Whichever set of files you land on, environment variables and flags still sit on top.

The typo that does nothing

Now for the failure I keep circling. Say you want to change the greeting. Open your config, but fat-finger the key:

hello:
 greting: Oi  # meant to be greeting

Run it, and you get a blank line. The greeting you set never applied: the misspelled key was read, matched nothing, and was silently dropped, and the real greeting is now nowhere to be found. Nothing said a word. For a greeting it’s a shrug. For a timeout or a retry count it’s the bug you chase at 2am, and I wrote up the why of it in the config key that quietly did nothing.

go-tool-base won’t catch this for you by default, and that’s a choice rather than an oversight. There’s no central schema that knows every key your tool could ever take, because keys belong to the commands that use them. What you get instead is a way to opt a command in, so it validates its own slice and nobody else’s.

Making mistakes loud

Tell the generator you want validation for a command and it scaffolds exactly this (gtb generate command --name hello --with-config-validation). Since hello already exists, it’s a small file to add by hand. Create pkg/cmd/hello/config.go:

package hello

import "gitlab.com/phpboyscout/go-tool-base/pkg/config"

// HelloConfig describes the config keys the hello command consumes.
type HelloConfig struct {
	Greeting string `config:"hello.greeting" validate:"required"`
	Style string `config:"hello.style" enum:"plain,loud" default:"plain"`
}

// ValidateHelloConfig checks the hello config against its schema.
func ValidateHelloConfig(cfg config.Containable) error {
	return config.ValidateStruct[HelloConfig](cfg)
}

The tags carry the rules. validate:"required" means the key has to be present and non-empty. enum:"plain,loud" means style has to be one of those two words. config.ValidateStruct[HelloConfig] does the rest: it derives a schema from those tags and checks the config against it, returning a readable error if anything is off. It takes props.Config as it is, the Containable interface, so there’s no casting to a concrete type. Call it at the top of RunHello, before you trust any of the values, and use the style while you’re there:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	if err := ValidateHelloConfig(props.Config); err != nil {
		return err
	}

	greeting := props.Config.GetString("hello.greeting")
	if props.Config.GetString("hello.style") == "loud" {
		greeting = strings.ToUpper(greeting)
	}

	props.Logger.Info(greeting)
	return nil
}

(You’ll add strings to the imports at the top of main.go.)

Now make a real mistake. Set the style to something that isn’t allowed:

hello:
 greeting: Hello
 style: shout

ERRO config validation failed:
 hello.style: value "shout" is not allowed (hint: allowed values: plain, loud)

That’s the difference. The command stops and tells you the key, the bad value, and what it would have accepted. The same check catches a misspelled greeting: the moment the real key goes missing, required fails with hello.greeting: required field is missing instead of quietly running on nothing. Set style: loud and you get HELLO, because the value finally passes and the code downstream can trust it.

If you switch on the optional config feature (it isn’t in the default set, so you opt into it), you also get a ready-made mytool config validate command that runs these checks without you wiring anything into a command at all. Either way, the principle holds: the program already knows what good config looks like, so make it say so when the config is bad.

The upshot

Your hello command now reads a real setting, ships a sensible default that init writes into place, honours overrides from the environment and from layered files in a predictable order, and refuses to run on a value it doesn’t understand. That’s most of what configuration ever needs to be, and you wrote almost none of the machinery.

One thing I’ve skipped: config can also reload while the tool is running, so a long-lived process picks up a changed file without a restart. That’s its own capability with its own moving parts, and I pulled it apart in reloading config without a restart if you need it.

Next part, we give the tool something to do with all this config: we turn it into an AI tool, with a chat command and an MCP server. Until then, go add a couple of validated settings to your own commands. You’ve got the shape of it now.

Moving this blog off Jekyll

Thu, 21 May 2026 00:00:00 +0000

The blog you’re reading used to be a Jekyll site on GitHub Pages, built on the lovely beautiful-jekyll theme. It isn’t any more: it’s Hugo now, published to GitLab Pages. The hosting move rode along with go-tool-base leaving GitHub for GitLab, but dropping Jekyll for Hugo was its own decision, and the more interesting one. Most of the migration was painless. Two bits were not, and they’re the two bits worth writing down.

Why leave Jekyll

Jekyll hadn’t done anything wrong, exactly. beautiful-jekyll is a genuinely nice theme and the site worked fine for years. But it had started to show its age. It hadn’t seen much improvement in a long while, and keeping it building meant staying on older versions of Ruby. Ruby is a perfectly good language, just never one I’ve much enjoyed living in, and I could feel one of those fork-it-and-drag-it-up-to-date afternoons coming, the kind I’d done before and didn’t fancy repeating.

So rather than patch up what I had, I asked the more interesting question: what else is out there? It came down to a shortlist of two, Astro and Hugo. Hugo won, fairly narrowly. Partly I just liked more of its out-of-the-box themes. And partly because it’s written in Go: one portable binary, no toolchain to wrangle, the sort of thing an engineer can drop onto any machine and run without a second thought.

The day every image on the blog tripled

The first proper snag was about where images live. Hugo would happily have let me keep Jekyll’s arrangement, one big /assets/images/ folder with every post linking into it by absolute path. But I’d picked the Stack theme, and Stack leans towards page bundles: each post is a directory, and the post’s own images sit right next to its index.md, referenced by plain relative name. The cover image becomes a resource of the post rather than a file in a shared bucket.

That’s a better model, and I decided to commit to it. Getting there, I managed to make a proper mess. The migration copied the old assets/images across, and Hugo’s static/ directory wanted a copy too, and then I started moving covers into the bundles, and at one humbling point a count turned up every image existing three times: once in assets/images, once under static/, and once in a bundle. A blog with a hundred-odd images had become a blog with three hundred-odd, most of them duplicates nobody referenced.

The fix was to go all the way to the bundle model: move each post’s images into its own directory, rewrite the references from absolute /assets/images/x.png paths to bare x.png, and delete the two shared piles entirely. Once the images lived with the posts, there was exactly one copy of each and the path was obvious. But for an afternoon the repository was a hall of mirrors, and the lesson was to pick the new tool’s model and go all the way to it, rather than carrying the old one alongside it and ending up with both.

The “extended” image that wasn’t new enough

The second one cost me the most time, and it’s the most transferable, so it gets the most words.

Hugo comes in two flavours, ordinary and extended, and the Stack theme needs extended because it compiles SCSS. So I reached for an off-the-shelf extended Hugo container image, wired it into the pipeline, and watched the build fail with an error about a template function the theme was calling that simply didn’t seem to exist.

I spent far too long suspecting the theme, my config, my content. The actual culprit was a version. The image I’d grabbed was a couple of minor releases behind, and Stack v4 uses .Site.Language.Locale, a Hugo feature that only landed in 0.157. The image was older than that, so the function genuinely wasn’t there, and the error was telling me the literal truth in a way I wasn’t ready to hear. “Extended” had told me the flavour was right and lulled me into not checking the version, which was the thing that actually mattered.

The fix was to pin a specific, recent extended image rather than trusting a floating “extended” tag to be new enough. The pipeline now runs on a pinned hugomods/hugo:debian-git-0.161.1, comfortably past the 0.157 the theme needs, and the build that had been failing on a missing function went green the moment the version was right. A theme has a minimum Hugo version the same way any dependency has a minimum, and “extended” is a feature flag, not a version number.

What it comes down to

Moving this blog from Jekyll to Hugo, and from GitHub Pages to GitLab Pages on the way, was mostly a pleasant afternoon, with two frustrations worth sharing. Commit fully to your theme’s page-bundle model rather than dragging Jekyll’s shared-assets layout along beside it, or you’ll briefly own three copies of every image. And pin your Hugo version explicitly, because a theme needs a recent enough Hugo, and the “extended” label tells you nothing at all about whether yours is.

If the site renders for you now, both got sorted. If it doesn’t, well, you’re reading this in a text editor, and I’ve some more debugging to do.

Building a CLI with go-tool-base, part 1: scaffold and your first command

Wed, 20 May 2026 00:00:00 +0000

Every time I start a new Go CLI, the first hour goes the same way, and none of it is the actual tool. Config loading. A logger. An update command. An error path that prints something a human can act on. A help system. I built go-tool-base so I’d never write that hour again, and I’ve spent a good few posts explaining how the pieces work inside. This series is the other half: how you use it. By the end you’ll have a real CLI with all that wiring for free. This part scaffolds one and gives it its first command.

One note on shape before we start: each part stands on its own. Finish this one and you’ve got a working, buildable tool. Later parts add configuration, AI, self-update and telemetry, one at a time. Where you want to know how a piece works underneath, I’ll link the deep-dive as we go.

Install the gtb CLI

go-tool-base ships an automation CLI called gtb. Install it with the script from the installation docs:

curl -sSL \
 "https://gitlab.com/phpboyscout/go-tool-base/-/raw/main/install.sh" | bash

That fetches a pre-built release, embedded docs and all, and drops gtb in ~/.local/bin, so make sure that’s on your $PATH. Then check it’s there:

gtb version

One thing to get out of the way before we build: versions. As I write this, gtb version prints go-tool-base v0.6.0, and that’s what every command and snippet in this series is verified against. It’s a young tool that’s still moving quickly, and the install script always pulls the latest release, so if you’re reading this later and something doesn’t line up, a newer version is the likeliest reason. When a release changes something that matters to this series, I’ll cover it in a follow-up.

Scaffold a project

One command stands up a whole project, and gtb gives you two ways to drive it.

The direct way, with flags, is good for scripting and for repeating a setup exactly:

gtb generate project \
 --name mytool \
 --repo myorg/mytool \
 --description "My CLI tool" \
 --env-prefix MYTOOL \
 --path ./mytool

--env-prefix is worth setting now: it’s the prefix for the environment variables that can override your config later (so MYTOOL_LOG_LEVEL rather than a bare LOG_LEVEL that would clash with every other tool on the box). The wizard defaults it to your tool’s name in capitals; with flags it’s worth being explicit. We’ll lean on it in part 2.

(gtb generate cli is the same command, if you prefer that name.) Or leave the flags off and gtb walks you through an interactive prompt instead, which is the gentler way the first time:

gtb generate project

Either way, one of the choices is worth calling out now, because it explains something you’ll see in a minute: features. go-tool-base bundles a set of ready-made commands, self-update, embedded docs, a doctor health check, an MCP server, a changelog, OS-keychain storage, and you choose which ones your tool ships with at generation time, either through the --features flag or a checklist in the wizard. The default set is a sensible starting point, and you can add or drop features later. That is why, a moment from now, your brand-new tool already answers --help with commands you never wrote. The full flag list is in the generate reference.

What you just got

cd mytool and look around. It’s a complete, releasable project, not a hello-world:

mytool/
├── cmd/mytool/main.go # entry point
├── pkg/cmd/root/
│ ├── cmd.go # builds Props, wires the root command
│ └── assets/init/config.yaml # embedded default config
├── internal/version/version.go # version info, stamped at release
├── .gtb/manifest.yaml # the generator's record of your command tree
├── .github/workflows/ # lint, test, docs, release pipelines
├── justfile # build / test / lint / docs tasks
├── go.mod # with `go tool` deps pinned
└── ... # .golangci.yaml, .goreleaser.yaml, README, CHANGELOG

One file there is worth understanding before anything else: .gtb/manifest.yaml. It is the generator’s source of truth, a record of every command your tool has, how they nest, and a content hash of each generated file. You won’t edit it by hand, but gtb reads and rewrites it constantly. It is how the generator knows what your command tree looks like, and how it can tell whether you have changed a file it owns. Think of it as the map the generator builds from: it’s committed to git for you, and as long as it’s there, your tool’s structure stays reproducible. We’ll see it earn its keep when we regenerate.

The entry point, by contrast, is tiny, because the framework does the lifting. Here’s the generated cmd/mytool/main.go in full:

// Code generated by gtb. DO NOT EDIT.

package main

import (
	"mytool/internal/version"

	gtbRoot "gitlab.com/phpboyscout/go-tool-base/pkg/cmd/root"
	"mytool/pkg/cmd/root"
)

func main() {
	rootCmd, p := root.NewCmdRoot(version.Get())
	gtbRoot.Execute(rootCmd, p)
}

Two lines of body. root.NewCmdRoot (in your pkg/cmd/root/cmd.go) builds a Props, the container that carries the logger, config, filesystem and version to every command. gtbRoot.Execute runs it and routes any failure through one consistent error handler, so there’s no os.Exit scattered about. Note the DO NOT EDIT header: main.go and the root cmd.go belong to the generator. Your code goes elsewhere, which matters in a minute.

Build it and you already have a working CLI:

just build # or: go build -o bin/mytool ./cmd/mytool
./bin/mytool --help

You’ll see the built-in commands from the features you picked, update, docs, doctor and the rest, with not a line written by you.

There’s one step before those commands will actually run. Try one, say ./bin/mytool docs, and the tool stops with please run init: it has no configuration yet and won’t guess at one. So give it some:

./bin/mytool init

That writes ~/.mytool/config.yaml from the defaults your tool ships with, and now its commands run. (init is itself one of the features. You can switch it off for a tool that should run straight from its built-in defaults with no file at all, but leave it on for now.) Part 2 takes configuration apart properly; for now, init once and carry on.

Add your first command

Don’t hand-roll a command file. gtb generates the boilerplate and leaves you the logic:

gtb generate command --name hello --short "Say hello"

That creates two files (see the command reference):

pkg/cmd/hello/cmd.go (generated, DO NOT EDIT): the options struct, flag wiring, and the NewCmdHello(props *props.Props) constructor.
pkg/cmd/hello/main.go (yours): a RunHello function, where all your real business logic goes.

The split is the whole point. Open pkg/cmd/hello/main.go and write what the command does:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	props.Logger.Info("hello from mytool")
	return nil
}

Rebuild, and the command is wired into the tree:

just build
./bin/mytool hello

You never touched the root command to register it. gtb recorded hello in that .gtb/manifest.yaml and wired it in for you. (If you’d rather wire commands by hand against the library directly, the custom-commands how-to shows that path; the generated route is the one this series follows.)

Regenerate without losing your work

Here’s the bit people are right to be wary of. If the generator owns cmd.go and the root wiring, what happens when it runs again, after you’ve made changes? And it runs often: every gtb generate command rebuilds the wiring.

gtb regenerate project

Your edits survive, and not by luck. Three separate things protect them:

Your logic sits in a file the generator never rewrites. Command logic lives in main.go; only the boilerplate cmd.go is regenerated. The split isn’t cosmetic, it’s the contract.
It notices if you edited a generated file. That manifest stores a content hash of every generated file, so if you’ve changed one, regeneration stops and asks before overwriting rather than silently stamping over you.
You can fence files off entirely. A gitignore-style .gtb/ignore tells the generator to leave specific paths alone, even under --force.

I wrote up how that edit-preserving diff actually works if you want the mechanism; the regenerate reference has the flags. For now, the thing to trust: scaffolding here is not a one-way door. You keep regenerating as the tool grows, and your edits stay put.

Where this leaves you

A few minutes in, you have a real CLI: config, logging, a consistent error path, self-update, embedded docs and a release pipeline, none of it written by you, plus your own hello command and the confidence to regenerate without fear. That’s the head start go-tool-base exists to give.

Next part: configuration. Typed settings, defaults the tool ships with, and how to turn a misspelled config key from a silent shrug into an error that tells you what you got wrong rather than a mystery you debug at 2am. Until then, go add a few more commands. You’ve got the pattern now.

Two bugs that taught me the rules

Wed, 20 May 2026 00:00:00 +0000

Some bugs are interesting because they’re subtle. These two were interesting because they were the exact opposite… in each case the tool had a hard rule I simply didn’t know about, and its error message couldn’t be bothered to tell me what that rule was. Both came out of building the infrastructure toolchain, both cost me a good deal more time than they had any right to, and both are the sort of thing that looks blindingly obvious the moment you know it and utterly baffling until you do.

So here they are, written down, partly to save you the bother and partly so I don’t go and forget them myself.

Bug one: the rule-less job that skips your merge requests

The cicd gate components, in their first cut, shipped with no rules: block. They were dead simple jobs: lint, scan, validate. No conditions, because they should just always run. Obviously.

They ran on branch pipelines. On merge requests, they didn’t run at all! The gates that were the entire point of the components were simply absent from the one place you’d most want to see them… the merge request.

The cause is a GitLab CI rule that’s remarkably easy to go years without ever learning: a job with no rules: block runs only on branch and tag pipelines. It does not run on merge-request pipelines. So “no conditions” doesn’t mean “runs everywhere” at all. It means “runs everywhere except a merge request”, which is about the least intuitive default I can think of.

The fix is faintly absurd, and that’s exactly what makes it stick. You add an unconditional rule: rules: [{ when: on_success }]. The content of that rule does precisely nothing. It always matches. What actually matters is that the job now has a rules: block at all, because merely having one is what makes a job eligible for merge-request pipelines. A rule whose content is meaningless, added solely so the block exists. That’s the fix. I’ll admit I stared at it for a moment.

Bug two: the import block that only works at the root

The second one came from terraform-aws-security-baseline. The account-hardening module needed to adopt a resource that already existed in the account, which is exactly what OpenTofu’s import {} block is for. So an import block went into the account-hardening module, right next to the resource it was adopting. The natural home for it, surely.

OpenTofu disagreed, and rejected it outright. The rule: an import block is only allowed in the root module. It can’t live inside a child module. A module that wants one of its own resources imported can’t declare that import itself… the import has to be declared up at the root, and the root caller does the adopting.

The fix was to take the import block out of the module and document caller-side adoption instead. The module describes the resource, and the root configuration that calls the module is where the import actually lives.

Two unrelated bugs, in two completely different tools, and the same shape sitting underneath both of them.

In each case the tool has a hard structural rule. Where a block is allowed to live. What makes a job eligible for a particular kind of pipeline. And in each case the error told me the tool was unhappy without telling me which rule I’d broken, so the obvious next move (debugging my own logic) was the wrong move entirely. There was nothing wrong with the logic. The thing was simply in a place the tool doesn’t allow, or missing a block the tool quietly insists on.

The lasting lesson here isn’t the two specific rules, useful as they are to know. It’s the reflex. When something that should obviously work just doesn’t, and the error is unhelpful, stop debugging your logic and start suspecting a structural rule about where something is allowed to be, or whether a thing is eligible in the first place. GitLab CI and OpenTofu both have a handful of these, and you mostly learn them the hard way, by tripping over them. Knowing the shape of the category at least means the next one costs you an hour instead of a whole afternoon.

Worth remembering

Two bugs from building the toolchain, one shape. A GitLab CI job with no rules: block runs on branches and tags but silently not on merge requests, and the fix is an unconditional rules: block whose content does nothing and whose mere existence is the entire point. An OpenTofu import block gets rejected inside a child module, because imports are only legal at the root, so the caller adopts and the module just describes.

Neither error named the rule it was enforcing, and that’s the category to watch for. When sound logic fails against an unhelpful error, suspect a structural rule about where a thing may live or whether it’s even eligible… not a bug in what you actually wrote. It’ll save you an afternoon. It certainly cost me a couple.

Reviewed, then applied

Mon, 18 May 2026 00:00:00 +0000

The genuinely dangerous moment in infrastructure-as-code isn’t the apply. It’s the gap between the plan a human read and approved, and the change that actually runs a moment later. If those two are different computations (and by default they are) then nobody really reviewed the thing that touched your account. The infra repo closes that gap from both ends.

The gap between “reviewed” and “ran”

Here’s the moment in infrastructure-as-code where things go wrong.

Someone opens a merge request. CI runs tofu plan and the output is there to review: these three resources change, this one is destroyed. A human reads it, decides it’s correct, approves, merges. Then apply runs.

The trap is in what apply actually applies. If apply does its own fresh tofu plan and then applies that, the change that runs is not necessarily the change that was reviewed. State can have moved. A provider can have drifted. Someone else can have applied something in between. The reviewed plan and the applied change are two separate computations done at two different moments, and every difference between those moments is a change nobody looked at.

infra closes that gap from both ends.

Plan as an artifact

The first end is making the reviewed plan and the applied plan the same object.

The tofu-plan component runs the plan and saves it. It writes tfplan.cache, OpenTofu’s binary plan file, as a CI artifact. It also writes tfplan.json, which GitLab renders as a plan widget right in the merge request: the add, change and destroy summary, there to review without leaving the MR.

The tofu-apply component then does not re-plan. It applies that saved tfplan.cache. And OpenTofu itself enforces the safety net: applying a stale plan file, one captured against a state that has since moved, is rejected by the tool. So what reaches the account is provably the plan that was reviewed, or it’s nothing at all. There’s no third option where something unreviewed slips through.

Applying is a human decision

The second end is when apply runs.

infra is trunk-based: it dropped the develop branch and works on main. But a naive trunk setup auto-applies every push to main, which means there’s no human gate at all, just whatever the last merge happened to contain.

So the gate is built explicitly. releaser-pleaser keeps a release merge request open against main. Ordinary merges to main run plans but apply nothing. The apply happens only when a person merges the release MR. Merging it cuts a release tag, and the tag pipeline is what runs tofu-apply, against the plan banked by the latest main pipeline.

The effect is that the act of applying to the account is the deliberate, visible act of merging the release request. Nothing reaches the account because a commit landed. It reaches the account because a person decided a release should go out and merged it. (Which, after the accidental v2.0.0 that kicked off the whole GitLab move, is a discipline I’d freshly relearned the value of.)

The guard on the gate

There’s one more piece, because a gate is only as good as its precondition.

A verify-main-plan job blocks the release MR from being mergeable unless the latest main pipeline is green. You can’t cut a release, and therefore can’t apply, on top of a main whose plan didn’t even succeed. The human gate has its own gate: the thing you’re about to merge has to be standing on a known-good plan before you’re allowed to merge it.

The bottom line

The risk in infrastructure-as-code is the gap between the plan a human reviewed and the change that runs, because a re-plan at apply time is a different computation from the one that was approved.

infra closes it twice over. tofu-plan saves the plan as a tfplan.cache artifact and renders it as a merge-request widget; tofu-apply applies that exact artifact, and OpenTofu rejects it outright if the state has moved underneath it. And applying is gated on a human merging a releaser-pleaser release request, not on a push, with a verify-main-plan check making sure that request can only be merged on top of a green plan. What gets applied is what was reviewed, when a person decided it should be.

One graph, not micro-stacks

Sun, 17 May 2026 00:00:00 +0000

Once an infrastructure repo has a few concerns in it (account hardening, the security baseline, the signing stack still to come) there’s a steady pressure to split them into separate stacks with separate state, and Terragrunt is right there to help you do it. The infra repo keeps everything in one OpenTofu graph instead. The reason comes down to who enforces your dependency ordering: the engine, or you.

The pressure to split

The infra repo’s src/ has several concerns in it, and more coming, the signing stack among them. Once a repo reaches that point, there’s a steady pressure to split: one stack per concern, each with its own state file.

It’s an appealing pressure. Separate stacks feel modular. Each apply touches less, so the blast radius of any one run is smaller. And Terragrunt exists, popular and well-regarded, precisely to orchestrate a fleet of separate stacks. The path is well trodden.

infra didn’t take it. src/ is a single OpenTofu root stack: each concern is a module block, in its own main.<concern>.tf file, all sharing one state and one graph.

What one graph gives you

The thing a single graph gives you is engine-enforced truth about ordering and data.

Inside one OpenTofu graph, the tool builds the full dependency DAG itself. When the signing stack needs a value the security baseline produced, you reference it directly, module.baseline.something, and OpenTofu guarantees two things: the baseline is created before the thing that depends on it, and the value handed across is the current one from this same apply. Ordering and data-passing aren’t things you arranged. They’re facts the engine checks and enforces, every plan, every apply.

What splitting costs

Split src/ into per-concern stacks with separate state, and that guarantee is the thing you spend.

Now one stack reads another’s outputs through terraform_remote_state. That’s a lookup of a snapshot: the other stack’s last applied state, whatever it was, whenever that was. It’s not a live edge in a graph. Ordering is no longer enforced by the engine either; it becomes something you arrange yourself, in CI stage sequencing or in Terragrunt’s own dependency blocks.

That’s the trade, stated plainly. You give up a strong, engine-checked guarantee, and you buy back a weaker, hand-arranged imitation of it. Terragrunt is a good tool for managing that weaker world tidily. But the question worth asking first is whether you should be in the weaker world at all.

When splitting is genuinely right

This isn’t an argument that splitting is always wrong. Separate states genuinely earn their place when concerns have different change cadences, different access boundaries, or different teams owning them: when you actively want an apply of one to be unable to touch another, and you want different people holding different state.

infra has none of those. It’s a single account, a single operator, one cohesive set of concerns. The only thing splitting would buy here is a smaller per-apply blast radius, and that’s better handled by reviewing the plan before it applies, which the next post is about, than by fragmenting the dependency graph. So src/ stays one graph, and Terragrunt was considered and deliberately not adopted.

If ordering between graphs is ever needed

If infra ever does genuinely need more than one stack, the plan isn’t Terragrunt. It’s to keep each stack a single strong graph internally, and to sequence the stacks with CI stages. Keep the engine-enforced guarantee where it’s strongest, inside each graph, and reach for hand-arranged ordering only at the one seam where it’s unavoidable.

Boiling it down

A multi-concern infrastructure repo feels like it should be split into per-concern stacks, and Terragrunt is right there to manage the result. infra keeps src/ as one OpenTofu graph instead.

Inside one graph, OpenTofu enforces dependency ordering and passes current values across module boundaries as checked facts. Split into separate states and that becomes a terraform_remote_state snapshot lookup plus ordering you arrange by hand: a weaker version of what you gave up. Splitting is right when concerns have different cadences, boundaries or owners; for a single-account, single-operator repo none of that applies, so the strong guarantee is worth keeping, and Terragrunt is the tool for a problem infra chose not to have.

CI you include, not copy

Sat, 16 May 2026 00:00:00 +0000

Every infrastructure repo runs the same CI: lint the OpenTofu, scan it, validate it, plan, apply. The first repo, you write that .gitlab-ci.yml by hand. The second, you copy it. By the third, you’ve got three copies of the same pipeline quietly drifting apart, which is the exact problem you’d never tolerate in application code. The cicd repo is the fix, and it’s just the library-first instinct pointed at the pipeline.

The `.gitlab-ci.yml` you keep copying

The infrastructure repos in this series all run the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security issues and secrets, and on the deploy side, plan and apply.

The first repo, you write that .gitlab-ci.yml by hand. The second repo needs the same jobs, so you copy it. The third repo, you copy it again. Now there are three copies of the same pipeline, and they do what copies always do. They drift. A fix you make in one repo’s CI doesn’t reach the other two. A tightened scan rule lands in the repo you were working in and nowhere else. It’s the copy-paste problem, exactly as it shows up in application code, just written in YAML and therefore that bit easier to pretend isn’t code.

GitLab has a feature for exactly this

GitLab CI/CD Components are the answer to that problem. A component is a reusable, versioned piece of pipeline that you publish, and other projects pull in with an include: pinned to a version:

include:
 - component: gitlab.com/phpboyscout/cicd/tofu-lint@v0.5.0

That’s a library import, for pipeline. The component has a defined interface, a version, and a home in GitLab’s CI/CD Catalog. A consuming repo includes it instead of carrying its own copy, and when the component improves, the consumer moves a version pin rather than re-copying YAML.

Why a monorepo of components

The cicd repo holds all of the components together: tofu-lint, tofu-security, tofu-validate, tofu-plan, tofu-apply, and more. One project, not one project per component.

That’s a deliberate call, and the reason is how GitLab versions things. A version is a tag, and a tag belongs to a project. A component’s version is its project’s tag. So a monorepo of components, versioned together as one tag stream, is the natural unit: a consumer pins @v0.5.0 and gets a known-good set of components that were tested together, rather than juggling a separate version for each one.

Authoring discipline

A component is a file under templates/, and it opens with a spec: inputs: block: the typed inputs, their defaults, the component’s public interface.

The discipline that keeps the library usable is that a component must be consumer-agnostic. It never hardcodes a token, and it never names a particular consumer’s variable. Inputs have sensible defaults, and a consuming repo overrides them. A component that reaches out and assumes something about the repo including it is a component that works in one repo and surprises the next. An authoring guide in the repo keeps that consistent across everyone who adds a component.

The self-test you cannot fully write

The cicd repo tests its own components with a self-test pipeline. It’s worth knowing where that self-test stops.

When a repo tests its own components by running them in child pipelines, GitLab masks $CI_PIPELINE_SOURCE as parent_pipeline. A component’s rules:, which often branch on the pipeline source to behave differently for a merge request than for a branch or a tag, therefore can’t be exercised honestly by the self-test: the source they’d branch on has been flattened. The self-test covers what it can, and the component rules: are, in the end, validated by real consumers using them for real. That’s a genuine limit, and naming it is better than pretending the self-test proves more than it does. (It’s also, not coincidentally, the exact rules: quirk that bit me in one of the two bugs I closed the series with.)

The same instinct, again

This blog keeps circling the same instinct. go-tool-base exists because the same CLI scaffolding kept getting rewritten, so it was extracted into a library. cicd is that instinct pointed at the pipeline: the same gate jobs kept getting copied between repos, so they were extracted into a versioned, included library.

Stop copy-pasting. Publish, version, include. It’s true for CLI code, and it turns out to be just as true for the YAML that builds and ships it.

The gist

Every infrastructure repo needs the same CI, and copying the .gitlab-ci.yml between them produces copies that drift apart. GitLab CI/CD Components fix it: reusable, versioned pipeline that a repo include:s and pins, instead of carrying its own copy.

cicd is a monorepo of those components, versioned together as one tag stream, because GitLab tags a project and a component’s version is its project’s tag. Components are authored consumer-agnostic, with typed spec: inputs: and no hardcoded assumptions, and their rules: are validated by real use because the self-test can’t see the pipeline source. It’s the library-first instinct, applied to CI: publish it once, include it everywhere, fix it in one place.

One image for the whole toolchain

Fri, 15 May 2026 00:00:00 +0000

Every CI gate job across the infrastructure repos reaches for the same pile of tools: OpenTofu, tflint, trivy, checkov, gitleaks, terraform-docs, the AWS CLI. Installing that pile per job is both slow and quietly dangerous, because nothing pins it consistently. infra-tools is the obvious fix (one image, one source of truth for versions), but two of its build decisions are less obvious and worth a look: it publishes with crane instead of a second build, and it deliberately lets its own vulnerability scan fail.

The same pile of tools, in every repo

Every infrastructure repo in this series runs the same CI gate jobs: format and validate the OpenTofu, lint it, scan it for security problems and secrets, check the docs. Those jobs need a specific set of tools, and it’s the same set in every repo.

Install them per job and you pay twice. You pay in time, because every pipeline downloads and installs the whole set again. And you pay in drift, because unless every repo pins every tool identically, the repos slowly diverge on which version of trivy or tflint they actually run, and a check that passes in one repo fails in another for no reason anyone can see.

One image, one source of truth

infra-tools is the answer: a single Debian-based container image with the whole toolchain baked in. Every CI job in every repo uses it with one image: line.

The real value isn’t the convenience. It’s that the image is the one place tool versions are pinned. The Go-based tools are pinned in a mise.toml. checkov, which has no mise plugin, is pinned in a requirements file installed with pipx. The AWS CLI is pinned by a build argument. Three mechanisms, because the tools come from three kinds of source, but one image, and every pin wired to Renovate so a version bump arrives as a reviewable pull request. There’s exactly one answer to “what version of trivy does the toolchain use”, and it lives here.

Publishing with crane, not a second build

A build-pipeline detail that took a real bug to discover.

The pipeline builds the image with kaniko, which builds images without a privileged Docker daemon, something that matters a great deal on shared CI runners. Then it scans the image, then it publishes it.

The obvious way to write the publish stage is “build the image and push it”. But kaniko has no mode for “just push this tarball I already built”. A second kaniko invocation re-executes the entire Dockerfile from the top, including a second mise install, which makes a fresh round of calls to GitHub’s API to fetch tools. GitHub’s anonymous API limit is low and shared by IP, so on a CI runner that second install reliably trips a 403 rate-limit. (Yes, another 403. They do get everywhere.)

So the publish stage doesn’t rebuild. It uses crane to push the exact image tarball the build stage already produced. The image is built once. And because the published bytes are the same bytes the scan stage scanned, there’s no gap between “the image we checked” and “the image we shipped”.

Soft-failing the scanner on purpose

The decision that looks wrong until you see the reasoning: the pipeline scans the image with trivy, and trivy is allowed to fail without failing the pipeline.

A vulnerability scanner that doesn’t gate the build sounds like a scanner switched off. It isn’t. It’s a scanner pointed at something it can’t helpfully gate.

The tools in the image are prebuilt Go binaries. trivy inspects them, reads the version of the Go runtime each was compiled with, and reports every known CVE in that Go runtime. Those findings are real, but they aren’t mine to fix. The only fix is the upstream tool rebuilding itself against a patched Go. With seven such tools in the image, at any given moment one of them is usually a little behind on its Go version.

A hard gate would mean the image becomes unpublishable whenever any single upstream lags, over a CVE in code I don’t own and can’t patch. That’s not a security control; it’s a way to be unable to ship. So the scan is allow_failure. The findings stay fully visible, and the residual count is genuinely useful as a metric for how far behind upstream the toolchain has drifted. It just doesn’t block shipping an image whose only “vulnerabilities” are other people’s build timelines.

What it comes down to

The infrastructure repos all run the same CI gate jobs, needing the same tools, so infra-tools bakes the whole toolchain into one image and pins every version in one place, wired to Renovate.

Two build choices are worth copying. The publish stage uses crane to push the already-built, already-scanned tarball, because a second kaniko build would re-run mise install and hit GitHub’s anonymous rate limit, and because pushing the scanned bytes means shipping exactly what was checked. And the trivy scan is deliberately allow_failure, because it reports Go-runtime CVEs in prebuilt upstream binaries that no change to this repo can fix, so a hard gate would only make the image unshippable over someone else’s lag.

A 403 you can't fix in IAM

Thu, 14 May 2026 00:00:00 +0000

The OIDC post explained the handshake that lets a GitLab pipeline deploy to AWS with no stored key. This is the story of the first time I got it wrong, and spent an afternoon fixing the wrong thing. The error was a flat 403 from AWS, and the maddening part is that no amount of editing the IAM policy was ever going to fix it.

A 403 on the first real run

The OIDC post covered the handshake: GitLab CI mints a signed token, AWS exchanges it for short-lived credentials against a role whose trust policy names the pipeline. During the GitLab migration I wired exactly that up for the infra repo, including a trust policy condition meant to let merge-request pipelines run a plan.

The first merge request that should have triggered tofu-plan didn’t run it. The job failed, and the error from AWS was a flat AccessDenied. A 403.

The instinct, and why it wastes an afternoon

The instinct on an IAM 403 is immediate and almost always right: the policy’s wrong, so go and edit the policy. Tighten the condition. Loosen the condition. Check the wildcard. Re-read the sub pattern character by character.

All of that was wasted, and it was wasted for a reason that took me far too long to see. The trust policy wasn’t matching the wrong value. It was matching a value that does not exist. No amount of editing a condition makes it match a thing that’s never present.

What is actually in the token

GitLab’s OIDC token has a sub claim that encodes the pipeline’s context, and part of that encoding is a ref_type. I’d assumed ref_type could be branch, tag, or mr, because a pipeline can certainly be a branch pipeline, a tag pipeline, or a merge-request pipeline. So the trust policy, for the plan job, matched a sub containing ref_type:mr.

That assumption was wrong. GitLab’s ref_type is branch or tag. That’s the entire set. There is no mr.

A merge-request pipeline doesn’t run against a merge-request ref. It runs against the source branch. So its token’s sub carries ref_type:branch, like any other branch pipeline. The trust policy condition asked for ref_type:mr, GitLab never puts mr in a token, the condition was therefore never true, and every merge-request pipeline got a 403. Forever, until the policy stopped asking for a claim that isn’t real.

The fix, and the lesson worth more than the fix

The fix is small once it’s visible: match ref_type:branch and narrow it down by branch name or project path instead. An afternoon of policy edits, and the actual change is one word.

The lesson is the part worth keeping. When an OIDC trust fails, the useful question is never “is my policy clever enough”. It’s “what’s actually in the token”. An OIDC trust policy can only ever match the claims the identity provider genuinely asserts, and the gap between what a provider asserts and what you assumed it asserts is precisely where this class of bug lives.

So the move, when an OIDC handshake 403s, is to get hold of a real token and decode it. Look at the actual sub, the actual claims, the actual values. Match what’s there. A 403 that survives every sensible edit to the policy is usually not a policy that’s too loose or too strict. It’s a policy matching a claim that was never going to be in the token.

The habit it left behind

I wired an OIDC trust policy to let merge-request pipelines plan, by matching a sub claim with ref_type:mr. The first real merge request got a 403, and no edit to the policy fixed it, because GitLab’s ref_type is only ever branch or tag. A merge-request pipeline runs on a branch ref, so the mr value the policy demanded was never in any token.

The fix was one word. The habit it left behind is the valuable bit: when an OIDC trust fails, stop editing the policy and go and read a real token. A trust policy can only match what the provider actually asserts, and “what I assumed it asserts” is where the 403 was hiding the whole time. (If this shape of bug feels familiar by the end of the series, that’s not an accident: I come back to it with two more from exactly the same family.)

Pure-Rust Git, no git binary

Wed, 13 May 2026 00:00:00 +0000

go-tool-base’s VCS support has two halves that get confused for one. One half talks to forge APIs (GitHub, GitLab) for releases and pull requests. The other talks to the .git directory on disk: clone, history, diff, status. This post is mostly about the second half, and specifically about a question that turns out to have three answers in Rust, only one of which I’d recommend: how do you actually do Git from inside a program?

A VCS subsystem with two halves

go-tool-base has a VCS subsystem, and it does two distinct jobs.

The first is forge APIs. GitHub and GitLab, Enterprise and nested group paths included. It authenticates, lists releases, fetches release assets, manages pull requests. The self-update machinery sits on this half, and it’s what a tool uses to ask “what’s the latest release?”

The second is local Git. go-tool-base also carries a RepoLike object, an abstraction over an actual Git repository on disk: clone it, read its commit history, diff two trees, check its status. This half doesn’t talk to a hosting service at all. It talks to the .git directory.

It would be easy to assume the second half grew out of the first. It didn’t, and where it actually came from is the part worth telling.

A capability ahead of its consumer

The RepoLike object wasn’t built for go-tool-base. It came from another project, where it had already proved itself, and it was pulled into go-tool-base on purpose, with a specific future consumer in mind: the code generator.

The plan is for the generator to use Git directly. When it scaffolds a new tool, that tool should start life as a Git repository, with a git init and an initial commit. When you later regenerate, the generator should diff the regenerated template output against your working tree to detect drift, the same idea as respecting your edits. Both of those are local Git operations, not API calls, so the generator needs a repository abstraction to call into.

That wiring isn’t finished yet. The generator doesn’t drive RepoLike today. But the capability is in place, deliberately, ahead of the consumer that will use it, because the alternative is bolting Git support on later under deadline pressure, and that’s how you end up with the wrong abstraction.

So when rust-tool-base was built, a repository abstraction was never in question. The Rust port carries the same capability for the same reason: a Repo type with init, open, clone, walk, diff, blame, status, commit, fetch and checkout, present and ready for the generator to wire into. The open question was never whether to have it. It was how to do Git from inside a Rust program, and there are three answers, only one of which is any good.

Three ways to do Git, and the one worth picking

Shell out to git. Run the git binary as a subprocess and parse its output. It works until it doesn’t. The binary might not be installed. It might be a different version with different output. Its output is formatted for humans and changes between releases, so parsing it is a standing liability. You’ve made an undeclared dependency on a program you don’t ship.

Link libgit2. libgit2 is the C library that reimplements Git as something you can call from code, and git2 is the Rust binding to it. It’s solid and widely used. But it’s a C dependency, which means a C toolchain in the build, and it’s consistently the single biggest source of cross-compilation pain in the Rust Git ecosystem. The musl builds, the Windows builds, the static linking: libgit2 is where they tend to break.

Use gix. gix is a reimplementation of Git in pure Rust. No C library, no subprocess. It’s just Rust code, and it compiles and cross-compiles like any other crate, because that’s all it is. It’s also generally faster, and being pure Rust it fits the no-unsafe-in-first-party-code story far more comfortably than dragging a C library along.

rtb-vcs is gix-first. The Repo type is built on it. There’s no git binary dependency, and there’s no libgit2 in a default build.

gix is still maturing, and a few write paths, push in particular, aren’t ready in it yet. For those, git2 (the libgit2 binding) is held in reserve as a documented fallback, to be wired behind an opt-in Cargo feature if and when a write path actually needs it. Until then a default build carries no libgit2 at all, and the common case, a tool that clones, reads history, diffs and commits, never pays its cross-compile cost. (The gix backend itself sits behind an opt-in git feature, which is exactly the feature-flag story from a couple of weeks back, doing real work.)

`Repo` is a foundation, not a façade

One design decision is worth calling out, because it came straight from a go-tool-base lesson.

It would have been easy to build Repo as a narrow façade exposing exactly what the scaffolder and the release-notes feature need today, and nothing else. That was rejected on purpose. go-tool-base’s RepoLike is itself the cautionary tale: it arrived from another project, settled into a sensible abstraction, and is already lined up to carry a consumer, the generator, that wasn’t driving its design when it was first written. A repository abstraction gets used by code that doesn’t exist yet. Build one as a narrow façade around today’s needs and you’ve guaranteed a rewrite the first time a downstream tool wants something slightly different.

So rtb-vcs’s Repo is built as a foundation: a sensible, reasonably complete vocabulary of Git operations that a tool author can compose richer behaviour on, without re-importing gix directly and re-deriving the framework’s auth and concurrency conventions. The errors back this up. gix’s error types aren’t leaked through the public API; they’re wrapped in semantic RepoError variants, so the backend could be swapped, gix to git2, or to something else entirely, without breaking a single downstream caller.

Stepping back

go-tool-base’s VCS support has two halves: forge-API calls for releases and pull requests, and a RepoLike object for local Git operations. The repo half arrived from another project and is wired in ahead of its intended consumer, the code generator, which will use it to initialise repositories for scaffolded tools and to diff regenerated output for drift.

rust-tool-base carries the same capability on purpose. Its Repo type is built on gix, a pure-Rust Git implementation, so there’s no dependency on an installed git binary and no libgit2 C library in a default build, which keeps cross-compilation clean. git2 stays an opt-in fallback for the few write paths gix can’t do yet. And Repo is built as a foundation for downstream tools, with the backend wrapped behind its own error type so it can be replaced without breaking callers.

Routing security findings without the noise

Tue, 12 May 2026 00:00:00 +0000

Turning on GuardDuty and Security Hub gives you threat detection. It also gives you a firehose. And an alert system that dutifully forwards everything in that firehose isn’t monitoring, it’s a very efficient way of training your team to ignore alerts. So the alerts module’s real job isn’t detection at all. It’s deciding what’s actually worth interrupting a human for, and the interesting part is everything it deliberately throws away.

Detection is the easy half

Switching on threat detection in an AWS account is a few resources. GuardDuty, Security Hub with its standards, IAM Access Analyzer: the security baseline does exactly that. From then on, the account is generating findings.

And it generates a lot of them. Plenty are low-severity, informational, or simply the normal texture of a cloud account. If you wire every finding to an email or a pager, you haven’t built monitoring. You’ve built noise. And noise has a specific failure mode: people stop reading it, and the one finding that genuinely mattered scrolls past unread alongside two hundred that didn’t.

So the valuable work isn’t detection. It’s routing: deciding what’s worth interrupting a human for, and letting the rest sit quietly in a console for whenever someone reviews it.

Forward the severe, leave the rest

The alerts module routes findings with EventBridge rules into an SNS topic that emails out. The rules are deliberately picky. GuardDuty findings are forwarded only at severity 7 and above. Security Hub findings are forwarded only at HIGH and CRITICAL.

Everything below those thresholds isn’t discarded. It’s still in GuardDuty and Security Hub, where someone doing a review will see it. It just doesn’t get to interrupt anyone’s day. The threshold is the line between “look at this now” and “look at this sometime”.

The duplicate you would otherwise send twice

Here’s the subtle one, and it’s the kind of thing you only find by looking closely at where findings come from.

Security Hub is an aggregator. It pulls findings in from other services, GuardDuty among them. So a single GuardDuty finding can show up in two places: in GuardDuty itself, and again in Security Hub as an aggregated copy.

A rule on GuardDuty findings and a rule on Security Hub HIGH/CRITICAL findings would therefore both fire for the same underlying GuardDuty finding. One event, two emails. Do that across an account and a meaningful fraction of your alert volume is just the same findings counted twice, which is its own kind of noise.

So the Security Hub rule explicitly excludes findings whose ProductName is GuardDuty, with an anything-but match. GuardDuty findings come through the GuardDuty rule. The Security Hub rule handles everything Security Hub adds that GuardDuty didn’t already report. One finding, one alert, regardless of how many services it passed through.

Two tripwires on the root account

Findings are about threats the detectors recognise. The module adds two alarms about something simpler: the root account doing anything at all.

One CloudWatch alarm fires on a root console sign-in. The other fires on any root API call that isn’t a console login. In a well-run AWS account, the root user does almost nothing after initial setup: day-to-day work happens through roles. So root activity isn’t a “finding” to be assessed for severity. It’s a tripwire. Any of it, in an account that should be silent, is worth an immediate look, and the two alarms say so directly.

Why a quiet alert stream matters here

This is monitoring for the account that’s going to hold the release-signing key, and that raises the stakes on getting the routing right.

If a key-bearing account ever does come under attack, the alert that says so has to be seen. An alert stream that’s mostly noise and duplicates is, functionally, no alerting at all, because the people who’d act on it have long since tuned it out. Routing the stream down to “severe, deduplicated, plus root tripwires” is what keeps it something a human will still read on the day it finally matters.

The short version

GuardDuty and Security Hub make detection easy. The hard, valuable part is routing: forwarding what deserves to interrupt someone and leaving the rest in a console.

The alerts module forwards GuardDuty at severity 7-plus and Security Hub at HIGH/CRITICAL, and it drops the duplicate that aggregation creates by excluding GuardDuty-sourced findings from the Security Hub rule, so one finding is one alert. Two CloudWatch alarms act as tripwires on root-account activity, which should be near-zero. For the account that will hold the signing key, a quiet, trustworthy alert stream isn’t a nicety. It’s the difference between monitoring and theatre.

Why go-tool-base left GitHub for GitLab

Mon, 11 May 2026 00:00:00 +0000

A botched version bump made me stop and actually look at where go-tool-base lived, and I didn’t much like what I saw. GitHub had spent months quietly falling over, and when Mitchell Hashimoto (GitHub user #1299, no less) publicly walked Ghostty off the platform, it stopped feeling like just my problem. I’ve been a GitLab fan for years, so the move was less a leap and more an overdue nudge. This is the why, not the how.

It started with a wrong number

Every migration has a trigger, and mine was embarrassingly small. A commit landed on main carrying a BREAKING CHANGE: footer it didn’t really deserve. Semantic-release did exactly what it’s told to do with that footer: it cut a major version. go-tool-base lurched from the v1 line straight to v2.0.0, and a chain of things that keyed off the version went sideways with it.

It was fixable. It wasn’t a disaster. But it was the kind of small, stupid breakage that makes you stop and actually look at your setup instead of just patching it and moving on. And when I looked, the version bump wasn’t the thing that bothered me. It was everything around it.

The platform had been quietly failing

I’d been losing time to GitHub for months. Not dramatically. No single outage you’d write home about, just a steady drip of Actions queues that wouldn’t drain, pull requests that wouldn’t merge, the occasional morning where the thing simply wasn’t there. You absorb it. You re-run the job. You make a coffee and try again. You tell yourself it’s a blip.

The trouble with a steady drip is that you stop counting it. It becomes weather.

The canary left the mine

Then, in late April, Mitchell Hashimoto (co-founder of HashiCorp, creator of Vagrant, Terraform and the Ghostty terminal) published Ghostty Is Leaving GitHub, and The Register picked it up a day later under the headline “GitHub ’no longer a place for serious work’”.

This is not a man with a casual relationship to GitHub. He’s, by his own account, user #1299, joined February 2008. He called it “the place that has made me the most happy”. And he still wrote this:

This is no longer a place for serious work if it just blocks you out for hours per day, every day.

The detail that landed hardest for me wasn’t a quote, it was a habit. He’d kept a journal for a month, marking an “X” on every day a GitHub outage had cost him working time. Almost every day had an X. Reading that, I realised I’d been having the same month. I’d just never been disciplined enough to write it down. He’d turned my vague “it’s been flaky lately” into a row of crosses on a calendar.

I want to ship software and it doesn’t want me to ship software.

When the person who’s been on the platform for eighteen years and loves it says that out loud, it stops being your private grumble. It’s the canary, and the canary has stopped singing.

Why GitLab, and not just “somewhere else”

Being annoyed at GitHub is a reason to leave. It is not, on its own, a reason to pick a destination. The destination has to be a positive choice.

For me GitLab was an easy one, because I’ve been a fan for years. Long enough, in fact, to have also been a reliable grumbler about their pricing tiers, which is how you know it’s a real relationship and not a honeymoon. What I’ve always rated is the model: GitLab treats source hosting, CI/CD, the package registry, releases and Pages as one integrated product, not a marketplace of bolted-on parts you assemble yourself.

That integration is the actual prize. On the old setup, “CI” meant a folder of separate GitHub Actions workflow files, each pinned, each its own little world. On GitLab it’s a single .gitlab-ci.yml pipeline with proper stages (lint, test, security, docs, release) and the release stage talks to the built-in package registry and Pages without me wiring up a single external credential. The CI job that builds the project can authenticate to the things the project needs because they’re the same platform.

There’s a second-order benefit too. A migration is a rare licence to fix things you’d never otherwise touch. Moving gave me the cover to reset go-tool-base’s versioning cleanly (back to a sensible v0.x line, the accidental v2.0.0 left behind as a cautionary tale) and to move the module path to its new home in one deliberate change rather than a thousand apologetic ones.

What I’m not going to claim

I’m not going to tell you GitHub is finished, or that GitLab never has a bad day, because it does, everyone does. This isn’t a teardown. GitHub gave go-tool-base a perfectly good home for its first year, and the archived mirror is still sitting there, read-only, pointing anyone who finds it at the new place.

What changed is simpler than a grand verdict. The friction crossed a line, someone I respect said the quiet part loudly enough that I couldn’t keep filing it under “weather”, and the place I’d have moved to anyway was sitting right there with a better model. Sometimes the prudent move and the move you secretly wanted turn out to be the same move, and you just need a wrong version number to give you permission.

Boiling it down

go-tool-base moved from GitHub to GitLab in May 2026. The proximate cause was a self-inflicted version-bump mess; the real cause was months of GitHub unreliability that I’d stopped consciously noticing until Mitchell Hashimoto’s very public departure named it for me. GitLab was a positive pick, not just an escape hatch: its integrated CI/CD, registry, releases and Pages are one product rather than a kit, and that integration is genuinely worth having. The migration also bought a clean versioning restart as a bonus.

If you’ve been absorbing a steady drip of friction and telling yourself it’s normal: try the calendar trick. Mark the X’s for a month. The page will tell you something you already half-know.

Why I hand-rolled every module

Sun, 10 May 2026 00:00:00 +0000

There are well-known community module libraries for AWS: Cloud Posse, the terraform-aws-modules collection, plenty more. Both terraform-aws-bootstrap and terraform-aws-security-baseline use almost none of them. Every sub-module is hand-rolled from raw AWS resources, and before you accuse me of not-invented-here syndrome (a perfectly fair first guess), hear me out, because the same evaluation kept landing the same way for a real reason.

The promise of a wrapper module

The community module ecosystem makes an appealing offer. Don’t write raw aws_s3_bucket and aws_s3_bucket_policy and aws_s3_bucket_public_access_block and the rest. Call a tested, popular module, pass it a handful of inputs, and get a correct, well-configured bucket. Less code in your repo, and the code you don’t write has been exercised by thousands of other users.

For a lot of infrastructure that’s a genuinely good deal, and I take it often. For the two infrastructure modules in this series, I took it almost never. Every sub-module is built from raw AWS resources. That wasn’t a reflex. It was the same evaluation, made over and over, landing the same way.

What kept going wrong

For each place a wrapper module could have fitted, I looked at the wrapper. And the recurring finding was one of two things. Either using the wrapper correctly, with all the overrides my posture needed, came to more configuration than the raw resources would have. Or the wrapper’s abstraction leaked the instant I needed something it hadn’t anticipated, and I was now writing code to fight it.

The CloudTrail bucket, concretely

The clearest example is the bucket that holds CloudTrail logs.

There are popular modules that set up CloudTrail and bundle an S3 bucket for the logs. Convenient. But that bundled bucket isn’t the bucket I want. It doesn’t carry lifecycle { prevent_destroy = true }, and its bucket policy is weaker than the one the state bucket taught me to want: TLS-only, SSE-KMS-only, wrong-key-denied.

So to use the wrapper I had two options. Accept a weaker audit-log bucket than the rest of the account, which rather defeats the point of an audit log. Or fight the wrapper: disable its bucket, create my own, wire it back in. Fighting the wrapper is more work than simply writing the fifty-odd lines of raw aws_s3_bucket plus policy that give me exactly the posture I’d already designed once. The wrapper didn’t save code. It added a negotiation.

A wrapper is a deal, and deals have terms

This isn’t an argument that community modules are bad. It’s an argument about when the deal is good.

A wrapper module is a good deal while its abstraction holds: while what it assumes you want matches what you want. The moment you need something it didn’t anticipate, the deal inverts. Now you’re working against the abstraction, and an abstraction you’re fighting costs more than no abstraction at all. (Regular readers will recognise that line from the LangChain argument; it’s the same principle in a very different language.)

Infrastructure that holds signing keys is precisely the case where you need to control the specifics: every encryption setting, every lifecycle rule, every line of every bucket policy. That’s a domain where wrapper abstractions leak fast, because the whole job is the details the wrapper smoothed over.

The cost, paid on purpose

Hand-rolling isn’t free. It’s more lines of HCL in the repo, up front, than a one-line module call.

What those lines buy is worth the price for this kind of infrastructure. There’s no transitive module-version churn to track. There’s no abstraction between me and the resource when something behaves oddly. And every line is one I can read, and defend, in a security review, because I wrote it and it says exactly what it does. For a foundation that will hold the most sensitive key in the system, “readable and mine” beats “short and someone else’s”.

That’s a deliberate trade, not a universal rule. For an internal tool on a deadline, reach for the wrapper. For the security-critical base of everything else, the raw resources won every time I checked.

To sum up

The community module ecosystem offers less code that more people have tested, and for plenty of infrastructure that’s the right call. For terraform-aws-bootstrap and terraform-aws-security-baseline it almost never was, because each wrapper turned out to be more configuration than the raw resources once my posture was accounted for, or it leaked the moment I needed a specific.

The CloudTrail log bucket is the pattern in miniature: the bundled bucket lacked prevent_destroy and a strong policy, so using the wrapper meant either a weaker bucket or fighting the module. A wrapper is a good deal while its abstraction holds and a bad one the moment you fight it, and security-critical foundation infrastructure is all specifics. Hand-rolling cost more lines and bought code I can read and defend. For this, that was the trade worth making.

Hardening the account that will hold the keys

Sat, 09 May 2026 00:00:00 +0000

Bootstrapping the account got it ready: somewhere to store state, an identity to deploy as, enough for the next tofu apply to run. Ready is not the same as safe. An account with no audit trail, nothing watching it, and no considered way for a human to get in is fine for experimenting and absolutely not where you put the most sensitive key in the system. So before the signing key goes anywhere near it, the account gets a security baseline.

Ready is not the same as safe

The bootstrap post ended with an account that was ready: it had somewhere to store state and a CI identity to deploy as. The next tofu apply could run.

Ready is not safe. That account still has no audit trail, so nobody could tell you afterwards what happened in it. It has no threat detection, so nothing is watching. Its defaults are AWS’s defaults, which are not a security posture. There’s no considered way for a human to get in. An account in that condition is fine for experimenting. It’s not somewhere you put the most sensitive key in the whole system.

So before the signing key is anywhere near it, the account gets a security baseline.

The baseline, in one downstream stack

terraform-aws-security-baseline is that baseline, and it’s exactly the downstream stack the bootstrap post promised: applied through the automation role bootstrap created, not bootstrapped specially.

It’s six sub-modules, each behind an enable_* toggle: account-hardening (IAM password policy, account-wide S3 public-access blocking, default EBS encryption), audit-logging (a multi-region CloudTrail with log-file validation), aws-config, threat-detection (GuardDuty, Security Hub, IAM Access Analyzer), alerts, and operator-role. Together they turn a bare account into one that records what happens, watches for trouble, and controls who gets in.

Most of those are the expected baseline. The operator role is the one worth slowing down on, because it’s built backwards from how people usually think about an admin role.

The operator role, and the inversion

InfraAdmin is the human way into the account: the role a person assumes to do operator work. Two things define it.

The trust policy decides who may assume it. It trusts only the account root principal, and it requires multi-factor authentication: the assume call must carry aws:MultiFactorAuthPresent, and aws:MultiFactorAuthAge bounds how recently that MFA was performed. No MFA, no role. So far this is a careful but ordinary admin role.

The inversion is a second, separate inline policy, and it’s almost entirely Deny. It denies, using NotAction, anything where aws:RequestedRegion falls outside an allowed set of regions. The role’s power comes from an admin grant. This inline policy fences that power.

That’s the part worth holding onto. People picture an admin role as a list of what it can do. This one is better understood by what it cannot: it cannot act outside its permitted regions, full stop. A fat-fingered command, or a compromised session, cannot quietly spin resources up in some region nobody’s watching. The fence is as much the point of the role as the grant is.

The carve-out, because honesty

There’s a fiddly detail, and it’s the kind of thing that makes the region fence real rather than theoretical.

Some AWS services are global. IAM, CloudFront, Route 53 and friends have no region, and they don’t honour aws:RequestedRegion. A naive region-deny would therefore deny calls to IAM, and you’d lock yourself out of the very service you manage access with. (A close cousin of the kind of self-inflicted lockout I’ll come back to in a later post.)

So the Deny carries explicit carve-outs for the global services. It isn’t elegant, and it can’t be: the global-versus-regional split is just a fact of AWS, and a correct region fence has to account for it. The carve-out list is the real cost of the control working.

Harden the room, then move the keys in

There’s an order to all of this, and the order is the argument.

The account that will hold the signing key has to be audited before the key arrives, so that from day one every call against it is in CloudTrail. It has to be watched before the key arrives, so GuardDuty is already looking. It has to be access-controlled before the key arrives, so the only human path in is MFA-gated and region-fenced.

You don’t move something valuable into a room and then think about locks. You build the room, fit the locks, check they work, and then move the valuable thing in. The security baseline is fitting the locks. The signing key comes later, into a room already built for it.

Worth remembering

Bootstrapping an account makes it ready for the next deploy. It does not make it safe to hold anything that matters. terraform-aws-security-baseline is the downstream stack that closes that gap: audit logging, AWS Config, threat detection, account hardening, and an operator role, applied through the CI role bootstrap created.

The operator role is the piece to study. It’s MFA-gated on the way in, and then fenced by a separate, almost-all-Deny inline policy that confines it to permitted regions, with carve-outs for the global services that have no region. An admin role defined as much by its fence as its grant. Harden the room first; the keys move in afterwards.

No access keys in CI

Fri, 08 May 2026 00:00:00 +0000

A long-lived AWS access key, sitting in a CI system, is just about the single credential I’d most like to be rid of. It’s powerful, it never expires unless someone remembers to rotate it (nobody remembers to rotate it), and it lives in one of the most attractive targets in the whole supply chain. For infrastructure that’s eventually going to hold a release-signing key, it’s exactly the wrong place to start. So the phpboyscout infrastructure has no AWS access key in CI at all. None.

The access key you don’t want

A CI pipeline that runs tofu apply against AWS needs AWS credentials. The traditional way to give it some is an IAM user with an access key pair, pasted into the CI system as a masked variable.

Look at what that key is. It’s long-lived: it works until someone remembers to rotate it, and rotating it is a chore, so mostly nobody does. It’s powerful: it can apply infrastructure, so it can do nearly anything. And it’s sitting in a CI system, which is one of the most attractive targets in your whole supply chain. You’ve taken your highest-value credential and stored a permanent copy of it in a place built for running automated jobs.

For infrastructure that’s going to hold a release-signing key, that’s precisely the wrong starting point. So the phpboyscout infrastructure has no AWS access key in CI at all. Not a well-guarded one. None.

Federation instead of a stored secret

The replacement is OIDC federation, and the shape of it is worth walking through, because it’s genuinely different from “a secret, but better”.

A modern CI platform can mint an OIDC token. GitLab does this with an id_tokens: block: at job time, GitLab issues a short-lived JSON Web Token, signed by GitLab, that asserts a set of facts. This is project X. This is pipeline Y. This is running on ref Z, of this type.

AWS can consume that. The sts:AssumeRoleWithWebIdentity call takes such a token and, if it satisfies an IAM role’s trust policy, returns short-lived AWS credentials for that role. The trust policy is where the control lives: it names GitLab as a trusted token issuer, and it constrains the token’s sub claim so that only the specific project, and the specific refs, you intend can assume the role.

Put it together: the pipeline asks GitLab for a token, hands it to AWS, and gets back credentials that last about an hour and are scoped to one role. Nothing long-lived is stored anywhere. The credential exists only for the job that needs it, and it can’t be stolen from a CI variable store, because it was never in one.

Two halves of one handshake

That handshake is built by two of the repos in this series, each owning one side.

terraform-aws-bootstrap builds the AWS half, in its automation-iam module: it registers GitLab as an OIDC identity provider in the account, and it creates the automation role with the trust policy that decides which pipelines may assume it.

The CI components build the consuming half: the id_tokens: block that asks GitLab for the JWT, and then simply letting the AWS provider’s own credential chain perform the exchange. The pipeline doesn’t call sts by hand. It presents the token; the SDK does the rest.

The gotcha: don’t set a profile

There’s one quiet way to break this, and a stack can look completely correct while doing it.

The AWS SDK finds credentials by walking a chain of sources in order. The web-identity path, the one that uses the OIDC token, is one link in that chain. It triggers off environment variables the CI sets up automatically.

But if the aws provider block has a hardcoded profile = "...", the SDK takes the profile link of the chain instead, and never reaches the web-identity link. A profile line is the sort of thing that ends up in a provider block from someone’s local development setup, where it’s exactly right. Committed and run in CI, it silently short-circuits the federation. The pipeline either fails to find credentials, or finds the wrong ones.

The rule is simple once you know it: the provider block that runs in CI must not name a profile. Leave the chain free to find the web identity. It’s the kind of bug that teaches you to be precise about which link of the credential chain you’re actually relying on.

The bottom line

Giving CI an AWS access key means storing your most powerful, longest-lived credential in one of your most exposed systems. OIDC federation removes it entirely. The CI platform mints a short-lived signed token, AWS exchanges it via AssumeRoleWithWebIdentity for hour-long credentials against a role whose trust policy names the exact pipeline, and nothing permanent is stored.

terraform-aws-bootstrap builds the AWS side, the identity provider and the trust policy; the CI components build the consuming side, the token request. The one trap is a hardcoded profile in the provider block, which short-circuits the SDK’s credential chain before it reaches the web-identity path. Get that right, and a pipeline deploys to AWS as a verifiable, short-lived identity, with no key to steal.

Two layers of tags, and which one wins

Fri, 08 May 2026 00:00:00 +0000

Tagging cloud resources is one of those jobs that’s trivial to do badly and surprisingly fiddly to do well. Everyone agrees resources should be tagged. The argument nobody quite has out loud is where the tags should come from, and getting that wrong gives you either a giant copy-pasted tag block on every resource, or a set of tags that quietly disagree with each other across the account.

Tags answer two different questions

If you look at what tags are actually for, they split cleanly into two kinds, and the split is the whole point.

Some tags are true of every single resource in the account, identically. The environment it belongs to. The fact that OpenTofu manages it. The project or owner it rolls up to for cost reporting. These are invariants: a resource that didn’t carry them would be the bug.

Other tags are specific to a particular piece of infrastructure. Which component this resource belongs to, what subsystem it’s part of. The CloudTrail bucket is part of audit logging; the Config recorder is part of aws-config. That’s a fact about the module, not about the account.

Treat those two kinds the same and you end up repeating the invariants by hand on every resource, which is exactly the copy-paste that drifts. So the infra setup gives each kind its own home.

Layer one: declared once, on the provider

The invariants live on the AWS provider itself, as default_tags, set one time in the provider block:

provider "aws" {
 # ...
 default_tags {
 tags = {
 # Environment, project, managed-by: the things true of
 # every resource in this account.
 }
 }
}

default_tags applies those tags to every taggable resource the provider creates, automatically, without a single resource having to mention them. Change the environment label once, here, and it propagates to everything on the next apply. No resource carries a copy; they all inherit the originals. The invariants are stated exactly once, in the one place that’s true for all of them.

Layer two: merged in by the module

The resource-specific tags live where the resource does: inside the module. Each module merges its own component tag over whatever tags it was handed, which you can see in the public terraform-aws-security-baseline modules:

tags = merge({ Component = "aws-config" }, var.tags)

So the aws-config module stamps Component = "aws-config" onto the things it builds, the account-hardening module stamps its own, and so on. The caller can pass extra tags down through var.tags, and because they come last in the merge, the caller can override the module’s defaults when it genuinely needs to. Module-specific knowledge stays in the module; per-call adjustments stay with the caller.

Which layer wins

Now the question that actually bites: a resource is getting tags from the provider’s default_tags and from the module’s merge. What happens when both set the same key?

The resource-level tags win. AWS’s provider treats tags set directly on a resource as an override of default_tags on a key collision, so the module’s merged tags take precedence over the account-wide defaults. That’s the right way round: the invariants are a sensible baseline, and a module that has a specific reason to set a key differently can, without having to reach up and edit the provider block that everything else depends on. Most of the time the two layers are simply disjoint, the invariants saying what account this is and the module tags saying what this resource is for, and they never collide at all. When they do, local intent beats the global default, which is the precedence you’d want.

Why bother splitting it

The payoff is that neither layer has to know about the other. The provider declares the invariants once and never thinks about components. Each module declares its component and never hard-codes the environment. Add a new module and it inherits every account-wide tag for free, while contributing its own. Change an account-wide tag and you touch one block, not two hundred resources. The tags stay consistent not because someone’s policing them, but because the place each tag is declared is the one place it can be declared.

The short version

Resource tags answer two questions, and they want two homes. Account-wide invariants (environment, ownership, managed-by) go on the provider’s default_tags, declared once and inherited by everything. Resource-specific tags go in the module, via merge({ Component = "..." }, var.tags), so each module owns its own labels and the caller can still override. On a key conflict the resource-level tag wins, which means the module’s intent beats the account default exactly when it should. Two layers, each declared in the one place it belongs, and no copy-pasted tag block anywhere in sight.

clap's global flag, except in a passthrough subtree

Thu, 07 May 2026 00:00:00 +0000

--output json worked everywhere. On the top-level command, on every ordinary subcommand, wherever the user fancied putting it. Then it stopped working in exactly one place, and of course it was the subcommand I’d been clever about.

How the global flag is meant to work

clap has a lovely feature for this. Define --output text|json once at the top, mark it global = true, and it’s reachable from every subcommand: mytool --output json widget and mytool widget --output json land the same. You stop thinking about it.

The one place it goes missing

One subcommand, credentials, is a passthrough: it sets subcommand_passthrough = true, which makes clap capture everything after the subcommand name as trailing_var_arg and hand it on, the way cargo run -- ... passes the trailing args to your program rather than to cargo. The handler then re-parses those captured tokens against its own clap definition.

The trouble is that the captured tokens include --output. clap’s global = true propagation doesn’t reach a passthrough subtree, because the post-name tokens are taken as trailing_var_arg before the outer parser ever sees them. So in this one subtree the global flag isn’t applied, and worse, when the inner parser re-parses the captured args it meets --output, which it doesn’t define, and rejects it as unknown. The code says so where it matters, in crates/rtb-cli/src/credentials.rs:

// clap's outer `global = true` propagation works for normal
// subcommands, but `subcommand_passthrough = true` captures
// post-name tokens as `trailing_var_arg`, so the global
// never reaches the outer parser for this subtree.
args = strip_global_output(args);

Parse it yourself, then strip it

The fix is two moves. First, parse --output out of the raw args by hand (there’s an OutputMode::from_args_os for exactly that), so the output mode is still honoured. Then strip --output out of the args before the inner parser runs, so the inner clap doesn’t choke on a flag it doesn’t define. strip_global_output is the second move, from crates/rtb-cli/src/render.rs:

if s.starts_with("--output=") {
 continue; // inline form: drop just this token
}
if s == "--output" {
 iter.next(); // space-separated form: drop the token and its value
 continue;
}

It handles both --output=json and --output json, and it’s idempotent, so it’s safe to call whether or not the flag is actually present.

The takeaway

global = true and trailing_var_arg are both “grab the args” features, and in a passthrough subcommand they reach for the same tokens. clap won’t arbitrate that overlap, and shouldn’t try to guess. So you arbitrate: parse the global out of the raw args yourself, strip it before you re-parse the rest, and the flag that “works everywhere” actually does.

Secrets that scrub themselves from RAM

Wed, 06 May 2026 00:00:00 +0000

A while ago I worked out where a CLI should keep your API key: env var, OS keychain, or, grudgingly, a literal in the config file. That answers where the secret lives. It says nothing about what happens to it once it’s loaded and sitting in your process memory, which is the half where secrets actually tend to leak. Rust, it turns out, can do something about that half that Go simply can’t.

What go-tool-base already settled

A while back I wrote about where a CLI should keep your API keys. The answer go-tool-base settled on was three storage modes, in a fixed precedence: an environment variable reference (the recommended default), the OS keychain (opt-in), or a literal value in the config file (legacy, and refused outright when CI=true).

rust-tool-base keeps that design unchanged. Same three modes, same precedence, same refusal of literal secrets in CI. A tool embeds a CredentialRef in its typed config, and a Resolver walks env, then keychain, then literal, then a well-known fallback variable, first hit wins. That part is a straight carry-over, because where to keep the secret was design, and design survives the port.

But storage is only half the life of a secret. The other half is what happens to it once it’s resolved and sitting in your process memory. That’s where Rust can do something Go can’t, and rust-tool-base takes the opening.

The two ways a secret leaks after you’ve loaded it

You’ve resolved the API key. It’s a value in memory now. Two very ordinary things can leak it from there, and neither involves your storage being wrong.

The first is the log line. Somewhere a developer writes a debug print of a config struct, or an error includes the struct that holds the key, or a panic dumps it. The secret is a string like any other string, so it renders like any other string, straight into a log aggregator that a lot of people can read.

The second is the leftover bytes. The key sat in a heap allocation. The variable goes out of scope, the allocation is freed, and on most runtimes “freed” just means “returned to the allocator”. The bytes are still there until something else writes over them. A core dump taken in that window contains your key. So does the next allocation that happens to land on that memory and gets logged before it’s overwritten.

A Go string can’t really defend against either. Go strings are immutable, so you can’t zero one in place; the runtime copies them freely, so you can’t even track every copy; and there’s no compile-time barrier stopping anyone printing one. You can be disciplined, but discipline is all you’ve got.

`SecretString` closes both

rust-tool-base routes every secret through secrecy::SecretString, and the crate is explicit that taking a plain &str or String for a secret is a type error, not a style preference.

For the log line, SecretString has its own Debug implementation, and it prints [REDACTED]. Always. A config struct holding a SecretString can be debug-printed, put in an error, caught in a panic, and the secret field shows up as [REDACTED] every single time. You don’t have to remember not to log it. The type already won’t.

For the leftover bytes, SecretString zeroes its memory when it’s dropped. When the value goes out of scope, before the allocation is handed back, the bytes are overwritten. The window where a freed allocation still holds your key is closed. A core dump taken afterwards finds zeroes.

There’s a third leak SecretString blocks that’s easy to miss. It deliberately doesn’t implement Serialize. You cannot serialise a SecretString. That sounds like an inconvenience until you see what it prevents: a tool that loads config, changes one setting, and writes the whole struct back would, with an ordinary string, faithfully write the resolved secret to disk in plain text. Because SecretString can’t be serialised, CredentialRef can’t be either, and that accident is structurally impossible. Writing a secret back is a deliberate, separate path, never a side effect of saving config.

When code genuinely needs the raw value, to drop it into an Authorization header, it calls expose_secret(). The name is the point. Getting at the plaintext is one explicit, greppable, reviewable call, and everywhere else the secret stays wrapped.

Discipline versus the type system

The plain framing is this. None of these leaks are exotic. Logging a struct, a core dump after a free, re-saving a config file: they’re all routine, and they’re all how real credentials end up somewhere they shouldn’t.

go-tool-base’s storage design is good, and rust-tool-base kept it. But in Go, not leaking the secret once it’s in memory comes down to every developer being careful every time. In Rust, SecretString makes the type system carry it. The redaction, the zeroing, the un-serialisability aren’t things you remember to do. They’re things the secret does to itself because of what it is. That’s the part Go structurally can’t match, and it’s why the port didn’t just copy the storage modes across, it tightened the handling underneath them.

The gist

go-tool-base settled where a CLI keeps a secret: env var, keychain, or literal, in a fixed precedence. rust-tool-base keeps that design and hardens what happens once the secret is loaded.

Every secret is a secrecy::SecretString. It debug-prints as [REDACTED], so it can’t fall into a log by accident. Its memory is zeroed on drop, so it doesn’t survive in freed heap. It isn’t serialisable, so it can’t be written back to config by a blanket save. Getting the plaintext is one explicit expose_secret() call. Go can only ask developers to be careful with a secret in memory; Rust lets the type be careful for them.

The chicken-and-egg of remote state

Wed, 06 May 2026 00:00:00 +0000

Here’s a puzzle that every infrastructure-as-code setup hits exactly once, right at the very beginning, and then never again. An OpenTofu stack stores its state in a backend. The bootstrap stack I wrote about last time has a particular job, and part of that job is to create the backend that remote state lives in. So where does the bootstrap stack store its own state, on the very first run, before it’s built the place state is supposed to go?

Where does the state of the thing that makes the state store live?

That’s the puzzle, and it’s a real ordering deadlock rather than a riddle.

An OpenTofu stack keeps a state file, and for anything shared that state file lives in a remote backend: on AWS, an S3 bucket. Fine. But the bootstrap stack has a particular job, and part of that job is to create the S3 bucket that remote state lives in.

So walk through the first run. Bootstrap has never been applied. The state bucket doesn’t exist, because creating it is what bootstrap is for. Bootstrap needs somewhere to store its own state. The only place that would make sense is the bucket it’s about to create, which isn’t there yet. The thing that builds the state store can’t store its state in the state store.

Run local, then migrate

The way out is a two-step that OpenTofu supports directly.

Bootstrap starts configured with a local backend: backend "local" {}. State is just a file on the operator’s machine. With that in place, the first tofu apply runs. It creates the S3 bucket and the KMS key, and records all of it in the local state file.

Now the bucket exists. So the backend configuration is rewritten to point at it: an s3 backend block naming the new bucket. Then tofu init -migrate-state. OpenTofu sees the backend has changed, picks up the local state file, and copies it into the S3 bucket. From that point on, bootstrap’s own state lives in the bucket that bootstrap created. The egg has laid the chicken.

The local backend was a scaffold. It existed for exactly one apply, to break the ordering deadlock, and then the state moved off it and it was never used again.

It happened twice

The infra repo actually did this migration twice, and the second time is the proof that the pattern is general rather than a one-off trick.

The first migration was the one above: local to S3, at the very start. The second came later, during the move from GitHub to GitLab. GitLab offers a managed HTTP state backend, and infra chose to use it. So the backend block was rewritten again, this time from s3 to http, and tofu init -migrate-state ran again, copying the state from the S3 bucket to GitLab’s backend.

The same move, twice, against three different backends. That’s the useful lesson hiding in the chicken-and-egg story. State is portable. The backend is just where you currently keep it, not a property of the stack itself, and moving it is a routine, supported operation rather than surgery.

Why this is the honest answer, not a hack

It’s easy to look at “apply once with a local backend, then migrate” and feel it’s a bit of a smell, a workaround for something that should have been cleaner.

It isn’t. It’s the honest answer to a real ordering problem, and the alternatives are worse.

The obvious alternative is to create the state bucket by hand, in the console, before running bootstrap at all. But then the most important bucket in the account is unmanaged. It exists outside every OpenTofu graph, nobody’s code describes it, its encryption and policy and prevent_destroy are whatever someone clicked that day, and it drifts. The local-then-migrate dance avoids exactly that. The bucket is created by bootstrap, described in code, and tracked in bootstrap’s own state from its very first apply. It’s managed from birth.

The chicken-and-egg isn’t a flaw to be embarrassed about. It’s just the shape of the problem when a stack has to build its own foundations, and OpenTofu’s -migrate-state is the supported tool for exactly that shape.

Pulling it together

Every OpenTofu stack needs a backend to store state, and the bootstrap stack’s job is to create the backend, so on its first run the bucket it needs doesn’t yet exist.

The resolution is to run bootstrap once with a local backend, let that apply create the bucket and key, then rewrite the backend configuration and tofu init -migrate-state the state into the bucket bootstrap just made. The infra repo did it twice, local to S3 and later S3 to GitLab, which shows the real point: state is portable, and the backend is just where you keep it. Doing it this way, rather than hand-creating the bucket, is what keeps that critical bucket managed in code from its very first day.

The cleanup tool that almost deleted its own hands

Tue, 05 May 2026 00:00:00 +0000

The first time I pointed aws-nuke at a real account, the dry-run printed hundreds of lines of angry red text and my stomach dropped. Then I read it properly, and two things turned out to be true at once. Almost all of that red was noise. And the one operation I genuinely should have worried about wasn’t red at all.

A tool whose whole job is destruction

aws-nuke deletes everything in an AWS account. That’s the point of it: when you spin up a throwaway account to try something, aws-nuke is how you tear it back down to nothing afterwards rather than leaving resources quietly billing forever. go-tool-base’s bootstrap renders a scoped aws-nuke config for exactly this, from a nuke-config module, so the teardown is described in code rather than typed by hand at the worst possible moment.

A tool that deletes everything is a tool you run in dry-run first, every single time, and read the output before you let it touch anything. So that’s what I did. And the output was alarming in a way that turned out to be completely meaningless, and reassuring in a way that turned out to hide the one real hazard.

The wall of red that means nothing

A fresh account threw up screen after screen of SubscriptionRequiredException. Hundreds of lines, all red, all looking like something had gone badly wrong.

They hadn’t. aws-nuke works by asking every region “do you have any of this kind of resource?”, for every kind of resource it knows about. On a brand-new account you’ve never enabled most services in most regions, so the API’s honest answer is “you’re not subscribed to that here”, which surfaces as an exception, which the tool dutifully logs in red. It isn’t a failure. It’s the sound of an empty account being asked four hundred questions and answering “nothing here” to almost all of them.

The skill, and it is a skill, is learning to read a destructive tool’s dry-run and tell the noise from the signal. SubscriptionRequiredException on a fresh account is noise. Once you know that, the wall of red stops being frightening and becomes scenery.

There’s a related trap in the same neighbourhood, and the nuke-config module’s own regions variable documents it. aws-nuke has a special global pseudo-region for things that don’t live in one place (IAM, Route 53, CloudFront), and then the actual regions for everything else. It also accepts all, meaning every enabled region. Mixing all with explicit region values scans some regions twice and muddies the output, so the module’s guidance is to pick one approach or the other. More scenery you have to learn to read before the genuinely important line will stand out.

The line that should have scared me, and didn’t look like it

Buried in that calm-looking eye of the storm, among the resources aws-nuke intended to delete, were the IAM resources granting the identity running the nuke its administrative access.

Sit with that for a second. aws-nuke runs as some principal with enough power to delete everything. To delete EVERYTHING, it has to delete IAM resources too. And if the plan deletes the very grant that gives the running identity its admin before it’s finished, the tool strands itself partway through: no permissions left to complete the teardown, and now you’ve got a half-nuked account and a principal that can’t act on it. The cleanup tool sawing off the branch it’s standing on, calmly, without a single red line to warn you, because from the API’s point of view deleting that resource is a perfectly valid request.

That’s the operation that actually mattered, and it was the quietest thing in the output.

Two ways to keep its hands attached

The fix has two halves, one explicit and one structural.

The explicit half is to preserve the privilege path. The nuke-config module passes a caller-supplied set of filters straight through into the rendered config, so you tell aws-nuke “everything except these”. You exclude the identity running the nuke, and the policy and path that grant it admin, from deletion. The tool cleans the account and leaves its own hands alone, because you told it which resources are off-limits.

The structural half is to not give it a tempting separate thing to delete in the first place. If an identity gets its admin through an IAM group it belongs to, that group is its own deletable resource, one more thing in the plan, one more way to be stranded. The automation role in terraform-aws-bootstrap instead takes its policies as direct attachments to the role itself:

resource "aws_iam_role_policy_attachment" "gitlab" {
 for_each = local.is_gitlab ? var.policy_arns : {}
 role = aws_iam_role.this.name
 policy_arn = each.value
}

No intermediary group sitting there as a separate, deletable object. Flattening the privilege onto the role makes the dependency simpler to reason about and gives the cleanup tool one fewer foot-gun to find. Belt and braces: filter the path out explicitly, and don’t build a structure that invites the problem.

What it comes down to

A destructive tool’s dry-run is the most valuable thing it produces, and reading it well is its own competence. On a fresh account, the screenfuls of red SubscriptionRequiredException are noise, the sound of an empty account answering “nothing here”, and the all-versus-global region wrinkle is more of the same. Learn to see past all of it, because the operation that can actually hurt you is rarely the one shouting. Mine was the calm, unremarkable line proposing to delete the admin grant the nuke needed to finish its own job.

Keep the cleanup tool’s hands attached: filter the privileged path out of the teardown so it’s never a candidate for deletion, and attach that privilege directly rather than through a group that’s just one more thing to delete. Then let it loose on everything else, which is, after all, what you brought it in to do.

The security finding you must not fix

Mon, 04 May 2026 00:00:00 +0000

A security scanner flagged a finding in my Terraform, and the correct response, the one I had to talk myself into, was to leave it exactly as it was. Not because the finding was wrong about what the code did. It was right. It’s that doing what it asked would have quietly bricked the account.

A finding that looks open and shut

I run checkov over the infrastructure as part of CI, and on the KMS key that protects the Terraform state bucket it raised CKV_AWS_111: a key policy that grants kms:* is overly permissive. On the face of it, unarguable. The policy says the account root can perform any KMS action on the key, with a resource of *. A wildcard action and a wildcard resource is the exact shape a scanner is built to shout about, and ninety-nine times in a hundred it’d be right to.

This was the hundredth time.

Why narrowing it bricks the key

Here’s the bit that turns the finding on its head. That kms:*-for-root statement isn’t an over-broad grant I left lying around. It’s the default key policy AWS itself applies, and it’s load-bearing in a way that’s easy to miss.

A KMS key is administered through its own key policy, and that policy is the only way in. Unlike most resources, IAM permissions elsewhere can’t grant access to a key whose policy doesn’t allow it. So if you “tighten” the key by removing root’s full control, and you don’t perfectly replace it with some other administrative principal, you can end up with a key that nobody can administer. Not you, not root, not a future you with a very good reason. KMS will not let you recover it. The key, and anything it encrypts, is stranded.

The kms:*-for-root statement is what keeps the account’s own root able to manage the key as a last resort. It’s not the vulnerability. It’s the escape hatch, and the scanner was asking me to weld it shut.

So the finding gets suppressed, out loud

The answer isn’t to silence the scanner globally, and it isn’t to obey it. It’s to suppress this finding, on this resource, with the reasoning written right there next to it, from modules/state-backend/main.tf:

data "aws_iam_policy_document" "kms" {
 # checkov:skip=CKV_AWS_111:kms:* on the CMK for the account root is the
 # AWS-documented pattern; narrowing it risks an unrecoverable lockout from
 # the key. See https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-default.html
 statement {
 sid = "AllowAccountRootAdmin"
 effect = "Allow"
 principals {
 type = "AWS"
 identifiers = ["arn:aws:iam::${var.account_id}:root"]
 }
 actions = ["kms:*"]
 resources = ["*"]
 }
}

The skip carries a reason and a link to AWS’s own documentation of why this is the recommended default. That matters more than it looks. A bare # checkov:skip with no explanation is indistinguishable from laziness, and the next person to read it (quite possibly me, a year on) has no way to tell whether it was a considered decision or someone making a red mark go away. A skip with a documented reason is a decision you can audit. The finding is still visible in the sense that the suppression is right there in the code, attached to the thing it’s about, defensible out loud.

A scanner is an argument, not an order

The wider lesson is the one worth keeping, because it generalises well past this one key. A static-analysis finding is a prompt to think, not an instruction to comply with. Most of the time thinking leads you straight to “yes, fix it”, and you should. But a scanner encodes a general rule, and general rules meet specific contexts where they’re wrong, or merely irrelevant, or, in the rare and dangerous case, actively harmful to obey. kms:* for root on a customer-managed key is that last kind: the tool’s general rule (“wildcards are bad”) collides with a hard AWS-specific fact (“root must retain control of the key or it’s gone”).

The discipline that keeps this honest is the one in the code above. You don’t get to ignore a finding. You get to suppress it, scoped to the exact resource, with a reason a reviewer can weigh. Cheap enough that you’ll do it properly, costly enough that you won’t paper over a real finding by reflex.

The bottom line

checkov was right that the state-bucket key grants the account root kms:* on *. It was wrong that I should narrow it, because that statement is AWS’s documented default and the thing that stops the key becoming permanently unadministrable. The fix was a scoped checkov:skip carrying its reasoning and a link to the AWS docs, so the decision lives next to the code and can be defended rather than merely trusted.

Treat your scanner as a sharp, tireless colleague who’s usually right and occasionally, confidently, about to lock you out of your own key. Read every finding. Obey most of them. And write down, in the open, the rare one you mustn’t.

Two telemetry events, one mangled line

Sun, 03 May 2026 00:00:00 +0000

A line in a log file that no parser would touch. Not a wrong value, not a missing field. Half of one telemetry event spliced into the middle of another, like two people typing into the same text box at once. Which, it turns out, is pretty much exactly what had happened.

A format with exactly one rule

rust-tool-base writes its telemetry to a file as JSONL: one JSON object per line, newline at the end, next object on the next line. It’s a lovely format to work with precisely because it has one rule, and the rule is simple. Every line is a complete object. Honour that and you can tail it, grep it, stream it into anything. Break it once and the whole file is suspect, because now a reader can’t trust that a line is a line.

So the one job the file sink has, beyond writing the right bytes, is to never let two events end up sharing a line.

“Appending is atomic, though, isn’t it?”

The mental model I started with, and I suspect I’m not alone, was this: open the file with O_APPEND, write the serialised event, and the operating system tacks it onto the end atomically. Two writers can’t tread on each other because each write goes to wherever the end currently is, no questions asked. I’d half-remembered O_APPEND as the thing that makes concurrent appending safe, full stop.

It’s half true, and the half that’s missing is the half that bit me.

O_APPEND does guarantee one thing: the seek-to-end and the write happen as a unit, so you never get the classic lost-update where two writers compute the same offset and clobber each other. Good. What it does not guarantee, on POSIX, is that a single write() of arbitrary size is atomic with respect to other writers. That atomicity has a ceiling, and the ceiling is PIPE_BUF: 4096 bytes on Linux. Under it, a write is all-or-nothing against other writes to the same file. Over it, the kernel is entirely within its rights to split your write into chunks, and another writer’s bytes can land in the gap between them.

The fat event that went over the edge

For a long time nothing went wrong, which is the most dangerous way for a bug like this to behave. A typical event, a command name, a duration, a status, an attribute or two, serialises to a few hundred bytes. Comfortably under four kilobytes, so comfortably inside the atomic window. Hundreds of them a day, never a problem.

Then an event came along with a lot of attributes on it, and its serialised form sailed past 4 KiB. Two of those emitted at roughly the same moment, both over the line, and O_APPEND did the only thing it had ever promised: it put each write at the end. It said nothing about not interleaving the bytes on the way, because past PIPE_BUF that was never on offer. One spliced line, one file a parser would now choke on.

The fix isn’t a bigger write, it’s a smaller gate

You can’t buy your way out of this with a bigger buffer, because there’s no buffer size that’s reliably atomic above PIPE_BUF. The fix is to stop relying on the kernel for mutual exclusion you can do yourself: serialise the events through a lock, so only one write is ever in flight at a time. The FileSink carries a mutex for exactly that, and the doc comment on it is the whole post in a paragraph, from crates/rtb-telemetry/src/sink.rs:

pub struct FileSink {
 path: PathBuf,
 // Serialises concurrent `emit` calls. Shared across `Clone`s of
 // the same `FileSink` so multiple handles to the same path also
 // serialise correctly.
 gate: Arc<tokio::sync::Mutex<()>>,
}

If you don’t write Rust day to day (the primer has the rest of the basics): tokio::sync::Mutex is an async-aware lock, .await is where a task waits its turn for that lock without blocking the whole thread, and the Arc wrapper is shared ownership. That Arc is the load-bearing bit: it means every clone of the FileSink points at the same gate, rather than each getting its own lock that guards nothing.

The detail I like is where the lock sits. The event is serialised to a string first, outside the critical section, because turning an event into JSON is the expensive part and there’s no reason to hold the gate while you do it. Only then does emit take the lock, and it holds it across the whole open-write-flush, so no other emit can interleave a single byte:

// Serialise the line outside the critical section.
let mut line = serde_json::to_string(&redacted)?;
line.push('\n');

let _guard = self.gate.lock().await;
// ...create parent dir, open with append(true)...
f.write_all(line.as_bytes()).await?;
f.flush().await?;

write_all makes sure the whole line goes out as one logical write from our side, and the gate makes sure ours is the only one happening. The 4 KiB cliff is still there in the kernel. We just never walk near it any more, because we’ve serialised the writers ourselves rather than hoping the OS would.

The bit even the lock can’t fix

There is however a genuine limit, and the comment is upfront about it. The mutex lives in the process. Two FileSinks in two different processes, both pointed at the same file, are back to relying on O_APPEND alone, and back under the 4 KiB ceiling. The lock can’t reach across a process boundary, so it doesn’t pretend to. The guidance there is the older, duller, correct one: give each process its own file and aggregate them somewhere else. Don’t have two processes fighting over one log file and expect the filesystem to referee.

What it comes down to

O_APPEND is a real guarantee, just a much smaller one than its name talks you into. It keeps your write at the end of the file, and it keeps concurrent writes from interleaving only while they stay under PIPE_BUF, which on Linux is 4096 bytes. A fat JSON event slides straight over that and takes your file’s one rule with it.

The fix was never exotic. Serialise the line, take a mutex, do the write under it, and the interleave can’t happen because there’s only ever one writer at a time. The POSIX manual had all of this written down long before I went and learned it the interesting way, which is, I’m told, how most people meet PIPE_BUF too.

A state bucket that defends itself

Sat, 02 May 2026 00:00:00 +0000

OpenTofu’s remote state file is, quietly, the most sensitive thing in an infrastructure repo. It’s a plain JSON document listing every resource you manage, every ID, and, depending on your providers, the odd secret in clear text. So the S3 bucket that holds it can’t just be a bucket. It has to actively defend itself, on three separate fronts.

The most sensitive file in the repo

OpenTofu, like Terraform, keeps a state file: a JSON document recording every resource the stack manages, its real-world ID, and its attributes. It’s how the tool knows what already exists. It’s also, quietly, the most sensitive file in the whole repo. It can hold resource identifiers an attacker would value, and depending on the providers in play it can hold secret values in clear text.

Three bad things can happen to it. It can be deleted, and now the tool has forgotten everything it manages. It can be read by someone who shouldn’t. It can be corrupted by two runs writing at once. The bucket that holds remote state has to defend against all three, and terraform-aws-bootstrap’s state-backend module is built around doing exactly that.

The DynamoDB lock table is gone

Start with the corruption problem, because the answer changed recently.

The long-standing pattern for remote state on AWS was an S3 bucket plus a DynamoDB table. S3 held the state; the DynamoDB table held a lock, so two apply runs couldn’t write at once. Everyone who’s done Terraform on AWS has provisioned that table, probably more times than they’d care to count.

OpenTofu 1.10 made it unnecessary. The S3 backend gained use_lockfile, which does the locking with a small lock object in the same bucket, using S3’s conditional-write support. No separate table. The state backend is now genuinely one bucket and one key, with the lock living beside the state. It’s one fewer resource to create, one fewer thing to pay for, and one fewer moving part to reason about. The module takes the new path, and the DynamoDB table simply isn’t there.

A bucket you can’t delete by accident

Deletion is guarded with lifecycle { prevent_destroy = true } on the bucket. With that set, OpenTofu refuses to produce a plan that would destroy the bucket. A stray tofu destroy, a refactor that drops the resource, an accidental rename: all of them fail loudly instead of quietly taking the state bucket with them.

This is also why the state-backend module is hand-rolled from raw aws_s3_bucket resources rather than wrapping a community module like terraform-aws-modules/s3-bucket. prevent_destroy has to sit on the actual resource, and a lifecycle block isn’t something you can pass into a wrapper module as an input. Hand-rolling the bucket keeps prevent_destroy somewhere you can put it and, just as importantly, somewhere the next reader can see it. (There’s a whole post coming on why I hand-rolled every module; this is one of the reasons in miniature.)

Reject anything encrypted wrong

Confidentiality is the subtle one, because the obvious control isn’t enough.

The bucket has a default encryption configuration: server-side encryption with the customer-managed KMS key. But default encryption is a default. A client making a PutObject call can override it per request, asking for plain AES256 or a different KMS key, and S3 will honour the override.

So the module doesn’t rely on the default. The bucket policy explicitly denies the upload it doesn’t want. It denies any request not over TLS. It denies any PutObject that isn’t using SSE-KMS. And it denies any PutObject that names the wrong KMS key. The default encryption config says “this is what you get if you don’t ask”; the bucket policy says “and you’re not allowed to ask for anything else”. State can only ever land encrypted, in transit and at rest, under the one key the module controls.

One small companion setting: bucket_key_enabled. With per-object SSE-KMS, every object operation is also a KMS API call, which costs money and can throttle. An S3 Bucket Key collapses those into far fewer KMS calls, cutting per-object KMS traffic by well over ninety per cent. It’s a one-line setting the module turns on and most people forget exists.

In short

Remote state is the most sensitive file an infrastructure repo has, and the bucket that holds it has to defend against deletion, disclosure and corruption.

terraform-aws-bootstrap’s state backend handles corruption with OpenTofu 1.10’s use_lockfile, dropping the old DynamoDB lock table entirely. It guards deletion with prevent_destroy, which is also why the bucket is hand-rolled rather than wrapped. And it guards confidentiality with a bucket policy that denies non-TLS traffic and denies any upload not encrypted with the right KMS key, because default encryption is only a default and a client can override it. The state bucket isn’t just a place to put state. It’s built to refuse every wrong thing that could happen to it.

Supporting a provider, or actually using it

Sat, 02 May 2026 00:00:00 +0000

If your CLI tool talks to an AI model, you don’t want to hard-wire one vendor. So you reach for a single client interface over several providers, which is the right call. The trap is the next step: build that interface on only what every provider has in common, and you quietly throw away the very features that made you want a particular provider in the first place. rust-tool-base’s rtb-ai refuses to make that trade.

The pull toward one interface

If your CLI tool talks to an AI model, hard-wiring one vendor is a poor bet. One user has an Anthropic key, another an OpenAI key. Someone’s on Gemini. Someone runs Ollama locally because their data can’t leave the building. Someone points at an OpenAI-compatible endpoint from a provider you’ve never heard of. You don’t want a separate code path for each, so you want one AiClient that all of them slot behind.

rtb-ai gets that unification from the genai crate, which already speaks to Anthropic, OpenAI, Gemini, Ollama and OpenAI-compatible endpoints. One interface, five providers, the tool author picks one in config. The Go sibling makes the same bet: go-tool-base’s chat package also unifies several providers, behind an interface deliberately kept to four methods. So far this is the obvious design, and if it were the whole design there’d be nothing to write about.

What “unified” quietly costs you

Here’s the catch in any unified interface. It can only expose what every provider behind it has in common.

The common subset is plain chat. Messages go in, text comes out, optionally streamed token by token. That’s real and it’s useful and every provider does it. But the common subset is also the floor, and the features that make a particular provider worth choosing are almost never on the floor. They’re the things only that provider does.

Anthropic is the sharp example, because it has three features that matter and not one of them is common-subset.

Prompt caching. You can mark the stable parts of a request, the system prompt and the tool list, as cacheable. The provider keeps them warm, and on the next turn you aren’t billed to re-send and re-process text that didn’t change. On a long agent loop, where the same large system prompt rides along on every single turn, that’s a substantial saving in both cost and latency.

Extended thinking. The model works through a hard problem in a visible, budgeted reasoning pass before it commits to an answer, and you can see that reasoning.

Citations. Structured references back to source material in the response.

A client built strictly on the common subset can’t express any of those. It has no field for them, because four of the five providers wouldn’t know what to do with the field. So a purely lowest-common-denominator client would “support” Anthropic and then use it badly, leaving its best features unreachable. Support as a checkbox, not as the point.

The escape hatch

rtb-ai’s answer is to not choose. It runs two implementations under one interface.

For OpenAI, Gemini, Ollama and OpenAI-compatible endpoints, calls route through genai, the unified path. For Anthropic, every method drops to a direct reqwest implementation straight against the Messages API. Same AiClient on the surface, a different implementation underneath, selected by which provider the config names.

And the request type has deliberate room for the difference:

pub struct ChatRequest {
 pub system: Option<String>,
 pub messages: Vec<Message>,
 pub temperature: Option<f32>,
 pub max_tokens: Option<u32>,
 /// Anthropic-only: enables prompt caching at every stable point.
 /// Ignored on non-Anthropic providers.
 pub cache_control: bool,
 /// Anthropic-only: extended-thinking budget. `None` disables.
 /// Ignored on non-Anthropic providers.
 pub thinking: Option<ThinkingMode>,
}

Set cache_control and the Anthropic-direct path inserts cache breakpoints at the three stable points: the system prompt, the tool list, and the first message. Set thinking and it adds the thinking block, and streaming surfaces a separate ThinkingToken event so you can show the reasoning apart from the answer. On a non-Anthropic provider, both fields are simply ignored. The interface carries them; only the implementation that understands them acts on them.

A hatch, not a leak

It’s worth being precise about why this isn’t the thing it superficially resembles, which is a leaky abstraction.

A leaky abstraction is one where implementation details bleed through that you didn’t intend and can’t reason about. The abstraction quietly fails to abstract, and you’re left guessing which provider you’re really talking to.

This is the opposite of that. The two Anthropic-only fields aren’t a leak. They’re named, documented as Anthropic-only, inert everywhere else, and right there in the public type for anyone to see. The interface is uniform for the common case and deliberately, visibly non-uniform at exactly the points where uniformity would have cost you the good features. You opt into provider-specifics by setting a field. You stay fully portable by leaving it at its default. Nothing bleeds; you decide.

The same design line explains what does stay in the unified path. Structured output, chat_structured::<T>, sends a JSON Schema derived from your Rust type with the request and validates the reply against it before handing you a typed T. That’s a portability win that costs nothing across providers, so it belongs in the common interface. The split isn’t “Anthropic versus the rest”. It’s “features that are free to unify go in the unified path; features that aren’t get a designed door”. Prompt caching and extended thinking get the door, because flattening them away would be the expensive kind of convenient.

To sum up

A CLI tool that integrates AI wants one client over several providers, and a unified interface can only expose what those providers share. The shared floor is plain chat, and the features worth choosing a provider for, like Anthropic’s prompt caching, extended thinking and citations, are never on the floor.

rtb-ai keeps both. genai provides the unified path across five providers; an Anthropic-direct reqwest path drops below the abstraction for the features genai can’t reach, and ChatRequest carries the Anthropic-only fields openly, ignored elsewhere. Uniform where uniformity is free, with a designed escape hatch where it isn’t. That’s the difference between supporting a provider and actually using it.

Errors without an error handler

Fri, 01 May 2026 00:00:00 +0000

In the porting post I said go-tool-base’s error handler was one of the bits that didn’t survive the move to Rust, and promised to come back to it. Here’s the come-back. The short version is that Rust hands you, for free, the single consistent error exit that go-tool-base had to build a whole component to get.

What go-tool-base built

A while ago I wrote about error handling in go-tool-base. The core of it: an error should carry a hint, a separate field of human guidance telling the user what to do next, kept apart from the error’s identity so code can still match on it.

The other half of that post was about consistency. Every go-tool-base command returns its errors the idiomatic Cobra way, and they all funnel into one Execute() wrapper at the root, which routes every error through one ErrorHandler. One door out. Presentation decided in exactly one place, so no command can render a failure differently from its neighbour.

That handler is a real object. It exists, it’s wired in, it’s the thing every error passes through. Building it was a deliberate piece of work, and it was the right call for Go.

When I rebuilt this in Rust, the handler didn’t survive the move. Not because consistency stopped mattering. Because Rust gives you the single exit for free, and an object to enforce it would just be re-implementing something the language already does for you.

The shape of a Rust error

Start with the type. In rust-tool-base every crate defines its own error enum, and every one of them derives two traits:

#[derive(Debug, thiserror::Error, miette::Diagnostic)]
pub enum ConfigError {
 #[error("config file not found at {path}")]
 #[diagnostic(
 code(rtb::config::not_found),
 help("run `mytool init` to create one, or set MYTOOL_CONFIG"),
 )]
 NotFound { path: PathBuf },
 // ...
}

thiserror::Error makes it a proper error type. miette::Diagnostic is the interesting one. A Diagnostic is an error that also carries the things you’d want when presenting it: a stable code, a severity, a help string, and optionally source labels pointing at spans of input. The help line is the same idea as go-tool-base’s hint, the recovery step, except here it’s an attribute on the variant rather than a field threaded through a wrapper.

So the guidance lives on the error, structured, from the moment the error is created.

There is no handler, there’s a convention

Here’s where Rust does the work go-tool-base’s handler was built to do.

A rust-tool-base main looks like this:

#[tokio::main]
async fn main() -> miette::Result<()> {
 rtb::cli::Application::builder()
 .metadata(/* ... */)
 .version(VersionInfo::from_env())
 .build()?
 .run()
 .await
}

main returns miette::Result<()>. Every command’s run returns a Result too. In between, errors propagate with the ? operator: a function that hits an error returns it upward, immediately, and the caller does the same, all the way to main. Nobody writes a “check this error” call. ? is the propagation.

And when an error reaches main and main returns it, something has to render it for the user. That something is a report hook. rust-tool-base installs one at startup, and from then on any Diagnostic that exits main is rendered through it: the code, the severity, the help text, the source labels, with colour. One renderer, installed once.

Look at what that adds up to. Every error in the program flows to one place, main. It’s rendered by one thing, the hook. Presentation is decided in exactly one location and no command can deviate from it. That’s precisely the property go-tool-base’s ErrorHandler was built to guarantee. The difference is that nobody built it. The single exit is just where ? propagation ends, and the single renderer is one hook. The language’s own convention for returning errors from main is the funnel.

Errors are values, all the way

The thing that took me a moment to fully trust is that there’s no funnel to maintain, because there’s no funnel as an object. go-tool-base’s handler is a component: it can drift, it has to be kept in the path, a command could in principle be wired to bypass it. The Rust version cannot be bypassed, because bypassing it would mean a command not returning its error, and an error you don’t return is a compile-time warning at best and dead-obvious wrong code at worst.

So the model is just: errors are values, you return them, ? carries them up, main hands the last one to the hook. The consistency isn’t enforced by a guard. It’s the only thing the shape of the language really lets you do.

go-tool-base reaches a single, consistent error exit by building one and routing everything through it. rust-tool-base reaches the same exit by having errors be ordinary return values and letting them fall out of main. Same outcome. One of them is a component you own; the other is a convention you inherit.

Worth remembering

go-tool-base funnels every error through one ErrorHandler so presentation stays consistent. That handler is a deliberately built component, and it’s the right design in Go.

rust-tool-base has no handler. Every crate’s error type derives miette::Diagnostic, carrying its code, severity and help text. Errors propagate with ? to main, which returns miette::Result, and a framework-installed hook renders whatever comes out. The single consistent exit is the end of ? propagation, and the single renderer is one hook. The funnel go-tool-base built by hand is, in Rust, just the language’s return-from-main convention.

The bootstrap that does almost nothing

Fri, 01 May 2026 00:00:00 +0000

A brand-new AWS account is a slightly nerve-wracking thing. It can do almost anything, it’s hardened against almost nothing, and the list of stuff you ought to set up before you trust it with anything real is long. The natural instinct is to write one big “set up the account” module that does the whole list in a single apply. I want to talk you out of that, because the bootstrap module I’m happiest with does almost nothing, on purpose.

The first-apply problem

A brand-new AWS account is not ready for anything serious. Before you’d responsibly run real infrastructure into it, you want an account baseline: a password policy, account-wide S3 public-access blocking, default EBS encryption, CloudTrail, AWS Config, GuardDuty, alerting, a sensible human operator role. It’s a long list, and all of it matters.

The instinct, faced with that list, is to write one big “set up the account” module and have it do everything. One tofu apply, a fully prepared account, done.

That instinct is worth resisting, and terraform-aws-bootstrap resists it deliberately.

Three things, and a hard line

terraform-aws-bootstrap does three things:

state-backend, an S3 bucket and a customer-managed KMS key to hold remote Terraform state.
automation-iam, an OIDC identity provider and an IAM role that CI assumes to apply everything else.
nuke-config, which renders an aws-nuke configuration scoped to the account, for tearing a throwaway account back down.

That’s the whole module. Account hardening, CloudTrail, AWS Config, GuardDuty, the operator role, the alerting: none of it is in here. And it’s not absent by accident. The README has a section headed “what’s deliberately NOT in scope” that lists those exclusions out loud. The boundary is written down, because the boundary is the design.

Why the line is exactly there

The reason the line sits where it does is the most useful idea in the module.

Everything bootstrap excludes belongs in a separate stack, applied through the automation role bootstrap creates. Bootstrap’s only job is to get the account to the point where the next tofu apply can run properly: somewhere to store state, and an identity to run as. Once those two things exist, hardening the account isn’t a special bootstrapping act. It’s just another apply, done the normal way: in CI, reviewed, versioned, deployed through the role.

So the account baseline doesn’t need to be bundled into the bootstrap. It needs to be downstream of it. Bootstrap builds the on-ramp; it doesn’t also have to be the motorway.

A narrow module stays re-runnable

There’s a practical payoff to the narrowness, and it’s about fear.

Bootstrap is the one stack that can’t be applied through CI, because it’s what creates the CI identity in the first place. It runs locally, by a human, rarely. That’s exactly the kind of operation you want to be small, boring, and safe to repeat.

A bootstrap module that also did account hardening would be a large, stateful thing managing dozens of resources. Re-running it would be a held-breath operation. Keeping it to three concerns keeps it the opposite: a small stack you can read top to bottom, re-run without anxiety, and reason about completely. The narrowness isn’t minimalism for its own sake. It’s what keeps the one human-applied stack trustworthy.

The boundary is the feature

It’s tempting to judge a module by how much it does. A bootstrap module is the case where that’s exactly backwards. Its value is in how cleanly it stops.

terraform-aws-bootstrap does the bare minimum to make an account ready for the next apply, writes down everything it refuses to do, and hands off to a downstream stack for all of it. The next post follows the trickiest of its three jobs: the state backend has a genuine chicken-and-egg problem, because it has to store Terraform state in a bucket Terraform hasn’t created yet.

Where this leaves us

A fresh AWS account needs a long list of things before it’s safe, and the obvious move is one big module that does the lot. terraform-aws-bootstrap deliberately does only three: a state backend, a CI identity, and an account-scrub config. Everything else is written down as out of scope.

The boundary is the design. The excluded work belongs in a downstream stack applied through the CI role bootstrap creates, so hardening is just a normal reviewed apply rather than a bootstrapping special case. And keeping the one human-run, locally-applied stack small is what keeps it safe to re-run. A bootstrap module is judged by where it stops.

Two kinds of feature flag

Thu, 30 Apr 2026 00:00:00 +0000

go-tool-base has feature flags: switches that decide which built-in commands are live in a given run. rust-tool-base has those too. But it also has a second, completely separate kind of flag, and the difference between them is one of those distinctions that’s obvious the moment you see it and dangerously easy to conflate before you do. One decides what a command does. The other decides whether a chunk of code is in the binary at all.

A workspace of crates

Before the flags, the shape that makes them possible. go-tool-base is one Go module with packages under pkg/. rust-tool-base is a Cargo workspace of seventeen crates: rtb-app, rtb-config, rtb-cli, rtb-vcs, rtb-ai, rtb-mcp, rtb-docs, rtb-telemetry, and so on, with an umbrella crate called rtb that re-exports the public surface.

That isn’t tidiness for its own sake. Each subsystem being a separately compilable crate is what gives you a unit you can include or exclude wholesale. Hold onto that, because it’s the hinge for everything below.

The flag go-tool-base already has

go-tool-base has feature flags, and I’d describe them as runtime flags. A tool built on it can enable or disable built-in commands:

props.SetFeatures(
 props.Disable(props.InitCmd),
 props.Enable(props.AiCmd),
)

At startup the framework resolves that set and decides which commands are reachable for this run. The init command might be present in the binary but switched off; the ai command might be switched on. It’s about the user-facing surface: which commands exist for someone typing --help.

rust-tool-base keeps this idea. A command carries a CommandSpec with an optional feature field, and the runtime decides whether a feature-gated command is reachable. Same purpose: shape the surface per invocation.

If that were the whole story, there’d be nothing to write. The reason there’s a post is the other kind of flag, which Rust makes available and Go really doesn’t.

The flag Rust adds

Cargo features are a compile-time mechanism. The rtb umbrella crate declares them like this:

[features]
default = ["cli", "update", "docs", "mcp", "credentials", "tui"]
cli = ["dep:rtb-cli"]
update = ["dep:rtb-update"]
ai = ["dep:rtb-ai", "rtb-docs?/ai"]
vcs = ["dep:rtb-vcs"]
telemetry = ["dep:rtb-telemetry"]
full = ["cli", "update", "docs", "mcp", "ai", "credentials", "tui", "telemetry", "vcs"]

Each subsystem is an optional crate dependency, and a feature switches it on. This is a different kind of switch entirely, and the difference is the whole point.

A runtime flag decides what a command does while the program runs. The code is in the binary either way; the flag just gates it.

A Cargo feature decides what’s in the binary in the first place. Build a tool without the vcs feature and rtb-vcs is not compiled. Its dependencies are not compiled. gix, the pure-Rust Git implementation rtb-vcs pulls in, roughly two and a half megabytes of it, is not compiled and not linked. It isn’t switched off in the binary. It was never in the binary. The compiler never even saw it.

That’s something a runtime flag cannot do, because by the time anything runs, the binary already exists with everything in it.

Two axes, kept separate

So rust-tool-base has two flag systems answering two genuinely different questions.

Cargo features answer: what is this binary made of? They’re decided when you build the tool, in Cargo.toml. They control compilation, binary size, dependency surface, and compile time. A tool that never touches Git builds without vcs and is smaller, faster to compile, and has a smaller dependency tree to audit. A tool that wants everything turns on full.

Runtime feature flags answer: what can the user do in this run? They’re decided as the program starts. They control which commands appear, which paths are reachable.

These could have been mashed into one mechanism, and it would have been a mistake. The app-context design notes are blunt about it: feature gating doesn’t belong on the per-command context object, because a feature-gated command “either exists or doesn’t” rather than changing its behaviour mid-run. Compile-time composition is one decision, made by the person building the tool. Runtime gating is another, made per invocation. Conflating them would mean you couldn’t reason cleanly about either.

The Go version of this had to be hand-built

This isn’t a thing Go simply lacks. I wrote a whole post about how go-tool-base keeps its optional keychain dependency out of binaries that don’t want it, using a blank import and the linker’s dead-code elimination. It works. But it was a piece of deliberate engineering for one dependency, and getting it right took care.

Cargo features make that same outcome a first-class, declarative thing, and not for one dependency but for every subsystem the framework has. You don’t engineer the exclusion. You name a feature and leave it off. The crate, and its whole subtree, stays out. Rust’s build system was designed for exactly this, and rust-tool-base leans on it across the entire workspace rather than hand-rolling it once.

Where this leaves us

go-tool-base has runtime feature flags: they decide, per invocation, which built-in commands are reachable. rust-tool-base keeps that, and adds a second kind that Rust makes available.

Cargo features decide what the binary is compiled from. Each of the framework’s seventeen crates is an optional dependency, and a feature switched off means that crate and its entire dependency subtree are never compiled or linked. A runtime flag gates what code does; a Cargo feature gates whether code is there at all. Two axes, two questions, deliberately kept as separate systems.

forbid means forbid, until linkme needs a word

Wed, 29 Apr 2026 00:00:00 +0000

There’s a line at the top of every production crate in rust-tool-base that I’m quietly proud of: #![forbid(unsafe_code)]. And there are a couple of files that have to say #![allow(unsafe_code)] instead. Not because I wrote anything unsafe. Because a macro did, on my behalf, and forbid doesn’t care whose unsafe it is.

Why forbid, and why it isn’t the whole story

rust-tool-base makes a bold promise: no unsafe in its own code. The strong form of that is forbid, not deny. deny(unsafe_code) makes unsafe a compile error that any module can quietly re-permit with its own #[allow]. forbid can’t be overridden from inside the crate at all. That’s the appeal: nobody gets to wave unsafe through in a hurry.

So the workspace lint sits at deny, and every production lib.rs then tightens it to forbid:

# Cargo.toml
[workspace.lints.rust]
unsafe_code = "deny"

// crates/rtb-error/src/lib.rs
#![forbid(unsafe_code)]

Why deny at the workspace but forbid in each crate? Because deny leaves an escape hatch open for the rare file that genuinely needs one, while forbid slams it shut everywhere it can. Almost every file gets forbid. A tiny number need the hatch.

The files that need the hatch

The command and provider registries use linkme’s distributed_slice so backends can register themselves at link time, without life before main. And the linkme attribute expands to code carrying a #[link_section], which the unsafe_code lint counts as unsafe. So any file using the attribute, whether it declares a slice or registers into one, can’t live under forbid.

Here’s the Gitea release backend doing exactly that, from crates/rtb-vcs/src/gitea.rs:

#![allow(unsafe_code)]
// ...
/// Link-time registration entry.
#[distributed_slice(RELEASE_PROVIDERS)]
fn __register_gitea() -> Box<dyn ProviderRegistration> {
 Box::new(RegisteredProvider { source_type: "gitea", factory: factory as ProviderFactory })
}

That #![allow(unsafe_code)] isn’t there because the backend does anything dangerous. It’s there because the registration macro emits a #[link_section], and forbid would, correctly by its own rules, refuse to compile the file.

Where that leaves the promise

The guarantee survives, with an exception you can point at. Every production crate forbids unsafe outright. The workspace sits one notch looser at deny, precisely so the handful of files that use linkme (and a couple of test files that need Rust 2024’s unsafe env mutation) can open a narrow, module-scoped #![allow(unsafe_code)] with a written reason. The absolutist rule met a macro that writes a link_section for you. The answer wasn’t to drop the rule, it was to keep forbid everywhere it can hold and clearly label the one or two spots where it can’t.

A framework that contains no unsafe

Tue, 28 Apr 2026 00:00:00 +0000

“It’s written in Rust” gets thrown around as if it were a memory-safety guarantee. It mostly isn’t. Rust is memory-safe by default, which is a wonderful thing, but the unsafe keyword exists precisely so any crate, any module, can step outside that default when it needs to. So “written in Rust” really means “mostly safe, probably”. rust-tool-base makes the stronger claim about its own code, and gets the compiler to enforce it.

Safe by default is not the same as safe

People reach for Rust because of memory safety, and the reputation is earned. Write ordinary Rust and the compiler will not let you have a use-after-free, a data race, or a buffer overrun. That’s the default, and it’s a very good default.

But it’s a default, and defaults can be turned off. Rust has an unsafe keyword precisely so that, when you genuinely need to, you can dereference a raw pointer, call into C, or tell the compiler you’ve upheld an invariant it can’t check itself. Inside an unsafe block, the guarantees are yours to maintain, not the compiler’s to enforce.

That keyword has to exist. Some of the most foundational crates in the ecosystem are built on it, carefully. But it means a fact worth being precise about: a project being “written in Rust” tells you its code is mostly safe. It does not tell you the project’s own code contains no unsafe. Those are different claims, and only the second one is a guarantee.

rust-tool-base makes the second claim about its own code, and has the compiler back it up.

`forbid`, not just `deny`

The mechanism is one line at the top of every crate:

#![forbid(unsafe_code)]

unsafe_code is a lint, and Rust lints have levels. The interesting choice is forbid rather than deny, because the two are not the same strength.

deny makes the lint an error. But it’s an error a downstream module can locally override. Anyone can write #[allow(unsafe_code)] on a function or a block and the deny is lifted right there. As a policy, deny is “don’t do this unless you really mean to”, and “unless you really mean to” is a door.

forbid is the strict one. It makes the lint an error and it makes that error impossible to override from inside the crate. A module cannot #[allow] its way back out. Once a crate root says #![forbid(unsafe_code)], there’s no unsafe anywhere in that crate, and no local exception can be carved out. The compiler simply refuses.

So every rust-tool-base crate that ships in a built tool forbids unsafe at its root. Not “discourages”. Cannot contain it.

The one subtlety

There’s a wrinkle, and it’s worth showing rather than hiding, because it’s where the design got specific.

The workspace sets unsafe_code = "deny" as the baseline for everything, including test files. But test code occasionally has a real need for unsafe. In the 2024 edition, std::env::set_var became unsafe, because mutating the process environment isn’t thread-safe, and a test that exercises environment-driven configuration has to call it.

So the split is deliberate. The workspace-wide level is deny, which a test file can locally #[allow] when it genuinely needs that one environment call. But every production lib.rs and main.rs additionally carries #![forbid(unsafe_code)], and forbid cannot be relaxed. Test scaffolding gets a controlled, visible exception for a specific standard-library call. Shipping code gets none. The guarantee that matters, “the code in the binary contains no unsafe”, holds, and the place it’s slightly loosened is exactly the place that never reaches a user.

What the guarantee is actually worth

Two things, one for users and one for reviewers.

For users: an entire family of bug is ruled out of first-party code mechanically. Use-after-free, double-free, data races on shared memory, reading off the end of a buffer. These are the classic memory-safety vulnerabilities, and in a crate that forbids unsafe they cannot originate, because the constructs that produce them cannot be written. That’s not careful coding. It’s the compiler refusing to build anything else.

For reviewers: the cost of an unsafe block is mostly the review burden it carries. Every one is a spot where a human has to check, by hand, that an invariant holds, and has to re-check it whenever nearby code changes. A crate that forbids unsafe has zero of those. There’s no unsafe block to audit, ever, because the compiler guarantees there isn’t one.

The promise has a boundary. It covers rust-tool-base’s own code; its dependencies are another matter, and some of them do contain unsafe, correctly. Keeping that side honest is a different job, done by vetting the dependency tree and gating it in CI. Within first-party code, though, the guarantee is real, and there’s no Go equivalent to it. Go has an unsafe package, but nothing that lets a codebase prove, to the compiler, that it never touches it.

The bottom line

Rust is memory-safe by default, but the unsafe keyword exists so that default can be set aside. “Written in Rust” therefore does not by itself mean a project’s own code contains no unsafe.

rust-tool-base makes that the stronger claim. Every crate root carries #![forbid(unsafe_code)], and forbid, unlike deny, cannot be overridden from inside the crate. Test files get a narrow, visible deny-level exception for the one standard-library call that needs it; shipping code gets none. The payoff is a whole class of memory-safety bug ruled out of first-party code by construction, and not one unsafe block left for a reviewer to audit.

Reloading config without a restart

Mon, 27 Apr 2026 00:00:00 +0000

A config file changes. Someone edits a setting, rotates a credential, flips a feature flag. How does the running process find out? For most processes the answer is blunt: it doesn’t, until you restart it. For a short-lived CLI that’s completely fine. For a long-running service, “just restart it” is a much bigger ask than it sounds.

The default answer is a restart

Configuration lives in a file. The file changes: someone edits a setting, rotates a credential, flips a feature flag. How does the running process find out?

Overwhelmingly, the answer is that it doesn’t. A process reads its config once, at startup, and that snapshot is frozen for the life of the process. Change the file and nothing happens until you restart, at which point a fresh process reads the fresh file.

For a short-lived CLI invocation that’s completely fine. It reads config, does its job, exits, and the next invocation reads whatever the file says then. But the same frameworks are also used to build long-running services, and for a service “just restart it” is not the small thing it sounds like.

What a restart actually costs

Restarting a long-running service means every open connection drops. Any in-flight request is lost, or has to be retried by whoever sent it. Caches that took real time to warm are cold again. There’s a window, short but real, where the service simply isn’t serving.

If the thing you changed was a log level, or a feature flag, or a timeout, you’ve paid a disruption wildly out of proportion to the change. And the calculation only gets worse as the service gets more important, because the services you least want to bounce on a whim are exactly the ones that matter most.

Hot-reload: re-read in place

Hot-reload is the alternative, and both go-tool-base and rust-tool-base support it.

The process doesn’t read config once and freeze it. It watches the config file. When the file changes, it re-reads it, re-applies it, and carries on running. No new process, no dropped connections, no cold start. The change lands in the live process.

The shape is the same in both frameworks:

A file watcher notices the config file changed. Underneath, this is the operating system’s own file-notification facility, inotify on Linux and its equivalents elsewhere. rust-tool-base reaches it through the notify crate; go-tool-base, through the watcher built into Viper.
A debounce step waits for the writes to settle. Saving a file is often several separate operations, and you don’t want to reload three times for one edit.
The config is re-parsed from disk.
The new config is swapped in atomically.
Observers are notified, so the subsystems that care can react.

Steps four and five are the ones worth slowing down on, because they’re where a naive hot-reload quietly goes wrong.

The two details that make it safe

The atomic swap. You do not mutate the live config object in place. A reader on another thread, partway through reading it, would see a torn mix of old and new values, and that’s a genuinely nasty class of bug. Instead the process builds a new, complete config value and swaps the pointer to it in a single atomic operation. Any reader sees either the entire old config or the entire new one, never a blend. rust-tool-base does this with arc-swap; go-tool-base does the equivalent. Reads stay cheap and lock-free, and an update is one pointer swap.

The observer notification. Re-reading the file isn’t the end of the job. Some subsystems have to do something when config changes: a connection pool resizes, a logger changes level, a rate limiter takes a new ceiling. So a hot-reload system has to let those subsystems subscribe. rust-tool-base hands observers a watch::Receiver, a channel that always holds the latest value; go-tool-base exposes an Observable interface. A subsystem subscribes once and reacts every time config changes, for the life of the process.

Where this earns its keep: a Kubernetes pod

Hot-reload is a nicety on a developer’s laptop. Inside a Kubernetes pod it becomes genuinely valuable, and the reason is a neat fit between how Kubernetes delivers config and how a file watcher works.

In Kubernetes you don’t usually bake configuration into the container image. It lives in ConfigMap and Secret objects, and the clean way to consume them is to mount them as volumes. Mount a ConfigMap as a volume and each key becomes a file in the pod’s filesystem.

Here’s the part that connects to everything above. When you update that ConfigMap or Secret, Kubernetes does not restart your pod. The kubelet notices the object changed and rewrites the projected files inside the still-running pod. The files on disk change underneath a process that never stopped.

That file rewrite is exactly the event a hot-reload watcher exists to catch. So the whole chain becomes:

You kubectl apply an updated ConfigMap, or rotate a Secret.
The kubelet updates the projected files inside the pod.
The framework’s file watcher sees the write.
The config is re-parsed, swapped in atomically, and observers are notified.
The new configuration is live, and the pod never cycled.

You’ve changed a running service, in a running pod, with no rollout, nothing terminated and recreated, no dropped traffic. Rotate a database credential, raise a log level to debug an incident in progress, flip a feature flag: all of it live. For a service where a restart is the very thing you’re trying hard to avoid, the kind of long-running service these frameworks are built for, that’s the difference between a config change being routine and being an event.

The caveats

Two things, so this doesn’t read as magic.

First, not everything can be hot-reloaded. Some configuration genuinely needs a restart: the port a server binds to, the size of a thread pool, anything wired up exactly once at process start. Hot-reload covers the large category of settings a subsystem can re-read and re-apply; it doesn’t abolish restarts. A config system worth its salt is clear about which settings are live and which are not.

Second, a Kubernetes gotcha that catches people out. The in-place file update happens for ConfigMaps and Secrets mounted as volumes. Consume the same ConfigMap as environment variables instead, and those are fixed when the container starts and never update, short of a restart. If you want hot-reload in a pod, mount config and secrets as files, not env vars. And even with volumes the update isn’t instant: the kubelet syncs on a period, around a minute by default, so a reload is “within a minute or so”, not “the moment you hit apply”.

What it comes down to

A config file changes, and the default way to pick it up is to restart the process. For a long-running service that restart costs dropped connections, lost work and a cold start, often for a change as small as a log level.

go-tool-base and rust-tool-base both support hot-reload instead: a file watcher catches the change, the config is re-parsed and swapped in atomically so no reader sees torn state, and observers are notified so subsystems can react, all in a live process. The setting where it pays off most is a Kubernetes pod, where ConfigMaps and Secrets mounted as volumes are rewritten in place by the kubelet and the watcher catches that write directly. Mount them as volumes rather than env vars, allow for the kubelet’s sync delay, accept that some settings still need a restart, and within those limits “the config changed” stops meaning “cycle the pod”.

A signing key needs somewhere to live

Sun, 26 Apr 2026 00:00:00 +0000

I left a door open a couple of posts ago, and it’s been quietly bothering me ever since. When I wrote about verifying your own downloads, I was honest that a checksum sitting next to the binary only catches accidents. Anyone who can compromise the release platform can swap the binary and the checksum together, and the tool will happily verify one fake against the other.

Closing that gap needs a signature. And a signature, it turns out, needs a surprising amount of infrastructure standing behind it. This is the first post about building that.

The door the last post left open

A while back I wrote about verifying your own downloads: go-tool-base’s self-update command now checks the SHA-256 of every binary it downloads against the release’s published checksums.txt before installing it.

That post was honest about its own ceiling. A checksum file hosted next to the binary it describes shares a trust root with that binary. Both come from the same release, on the same platform. Corruption, truncation, a CDN serving a stale object: a same-origin checksum catches all of those, because they’re accidents and the checksum wasn’t part of the accident. What it can’t catch is an attacker who’s compromised the release platform itself. Someone who can replace the binary can replace checksums.txt in the same breath, and the tool will cheerfully verify the malicious download against the malicious checksum and call it good.

The post named the fix and then deferred it: a signature whose trust root sits somewhere the release platform can’t reach. “That’s the next phase of this work.” This series is that phase.

What a signature actually needs

It’s worth being precise about why a signature helps where a checksum doesn’t, because it’s easy to wave the word “signature” around and assume it settles everything.

A signature closes the gap only under two conditions. The verifying key, the public half, must reach the user by a path the release platform doesn’t control. And the signing key, the private half, must live somewhere the release platform can’t reach.

The second condition is the one people skip. If the signing key sits in the same CI system that builds the release, you’ve gained almost nothing. An attacker who owns the CI owns the key, and a key they own will sign whatever they hand it. The signature verifies perfectly and means precisely nothing. A signature is only worth the distance between the signing key and the thing being signed. Put them in the same place and the distance is zero.

So the signing key has to live in a different security domain from the release pipeline. Not a different folder. A different account, with a different blast radius, that the release platform has no standing access to.

“Just sign the binary” is not a small feature

That reframes a line item that sounds tiny. “Sign the release binary” unpacks into a list:

there must be a private signing key;
it must live outside the release platform, in its own security domain;
it must be access-controlled, audited, and protected from exfiltration;
only the release pipeline may ask it to sign, and only by proving a short-lived, federated identity, never by holding a copy of the key.

That’s not a feature you bolt onto a CLI. That’s infrastructure.

The shape of it: a cloud account, with the key held in a managed key service so the private key material never exists as a file on a disk that anyone, me included, can copy. The release pipeline authenticates to that account as itself, briefly, and asks the key service to produce a signature. The key never moves.

But an account you’re going to trust with a signing key is itself something you have to get right first. An account with a weak baseline, no audit trail, and long-lived credentials lying around is not a safe home for the most security-sensitive key in the whole system. Before the key can move in, the house has to be built and the locks have to actually work.

What this series builds

So this turned into a rather longer project than “add a signature”, and the series follows it in order.

It starts with bootstrapping a fresh AWS account: the deliberately minimal first tofu apply, and the remote state backend that has a genuine chicken-and-egg problem. Then the credential question, which is the heart of it: how a CI pipeline deploys to AWS with no stored access key at all. Then hardening the account, so it’s genuinely safe to hold something valuable. Then the discipline of deploying changes to it: plans reviewed before they’re applied. Then the shared tooling that makes all of it repeatable.

Every one of those pieces exists for the same reason. The signing key needs somewhere to live, and somewhere safe is not a default you’re handed. It’s a thing you build, deliberately, before you have anything worth protecting in it.

The series ends where the verifying-downloads post pointed: a signing service whose key the release platform can’t touch, so a self-updating tool can finally verify that the binary it’s about to become is genuinely the one I published.

The upshot

go-tool-base’s self-update verifies downloads against a checksum, and a same-origin checksum stops accidents but not a compromise of the release platform. The fix is a signature, and a signature is only worth the distance between its signing key and the release pipeline.

Holding that key safely means a private key that never leaves a managed key service, in a separate cloud account, reached only by a short-lived federated identity. That’s infrastructure, and a safe account is something you build before you trust it with anything. The rest of this series builds it, piece by piece, right up to the signing service itself.

Waivers with an expiry date

Sun, 26 Apr 2026 00:00:00 +0000

A vulnerability scanner gives you a yes or a no. Is there a known advisory on a path you actually use? Yes, or no. That’s genuinely useful, and you should run one. But it’s a snapshot, taken on the day you ask, and supply-chain risk in a framework is a bigger and more ongoing thing than a single yes-or-no can capture.

So rust-tool-base treats its whole dependency tree as something to have a policy about, not something to scan and forget.

A scanner answers one question

When I had go-tool-base security-audited, part of the routine was running a vulnerability scanner over the dependencies. Go has a good one. It looks at your dependency graph, cross-references known advisories, and tells you whether any of them reach code you actually call.

That’s useful and you should do it. But notice the shape of what it gives back: essentially a yes or a no. Either there’s a known vulnerability on a reachable path or there isn’t. It answers one question, on the day you ask it.

Supply-chain risk in a framework is broader than that one question, because a framework drags its entire dependency tree into every tool built on it. rust-tool-base treats the whole tree as something to have a policy about, and the tool for that is cargo-deny.

A gate, not a scan

cargo-deny reads a deny.toml and checks the dependency graph against four kinds of rule.

Licences. There’s an allowlist: MIT, Apache-2.0, the BSD variants, ISC, a handful of others. Every transitive crate’s licence has to be on it. A dependency that pulls in something copyleft, or something with no licence at all, fails the build. You find out the first time it enters the tree, not during a release scramble when someone finally reads the legal implications.

Advisories. It checks the RustSec advisory database, and yanked crates are set to deny, so a dependency that’s been pulled from the registry stops CI.

Bans. Wildcard version requirements (version = "*") are denied outright, because a dependency that floats to whatever’s newest is a supply-chain hole by construction. Duplicate versions of the same crate get surfaced too.

Sources. Crates may only come from the official registry. An unknown registry or a stray git dependency is denied. Nothing sneaks in from a URL.

That’s a gate. It encodes, as rules in a file, what the project will and won’t accept into its dependency tree, and it enforces them on every build instead of once an audit.

The honest part is the waiver list

Here’s the thing every real project runs into. Sooner or later there’s an advisory you genuinely can’t fix this week. It’s against a crate three levels down your tree. The fix needs an upstream release that hasn’t happened. The crate is scheduled to be reworked two milestones from now anyway. The gate is going to fail, and the work to satisfy it honestly isn’t available to you yet.

The lazy response is a blanket ignore: silence the advisory, move on, forget. Now your gate has a hole in it that nobody remembers opening.

rust-tool-base’s deny.toml does something better. Every waiver in the ignore list is a documented record. Each one carries a comment that names the crate, traces the exact dependency path that reaches it, gives the reason, and names the condition that lifts it:

ignore = [
 # `instant` - reached via async-openai -> backoff -> rtb-ai (v0.3).
 "RUSTSEC-2024-0384",
 # `paste` - reached via ratatui -> rtb-docs (v0.2) / rtb-tui (v0.4).
 "RUSTSEC-2024-0436",
 # ...
]

The file states the policy out loud: “Every waiver points at a deferred stub crate that will be reworked before its ship milestone. Lift each waiver when the owning crate lands its v0.1.”

Some waivers go further and carry a structured reason field, so the why travels with the entry rather than living only in a comment above it:

{ id = "RUSTSEC-2025-0140",
 reason = "gix-date via gix is a stub dependency; rtb-vcs v0.5 will upgrade" },

Read that list and you don’t see a project that quietly stopped caring about seven advisories. You see seven advisories the project knows about, can trace, and has tied to a specific milestone. The waiver has an expiry condition. When rtb-vcs reaches v0.5, that gix entry is meant to come out, and the comment is the reminder that it should.

Why this is the bit to copy

A gate that can’t be relaxed is a gate people route around. They’ll find the broadest possible ignore and use it, because the alternative is being blocked on someone else’s release. The pressure to do that is real, and it’s not unreasonable.

So the design that actually holds up isn’t a stricter gate. It’s a gate with an honest, structured escape hatch: you can waive an advisory, but a waiver costs you a documented record with a dependency path and an expiry condition. That price is small enough that nobody routes around it, and high enough that waivers don’t accumulate silently. The ignore list stays readable, and every line in it is something you could defend out loud.

Supply-chain hygiene framed this way isn’t an audit you survive once a year. It’s bookkeeping: a ledger of what you accepted, why, and when each exception is due to close. Which, now I write it down, is just the Boy Scout rule again, pointed at a dependency tree. Leave it tidier than you found it, and write down the bits you couldn’t tidy yet.

Where this leaves us

A vulnerability scanner answers one question on one day. cargo-deny is a standing policy gate: licences against an allowlist, advisories and yanked crates denied, wildcard versions banned, sources restricted to the official registry, enforced on every build.

The part of rust-tool-base’s setup worth copying is the waiver list. Every advisory that can’t be fixed yet is recorded with its crate, its dependency path, its reason and the milestone that removes it. A waiver is a dated note, not a shrug, and that’s what keeps the gate honest enough that nobody actually wants to bypass it.

A builder that won't compile if you forget a field

Sat, 25 Apr 2026 00:00:00 +0000

go-tool-base configures things with functional options, and if you forget a required one, the best case is a runtime failure and the worst case is an empty value sailing silently into everything downstream. Most builder patterns share the same hole. rust-tool-base closes it in a way I find genuinely delightful: the .build() method simply doesn’t exist until you’ve set every required field.

When is a required field actually required

Every framework has constructors with a mix of required and optional inputs. An Application in rust-tool-base needs tool metadata and a version. It optionally takes a custom config type, extra commands, feature toggles. The metadata needs a name and a summary; a description and a help channel are optional.

The interesting question is when “required” gets enforced. There are really only two moments available: when the program runs, or when it compiles. Most APIs pick the first without ever framing it as a choice.

How go-tool-base does it

go-tool-base uses functional options, the standard Go pattern:

tool := props.New(
 props.WithName("mytool"),
 props.WithVersion(version),
)

New takes a variadic list of options and applies them. It’s flexible and it reads well. But look at what the type actually says. New accepts zero or more options. The signature is satisfied by passing nothing at all. If WithName is required, nothing in the type system knows that. Forget it and the code compiles cleanly, and you find out when the program runs, or worse, when it doesn’t visibly fail but quietly carries an empty name into everything downstream.

A plain builder is no better here. builder.name("mytool").build() and builder.build() are both perfectly valid calls as far as the compiler is concerned. The builder hopes you set the name. It can check at the end and return an error, but that check still happens at runtime.

In every one of these the required-ness of a field is a fact that lives in documentation and in the author’s head, not in the code.

Typestate: putting “required” in the type

rust-tool-base builds these with bon, and the pattern it generates is a typestate builder. The idea is that the builder’s type changes as you call it, and that type tracks which required fields you’ve set so far.

let metadata = ToolMetadata::builder()
 .name("mytool")
 .summary("my CLI tool")
 .build();

ToolMetadata::builder() returns a builder in a state that records “name not set, summary not set”. Calling .name(...) consumes that builder and returns a different type, one whose state records “name set”. Calling .summary(...) does the same for the summary.

The part that matters is .build(). It isn’t a method on the builder in general. It only exists on the builder type that represents “every required field has been set”. So this:

let metadata = ToolMetadata::builder()
 .summary("my CLI tool")
 .build();

doesn’t compile. Not because a runtime check fired, but because in the state “name not set” there’s no .build() method to call in the first place. The compiler stops you, and the error points straight at the missing .name(...).

Optional fields stay optional. You can call .description(...) or skip it, and .build() is reachable either way, because the description was never part of the state that gates it. The required and the optional are genuinely different in the type, which is exactly the distinction the functional-options version could only keep in a comment.

Application::builder() works the same way. It won’t produce an Application until it has metadata and a version, and “won’t” there means the method is absent, not that a check returns Err.

Why the moment matters

Moving the check from run time to compile time changes who finds the mistake, and when.

A runtime check finds it when that code path executes, which might be in a test, might be in CI, might be on a user’s machine at the worst possible moment. A compile-time check finds it the instant you write it, in the editor, before anything has run at all. The same mistake, caught at the cheapest possible point instead of one of the expensive ones.

It also changes what the API documents about itself. A functional-options constructor can’t tell you, from its signature alone, which options you must pass. A typestate builder can, because the set of methods available to you at each step is the documentation. You literally cannot reach .build() without having been walked past every required field on the way.

This is one of those places where Rust’s type system earns its reputation. The builder isn’t more careful than the Go version. It’s that “this field is required” stopped being a convention and became something the compiler enforces. (Another entry, if you’re keeping score from the porting post, in the column of outcomes that survived while the Go mechanism got left behind.)

The short version

Required fields have to be enforced somewhere. Functional options and ordinary builders enforce them at runtime, if at all, because .build() is always callable and the type system never learns which inputs were mandatory.

rust-tool-base uses typestate builders generated by bon. The builder’s type changes as you set fields, and .build() only exists once every required field is present. Forgetting one is a compile error that names the missing call, not a runtime surprise. The required-versus-optional distinction stops being a comment and becomes part of the type.

Process isolation won't save you from the filesystem

Sat, 25 Apr 2026 00:00:00 +0000

A test that passed every single time I ran it on its own, and failed maybe one run in five when I ran the whole suite. The failure was always the same: the self-update test downloaded a release archive, went to extract it, and found the archive corrupt. Half-written. As if something had been scribbling in the file while it read it. Something had.

The comfort I was leaning on

The self-update tests are heavier than a unit test wants to be. They stand up a fake release, download the artefact, verify its checksum, extract it, swap a binary. Real files, real I/O. So they’d been built to run as separate processes, not just separate threads, each one its own little world.

And I’d quietly filed that under “solved”. Separate processes don’t share an address space. One can’t reach into another’s memory and corrupt a value mid-read. That whole category of data race, the kind you reach for a mutex to fix, simply can’t happen across a process boundary. So I’d stopped thinking about concurrency in these tests at all, because I’d convinced myself the isolation was total.

It wasn’t total. It was isolation of memory, and I’d let myself hear it as isolation of everything.

Two processes, one path

The thing two processes very much do still share is the filesystem. And the self-update flow, sensibly, caches its download rather than re-fetching it. The default cache directory is computed from the tool’s name and the release version, in crates/rtb-update/src/flow.rs:

pub fn cache_dir_for(tool_name: &str, version: &str) -> PathBuf {
 let base = directories::ProjectDirs::from("", "", tool_name)
 .map_or_else(std::env::temp_dir, |p| p.cache_dir().to_path_buf());
 base.join("update").join(version)
}

Read that with two parallel test processes in mind. They’re testing the same tool, against the same fake release tag. So tool_name matches and version matches, which means cache_dir_for hands both of them the identical path. Two processes, isolated in every way that involves memory, both downloading and extracting into one shared directory on disk, at the same time. One writes the archive while the other is partway through reading it, and you get exactly the corrupt half-written file the test kept tripping over.

Process isolation did nothing here, because the contention was never in memory. It was on a path string that came out the same for both of them.

Once it’s framed as “they share a path”, the fix writes itself: don’t share the path. Give each invocation its own cache directory. The updater builder already had the seam for it, and the doc comment now says exactly why it’s there, in crates/rtb-update/src/updater.rs:

/// Tools call this when they want isolation per-invocation
/// (e.g. CI runners, tests with parallel processes) or to honour
/// a user-supplied `--cache-dir` flag.
pub fn cache_dir(mut self, cache_dir: impl Into<PathBuf>) -> Self {
 self.cache_dir = Some(cache_dir.into());
 self
}

Each test now builds its updater with cache_dir(its_own_tempdir), so two parallel processes land on two different directories and never meet. No lock, no serialisation, no clever cross-process file mutex. Just the realisation that the shared thing was a directory, and the cure for shared mutable state is usually to stop sharing it, not to guard it.

The fix that turned out to be a feature

The part I’m quietly pleased about is that this didn’t stay a test-only hack. The override I needed to isolate the tests is exactly the override a real tool wants for its own reasons. A CI runner doing self-update wants a writable cache path it controls, not wherever directories-rs decides the system cache lives. A user might reasonably want to point the whole thing somewhere specific. That’s a --cache-dir flag, and cache_dir() is precisely the hook you’d wire it to.

So the thing I added to stop a flaky test is the same thing a downstream tool reaches for to expose --cache-dir. The test forced the seam to exist, and the seam was worth having anyway. I’ll take that trade every time over a fix that only the test suite benefits from.

What it comes down to

I’d treated “separate processes” as a synonym for “can’t race”, and it isn’t. Processes don’t share memory, so the memory races are gone. They absolutely still share the filesystem, the network, every named resource the OS will hand to anyone who asks for it by the same name. My two test processes computed the same cache path from the same tool and tag, and raced on the files in it, and no amount of address-space isolation was ever going to touch that.

Shared mutable state on disk is still shared mutable state. The fix wasn’t a bigger hammer, it was giving each process its own directory and letting the isolation I thought I already had actually be true.

Registering commands without life before main

Fri, 24 Apr 2026 00:00:00 +0000

I ended the last post promising to show how a Rust command registers itself when the language flatly refuses to run any of your code before main(). This is that post, and it’s a lovely example of reaching the same outcome by a completely different road.

The outcome I wanted to keep is self-registration.

What self-registration buys

A command in go-tool-base lives in its own file, and that file puts the command into the framework itself. There’s no central list of commands to keep in sync. You add a file, the command appears. You delete the file, it’s gone. Nothing else changes.

That property is worth protecting. The alternative, a hand-maintained registry that every new command has to be threaded into, is exactly the sort of central file that turns into a merge-conflict magnet and quietly falls out of date. So when go-tool-base moved to Rust, self-registration was firmly in the column of things that had to survive.

The way Go did it was not.

How Go does it

A Go package can declare an init() function, and the runtime guarantees every init() runs before main() starts. A go-tool-base command file uses this to append itself to a package-level slice:

func init() {
 registry.Register(&DeployCommand{})
}

By the time main() runs, every command file’s init() has already fired and the registry slice is populated. It’s a tidy trick, and it leans entirely on a Go feature: code that executes before main().

Rust doesn’t have that

Rust has no init(). There’s no language-blessed phase that runs your code before main(). This is a deliberate decision, not an oversight. Code running before main() across many files has no well-defined order, and a startup phase whose ordering you can’t see is a classic source of subtle, miserable bugs. Rust closed that door on purpose.

Which leaves a real question. If nothing runs before main(), how does a command file insert itself into a registry without a central list editing it in?

Distributed slices

The answer is a crate called linkme, and the mechanism is the linker rather than a runtime phase.

You declare a slice the framework will collect into:

#[distributed_slice]
pub static BUILTIN_COMMANDS: [fn() -> Box<dyn Command>];

(Box<dyn Command> is just “a pointer to some value that implements the Command trait, whichever concrete type it turns out to be”; the primer covers it if that’s unfamiliar.)

A command file then contributes one entry to it:

struct Greet;

impl Command for Greet { /* ... */ }

#[distributed_slice(BUILTIN_COMMANDS)]
fn register_greet() -> Box<dyn Command> {
 Box::new(Greet)
}

Here’s the part that makes it work. The #[distributed_slice] attribute doesn’t generate any code that runs at startup. It places each entry into a dedicated section of the compiled object file. When the linker builds the final binary, it gathers everything in that section and lays it out as one contiguous array. BUILTIN_COMMANDS is that array.

So by the time the program exists as a binary on disk, the registry is already assembled. main() doesn’t build it. No init() builds it. The linker built it, statically, as part of producing the executable. At runtime the framework iterates a slice that was complete before the process ever started.

What you get from it

The outcome is the one Go’s init() gave, and then a bit more.

A command still lives in one file and still self-registers. Adding a command is still adding a file. There’s still no central list.

But there’s no startup phase to reason about, because there isn’t one. There’s no global mutable slice being appended to as init()s fire, because nothing is appended at runtime; the slice is immutable and finished. There’s no ordering question, because the linker isn’t running your code, it’s collecting data. And it costs nothing at runtime: assembling the registry happened at link time, so program start just reads it.

It’s the same idea go-tool-base had, expressed by the tool Rust actually gives you. Go reaches the registry through a controlled phase before main(). Rust reaches it without any phase at all, because the linker did the assembly while the binary was still being built. Two roads, one destination… which, if you’ve been following along, is becoming the whole theme of the Rust side of this project.

In short

Self-registration, where a command file inserts itself into the framework with no central list, is a property worth keeping. go-tool-base achieves it with a package-level init(), leaning on Go’s guarantee that such functions run before main().

Rust has no equivalent and wants none, because code running before main() has no clear ordering. rust-tool-base uses linkme distributed slices instead: each command is placed into a dedicated linker section, and the linker assembles them into one contiguous, immutable slice as it builds the binary. The registry is complete before the program runs. Same outcome as Go’s init(), with no life before main required.

Verifying your own downloads: how I solved it for self-updating CLI tools

Fri, 24 Apr 2026 00:00:00 +0000

Way back in the introduction I promised I’d come back to the self-update integrity checks. Here we are. And the starting point is a slightly uncomfortable admission: for a good long while, go-tool-base’s update command was the most trusting line of code in the entire tool.

The most trusting line of code in the tool

Self-update is a lovely feature. The user runs yourtool update, the tool fetches the latest release, swaps itself out, and they’re current. go-tool-base has had this since early on, wired to GitHub, GitLab, Bitbucket, Gitea and a few others.

But look closely at what that feature actually does. It reaches out to the internet, pulls down a file, and then replaces the executable that’s currently running with that file. The next time the user invokes the tool, they’re running whatever those bytes turned out to be.

The original implementation downloaded the release asset over HTTPS and extracted it. HTTPS gets you transport security: the bytes weren’t tampered with in flight. It tells you nothing about whether the bytes were right when they left, or whether they’re even the bytes you meant to fetch. A truncated download, a CDN cache serving a mangled object, a release asset that got swapped after the fact… HTTPS waves all of those straight through. For the one operation in the whole tool that replaces the binary, “we didn’t check” is an uncomfortable place to be sitting.

GoReleaser already does half the job

The good news is that the build side was already producing exactly what I needed. GoReleaser, which builds go-tool-base’s releases, generates a checksums.txt for every release: one SHA-256 per published artefact, the same format sha256sum emits. It was sitting right there as a release asset and nothing was reading it.

So Phase 1 of the integrity work is exactly that: read it.

When update downloads the platform binary, it now also fetches checksums.txt from the same release, looks up the entry for the asset it just pulled, and compares the SHA-256 of the downloaded bytes against the expected hash before anything gets extracted or installed. Mismatch, and the update aborts before it has so much as touched the installed binary. The hash comparison runs in constant time, which is more defence-in-depth than strictly necessary here, but it costs nothing and means every hash comparison in the codebase is the same and reassuringly audit-boring.

Fail open, or fail closed?

The interesting design question wasn’t the hashing. It was: what do you do when there is no checksums.txt?

Plenty of older releases predate this feature. A release might have been cut by hand without GoReleaser. If go-tool-base flatly refused to update whenever a manifest was missing, the very act of shipping this feature would brick the update path for every existing tool the moment they upgraded into it. That’s a cure worse than the disease.

So the default is fail-open: no manifest, log a clear warning, proceed. It matches how the existing offline-update path already behaved with its optional .sha256 sidecar, and it keeps upgrades working.

Fail-open as a default is not the same as fail-open being right for everyone, though. A security-sensitive tool should be able to say “no manifest, no update, full stop”. Two ways to get there:

Tool authors flip a compile-time switch (setup.DefaultRequireChecksum = true in main()) and their binary ships fail-closed from day one.
End users override either way through config (update.require_checksum) or an environment variable.

go-tool-base itself ships with the strict setting turned on, because a tool whose entire job is being a careful framework should hold itself to the stricter bar.

The caveat

Security features oversell themselves constantly, so here is the limit, stated plainly.

A checksum hosted next to the binary it describes protects you from accidents. Corruption, truncation, a CDN serving stale junk, a release asset that got partially clobbered. It does not protect you from a determined attacker who’s compromised the release platform itself. If someone can replace the binary, they can replace checksums.txt in the same breath, and your tool will cheerfully verify a malicious download against a malicious manifest and pronounce it good.

That’s not a flaw in the implementation. It’s the inherent ceiling of same-origin integrity: the manifest and the artefact share a trust root, so they fall together. Closing that gap needs a signature whose trust root is somewhere the release platform can’t reach, a key the attacker doesn’t have. That’s the next phase of this work, and it’s a bigger piece: GPG-signing the manifest, with the public half both embedded in the binary and published independently so a single platform compromise isn’t enough.

Phase 1 is the floor, not the ceiling. But it’s a floor worth having, because the overwhelming majority of real-world “the download was wrong” incidents are accidents, not attacks, and accidents are exactly what a same-origin checksum catches.

Pulling it together

The update command is the most trusting code in a self-updating tool: it fetches bytes from the internet and then becomes them. go-tool-base now verifies the SHA-256 of every self-update download against the release’s own checksums.txt before installing. It fails open by default so shipping the feature doesn’t strand anyone on an un-updatable version, fails closed for tool authors who ask (go-tool-base itself does), and stays honest that a same-origin checksum stops accidents, not a platform compromise.

Verifying your own downloads is a low bar. The point is that the previous height of that bar was zero.

Two API decisions that quietly contradict each other

Thu, 23 Apr 2026 00:00:00 +0000

Two design decisions on one enum, each sensible on its own, that would have quietly fought each other if I’d let them. I didn’t, but only because the second one is easy to get wrong and the compiler wouldn’t have said a word either way.

Decision one: promise the list can grow

#[non_exhaustive] on the Feature enum. It tells downstream code it can’t match the enum exhaustively, so it has to keep a wildcard arm, which in turn means adding a variant later is a non-breaking, minor-version change. Nobody’s match stops compiling just because the enum grew. The doc comment says exactly that: it “keeps variant additions a minor-version change for downstream matchers.”

Decision two: hand out the whole list

A convenience all() returning every variant, because iterating over the lot is something you genuinely want to do. The tempting signature is a fixed-size array, [Feature; 11]: you know precisely how many there are, so why not put it in the type?

Why those two can’t both be true

The catch is a quirk of Rust that often trips up people arriving from other languages: the length of a fixed-size array is part of its type. [Feature; 11], an array of exactly eleven features, and [Feature; 12], exactly twelve, are not one type holding a different number of items the way they might be elsewhere. They are two genuinely different, incompatible types, about as interchangeable as i32 and i64.

So the moment you add a twelfth variant, a fixed-size all() forces an unhappy choice, and both options are bad. Bump the array to [Feature; 12] and you break every caller who wrote the old length down. Leave it at 11 and the new variant is silently dropped, leaving you a function called all that doesn’t return all of them. Either way the #[non_exhaustive] promise (adding a variant breaks nobody) is quietly cancelled by a return type that welded today’s count into the public API.

So `all()` returns a slice

Which is exactly what it does, and the doc comment spells out why, in crates/rtb-app/src/features.rs:

#[non_exhaustive]
pub enum Feature {
 Init, Version, Update, Docs, Mcp, Doctor,
 Ai, Telemetry, Config, Changelog, Credentials,
}

pub const fn all() -> &'static [Self] {
 &[Self::Init, Self::Version, Self::Update, /* ...the rest... */]
}

A slice length is a value, not part of the type. Add a variant, the slice gets one longer, and not a single downstream signature changes. The promise holds.

The thing to watch for

#[non_exhaustive] is a promise about the future. A fixed-size array is a fact about the present. You can’t keep both at once, and nothing will warn you that you’ve contradicted yourself, because each decision is individually fine. The trap is always the second API surface that quietly re-bakes the flexibility the first one promised. When you mark a type “free to grow,” go and check that nothing in its public interface has secretly written down how big it is today.

What survives a port, and what doesn't

Thu, 23 Apr 2026 00:00:00 +0000

Rebuilding go-tool-base in Rust turned out to be the most honest design review I’ve ever sat through, and I didn’t have to do anything except keep going. Porting a framework into a language with completely different idioms forces a separation you can’t fake: the parts that survive the move are design, and the parts that don’t are just habit.

Two columns

When you port a system between languages that don’t share idioms, every piece of it sorts itself into one of two columns, without you having to make the call.

In the first column is the outcome a piece of the design produces: every command receives the framework’s services, configuration is layered with a fixed precedence, commands register themselves, errors carry guidance to the user. In the second column is the mechanism that produced that outcome in the original language.

Things in the first column survive the port. You rebuild them, differently, because the tool genuinely needs them. Things in the second column do not survive. You find their replacement, and the Go version turns out to have been one valid implementation of an idea, not the idea itself. Doing this for go-tool-base, mechanism by mechanism, was more honest about my own design than any amount of sitting and staring at it would have been.

The container

go-tool-base hands every command a Props struct. It carries the logger, the config, the assets, the filesystem handle. Some of it is reached through loosely-typed accessors. It works well, and I wrote a whole post about it.

The outcome is column one: a command should receive one object, and that object should carry the framework’s services so the command doesn’t go assembling them itself. That survived. RTB hands every command an App.

The loosely-typed accessors were column two. In Rust an App is a plain struct with concrete fields, each one an Arc<T> so a clone is a few atomic increments rather than a deep copy. Nothing is keyed by string. Nothing is fetched by name and asserted to a type. The thing the container is for survived; the way Go expressed it did not.

Registration

A go-tool-base command self-registers using a package-level init() function, which Go runs before main() and which appends the command to a global slice.

The outcome, column one, is that a command lives in its own file and inserts itself into the framework with no central list to edit. That’s genuinely worth keeping.

The init() mechanism is column two, and Rust doesn’t even offer it: Rust deliberately has no code that runs before main(). The replacement is link-time registration through distributed slices, which gets its own post next. Same outcome, no global mutable state, assembled by the linker rather than by a startup phase.

Configuration

go-tool-base layers configuration with a precedence: flags over environment over file over defaults. Some of it is read back through key lookups.

The layering and the precedence are column one. They survived exactly. RTB layers config with the same ordering.

The key lookups were column two. In Rust the merged configuration is deserialised into your own serde struct, so a config value is a typed field you access like any other field, and a typo is a compile error instead of a missing key at runtime. The precedence survived; reading values back out of a string-keyed bag did not.

The error path

go-tool-base routes every error through one handler so presentation is consistent, which I also wrote up.

One consistent exit for errors is column one. It survived. What didn’t survive was the handler: RTB has no error-handler object at all, because Rust’s own return-from-main convention plus a report hook does the job the handler was built to do. That one has its own post too.

What the exercise was actually worth

Every mechanism told the same story. The container, the registration, the config access, the error path, the cancellation signal that go-tool-base carries on a context.Context and RTB carries on a CancellationToken. In every case the thing it achieved walked across to Rust untouched, and the Go code that achieved it was left behind.

That’s the useful result. Before this port I couldn’t have told you, for any given pattern in go-tool-base, whether it was load-bearing design or just the idiomatic Go way to write it that day. Now I can, because each one was forced to prove itself by being rebuilt from nothing in a language that flatly wouldn’t accept the original. Whatever survived was real. Whatever I had to replace was always replaceable, which means it was never really the point.

The upshot

Porting a framework into a language with different idioms separates design from habit for free. The outcome a pattern produces is design, and it survives the move. The mechanism that produced it is idiom, and it gets left behind for the new language’s equivalent.

go-tool-base’s Props bag, its init() registration, its key-based config access and its error handler were all idiom. The single context object, self-registration, layered precedence and a consistent error exit were all design, and all four came through to RTB intact. The next three posts take the most interesting replacements one at a time, starting with how a Rust command registers itself when the language won’t run anything before main.

rust-tool-base: the same idea, in a language that argues back

Wed, 22 Apr 2026 00:00:00 +0000

I built go-tool-base because I was sick of rebuilding the same CLI scaffolding every time I started a new Go tool. You’d think that would have taught me a lesson about doing things more than once. Apparently not, because I’ve now started building rust-tool-base: the same idea, the same itch, for Rust.

In my defence, there’s method in it.

The same itch, a different language

go-tool-base exists because I kept writing the same couple of hundred lines of wiring every time I started a new Go CLI. Config loading, logging setup, an update check, an error path, a help system. None of it was the tool. All of it had to be there before the tool could be.

Lately I’ve been learning Rust, and two things collided. The first is how I tend to learn a language. I’ve always picked them up reasonably quickly, and the way I do it isn’t with a tutorial that builds a toy, it’s by rebuilding something whose shape I already know cold, so that every decision is about the language rather than the problem. The second is that every time I started a Rust CLI of any size, I hit the very same gap I’d already filled once in Go.

So rather than learn Rust on a throwaway, I decided to learn it by building rust-tool-base: the same idea, the same niche, for Rust.

One housekeeping note before the series gets going. You don’t need to write Rust to follow it. The posts lean on a handful of language ideas, and rather than stop and re-explain each one mid-flow, I’ve gathered them into a short primer. If a bit of syntax trips you up along the way, that’s where to look. If you already write Rust, ignore me and read on.

The gap in Rust

The Rust ecosystem has a well-earned reputation for sharp, focused crates and a deliberate shortage of big opinionated frameworks. clap for argument parsing, figment for layered config, tracing for logging, miette for errors, ratatui for terminal UI, reqwest and tokio underneath. Each of them is genuinely best-in-class.

What nobody hands you is the assembly. Wiring those into one coherent product, and then adding self-update, AI integration, an MCP server, embedded documentation, credential handling, telemetry and a scaffolder, is real work, and it’s the same work on every project.

The closest existing neighbours stop short of it. cli-batteries is a thin preamble: argument parsing plus a logging subscriber plus panic and signal handling. starbase has a proper session and lifecycle model but is CLI-agnostic and shaped around the moonrepo tooling it came from. cargo-dist and cargo-release are about release packaging, not the runtime. Good tools, all of them, but none is the opinionated, full-lifecycle, scaffolded base that go-tool-base is in the Go world. That space is empty, and rust-tool-base is built to fill it.

Why it is not a port

The obvious way to build this would be to open go-tool-base and translate it file by file. I’m not doing that, and the reason matters enough that it’s the rule the whole project is built around.

go-tool-base is full of Go. It leans on a Props struct that carries the framework’s services in loosely-typed fields. It configures things with functional options. It registers commands using package-level init(). It threads a context.Context through every call. Those are all good, idiomatic Go. Transliterated into Rust they’d become code that argues with the compiler on every single line, because Rust has its own answers to every one of those problems and they are emphatically not the Go answers.

So rust-tool-base reaches the same outcomes by Rust’s means. Commands still self-register, but through link-time machinery instead of init(). There’s still one context object per command, but it’s strongly typed rather than a loosely-keyed bag. Configuration is still layered, but it lands in your own typed struct instead of a string-keyed lookup. Same philosophy, same shape of product, an entirely different ecosystem underneath. The README says it plainly: it’s a sibling, not a port.

Why do it twice at all

Three reasons, and they reinforce each other.

The first is plain usefulness. The next time I want a Rust CLI tool, I want the same head start go-tool-base already gives me in Go.

The second is the learning. Rebuilding a system I understand forces me to meet Rust’s idioms where they actually bite, not where a tutorial gently stages them. You learn ownership properly when a real design is pushing back at you.

The third is the one I didn’t expect, and it’s the subject of the next post. Building the same framework twice, in two languages, turns out to be the cleanest way to find out which of your original decisions were genuine design and which were merely idiom. The design survives the move. The idiom does not. Sorting one from the other has been the most interesting part so far.

Boiling it down

rust-tool-base is the Rust sibling of go-tool-base: the same batteries-included, scaffolded, opinionated CLI framework, aimed at the same gap, which in Rust is the gap between a pile of excellent crates and a coherent product.

It’s not a port. Transliterating Go idioms into Rust produces code that fights the language, so RTB reaches the same outcomes through Rust’s own mechanisms instead. The posts after this one walk through the specific cases: how commands register, how the builder works, how errors are reported, and a few things RTB can do that the Go version structurally can’t. First, though, the thing the exercise taught me about my own design.

The blank import that keeps a dependency out of your binary

Wed, 22 Apr 2026 00:00:00 +0000

go-tool-base can stash your credentials in the OS keychain, which most people building on it are perfectly happy about. But some of them ship into regulated and air-gapped environments where the binary isn’t permitted to contain keychain or session-bus code at all… not dormant, not unused, simply not there.

So I had a feature most users want and a minority must be able to provably not have. The way I ended up solving it is one of my favourite little bits of honest Go.

A feature some users have to be able to not have

go-tool-base needs somewhere to keep secrets: AI provider keys, VCS tokens, the occasional app password. The best home for those on a developer’s machine is the operating system’s own keychain. macOS Keychain, GNOME Keyring or KWallet on Linux via the Secret Service, Windows Credential Manager. So I wanted go-tool-base to support all three. (This is the keychain mode I mentioned back in the credentials post, finally getting the explanation I promised it.)

The Go library for that is go-keyring, and it’s good. The catch is what it drags in behind it. On Linux it talks to the Secret Service over D-Bus, which means godbus. On Windows it pulls wincred. Perfectly reasonable dependencies for a desktop tool.

Now here’s the constraint that made this interesting. Some of the people building tools on go-tool-base don’t ship to developer laptops. They ship into regulated sectors and air-gapped deployments where a security review will scan the binary, enumerate every dependency, and ask pointed questions about anything that does inter-process communication. For those builds, “the keychain code is there but we never call it” is not an acceptable answer. The reviewer’s position, and it’s a fair one, is that code which isn’t in the binary cannot be a finding.

So I had a feature that most users want, and a minority of users must be able to provably not have. Same framework, same release.

Why I didn’t reach for a build tag

The obvious Go answer is a build tag. Compile with -tags keychain to get it, leave the tag off to not. I started down that road. I even spent a while on an inverted version, a nokeychain tag, on the theory that the regulated build should be the one that has to ask, so a forgotten flag fails safe.

It works. It also isn’t very nice. Build tags are invisible at the call site. Nothing in the source tells you that a file only exists in some builds. The two worlds drift, because the tagged-out path isn’t compiled in your normal editor session and quietly rots. And the ergonomics for a downstream consumer are poor: every tool built on go-tool-base would have to know the right magic incantation and thread it through their own release pipeline correctly, forever.

I tried a second approach too: pull the keychain backend out into a completely separate Go module. That genuinely solves the dependency question (a module you don’t require can’t contribute to your go.sum). But a separate module for one backend is clunky. Separate versioning, separate release, separate repo, all for a single file’s worth of behaviour. It felt like using a shipping container to post a letter.

The shape that actually fits: a registry and an `init()`

The version I’m happy with leans on two boring, well-worn Go mechanisms and lets them do something quietly clever together.

First, pkg/credentials defines a Backend interface and a registry. By default the registry holds a stub backend that politely returns “unsupported” for everything. The framework only ever talks to the registered backend, whatever that happens to be.

Second, the keychain implementation lives in its own package, pkg/credentials/keychain, still inside the same module, no separate release to manage. That package has an init() that registers its go-keyring-backed backend:

//nolint:gochecknoinits // registration via import is the whole point
func init() {
 credentials.RegisterBackend(Backend{})
}

And go-keyring, godbus, wincred, the whole IPC dependency chain, are only imported by that package.

Now the trick. To switch keychain support on, you import the package. You don’t have to use anything from it. A blank import is enough, because a blank import still runs the package’s init():

// cmd/gtb/keychain.go - the entire file.
package main

import _ "gitlab.com/phpboyscout/go-tool-base/pkg/credentials/keychain"

That single line is the on/off switch for the shipped gtb binary. The blank import means init() runs, the keychain backend registers itself, and credential operations start routing through the OS keychain. No flag, no tag, no config.

The part that makes it provable

Here’s why this beats the build tag, and it comes down to one guarantee in the Go toolchain: the linker only includes packages that are actually imported.

If cmd/gtb/keychain.go exists, the keychain package is in the import graph, so go-keyring, godbus and wincred are linked in. Delete that one file and rebuild, and the keychain package is no longer reachable from main. The linker performs dead-code elimination, and the entire go-keyring chain is gone. Not dormant. Not present-but-unused. Absent from the binary.

That’s the bit a regulated build needs. It isn’t a promise that the code won’t run. It’s a structural fact that the code isn’t there, and you can hand a security reviewer an SBOM that proves it. go-keyring won’t appear, because it genuinely isn’t linked.

For a downstream tool built on go-tool-base the story is the same, and just as cheap. Want keychain support? Add the one-line blank import to your own cmd package. Must ship keychain-free? Don’t. Your binary’s dependency graph follows your import graph, exactly as Go always promised it would. The default (no import) is the locked-down one, which is the right way round for a safety property.

Why I like this more than I expected to

Build tags hide a decision in the compiler invocation. This pattern puts the decision in the source, as an import, where it’s greppable, obvious in code review, and impossible to get subtly wrong. There’s a real file called keychain.go whose entire content is one import, and it reads as exactly what it is: a switch.

It’s also just honest Go. No reflection, no plugin loader, no clever runtime. A registry, an init(), and the linker doing the one job it’s always done. The cleverness, such as it is, is in the arrangement, not in any individual piece.

Stepping back

go-tool-base needed OS keychain support for the many, and a way to provably exclude it for the few. Build tags could express the toggle but hid it in the build invocation and rotted in the dark. A separate module solved the dependency question but was far too much machinery for one backend.

Putting the keychain backend in its own package, activated by a blank import _ that fires its init(), gets you both: a one-line, in-source, code-reviewable switch, and, because the linker only links what’s imported, a build with the import omitted that contains none of the keychain dependency chain. Provable absence, not promised disuse.

If you’re carrying an optional dependency that some of your users need gone rather than merely idle, this is the pattern. Let the import graph be the feature flag.

Just enough Rust to follow along

Tue, 21 Apr 2026 00:00:00 +0000

I’m about to write a run of posts about building rust-tool-base, and they lean on a handful of Rust ideas that I’d otherwise have to keep stopping to explain. So here they are, up front, in one place. You don’t need to write Rust to follow the series. You need a feel for maybe six concepts, and this is a quick, friendly tour of them. If you already write Rust, skip it with my blessing.

Ownership and borrowing

This is the one everybody mentions, and the one the whole language is built around. Every value in Rust has exactly one owner, and when the owner goes away, the value is cleaned up. No garbage collector deciding when, no manual free. If you want to let another piece of code use a value without handing over ownership, you borrow it: &thing lends it out for reading, &mut thing for writing, and the compiler enforces that you can’t, say, change something while someone else is reading it.

The payoff, and the reason people put up with the up-front fuss, is that an entire family of bug (use-after-free, data races, dangling pointers) becomes a compile error rather than a 3am one. When a post says something “moves” or is “borrowed”, that’s all this is.

Traits are Rust’s interfaces

A trait is a named set of methods a type can promise to provide, exactly like an interface in Go or Java. impl Command for Greet { ... } reads as “the Greet type fulfils the Command contract.”

Two bits of syntax show up a lot. dyn Command means “some value whose concrete type I don’t know, but which implements Command”, decided at runtime. And because the compiler needs a known size, you usually see it wrapped: Box<dyn Command> is “a pointer to some Command, whatever it turns out to be.” Whenever the series talks about a registry of Box<dyn Something>, it just means a list of different types that all satisfy the same trait.

Enums, `match`, and `#[non_exhaustive]`

A Rust enum is more than a list of named numbers; it’s a proper “one of these” type, and each variant can carry its own data. You handle one with match, which is like a switch that the compiler forces you to make complete: miss a case and it won’t build.

That completeness is usually a gift, but it’s awkward for a library, because adding a new variant would break everyone’s match. The fix is the attribute #[non_exhaustive]: it tells code outside the library “you must keep a catch-all _ => arm, because I reserve the right to add variants later.” With that in place, growing the enum is a non-breaking change. (One whole post turns on a subtle way to accidentally cancel that promise.)

The type system carries facts, not just shapes

Here’s an idea that surprises people coming from other languages: a Rust type often encodes more than “this is a number” or “this is a list.” The size of a fixed array is part of its type, so [Feature; 11] and [Feature; 12] are genuinely different, incompatible types, not one type holding a different count.

Pushed further, you can make the type track state. A “typestate” builder changes type as you call it, so .build() literally doesn’t exist as a method until every required field has been set, and forgetting one is a compile error rather than a runtime surprise. When a post says the compiler “won’t let you” do something, this is usually how: the mistake was made unrepresentable in the types.

`Result` and the `?` operator

Rust has no exceptions. A function that can fail returns a Result<T, E>: either Ok(value) or Err(problem), and you can’t use the value without acknowledging the error case. Writing that check by hand everywhere would be miserable, so there’s a shorthand: the ? operator. let x = thing()?; means “if this returned an error, return it up to my caller right now; otherwise give me the value.” Errors travel up the call stack as ordinary return values until something handles them, or until they fall out of main.

Crates, the workspace, and features

A crate is Rust’s unit of compilation, roughly “a library or binary.” A workspace is a bundle of crates built together, which is how rust-tool-base is laid out: rtb-app, rtb-cli, rtb-config and so on, each its own crate. And Cargo features are compile-time switches declared in Cargo.toml: turn a feature off and the code it guards, and any dependency it pulled in, is never compiled into your binary at all. Not disabled at runtime; simply absent. That distinction does real work in one of the posts.

That’s the toolkit

Ownership and borrowing, traits and dyn, enums and match and #[non_exhaustive], types that carry facts, Result and ?, and crates with features. Six ideas, and they’re enough to read everything else in this series without tripping over the language itself. Where a post needs a seventh thing, it’ll explain it in passing. Now, on with the actual building.

Where should a CLI keep your API keys?

Mon, 20 Apr 2026 00:00:00 +0000

Your CLI tool needs the user’s API key. It has to come from somewhere, and it has to survive between runs, so the obvious move is to ask once and write it into the config file. One tidy api_key: line. Job done.

It works beautifully on the first afternoon. And then, months later, it’s quietly become a liability nobody actually decided to create.

The config file that quietly becomes a liability

Your CLI tool needs the user’s API key. It has to come from somewhere, and it has to survive between invocations, so the obvious move is to ask once and write it into the tool’s config file. ~/.config/yourtool/config.yaml, a nice api_key: line, done.

It works on the first afternoon. It keeps working. And then, slowly, it becomes a problem nobody decided to create.

The config file gets committed to a dotfiles repo. It gets caught in a tar of someone’s home directory that lands in a backup bucket. It scrolls past in a screen share. It sits, world-readable, on a shared build box. None of these are exotic. They’re just a Tuesday. The plaintext key was fine right up until the file went somewhere the key shouldn’t, and config files go places.

I didn’t want go-tool-base handing every tool built on it that same slow-motion liability by default. So credential handling got rebuilt around a simple idea: the config file should usually hold a reference to the secret, not the secret itself.

Three modes, and which one you get

go-tool-base supports three ways to store a credential.

Environment-variable reference, the default. The config records the name of an environment variable, not its value:

anthropic:
 api:
 env: ANTHROPIC_API_KEY

The secret itself lives in your shell profile, your direnv setup, or your CI platform’s secret store, wherever you already keep that sort of thing. The config file now contains nothing sensitive at all. You can commit it, back it up, paste it into a bug report. The reference is inert on its own.

OS keychain, opt-in. The config holds a <service>/<account> reference and the actual secret goes into the operating system’s keychain: macOS Keychain, GNOME Keyring or KWallet via the Secret Service, Windows Credential Manager.

anthropic:
 api:
 keychain: mytool/anthropic.api

This one is opt-in by design, because the keychain backend carries dependencies that some deployments simply aren’t allowed to ship. (That opt-in mechanism turned out to be an interesting little problem all of its own, and it gets its own post in a couple of days.)

Literal value, legacy and grudging. The old behaviour. The secret sits in the config in plaintext:

anthropic:
 api:
 key: sk-ant-...

It still works, because breaking every existing tool’s config on an upgrade would be its own kind of vandalism. But it’s the last resort, it’s documented as the last resort, and the setup wizard puts a warning in front of you when you pick it.

The one place literal mode is not allowed

There’s a single hard “no” in all of this. If go-tool-base detects it’s running in CI (CI=true, which every major CI platform sets) the setup flow will refuse to write a literal credential, and exits non-zero.

The reasoning is that a plaintext secret written during a CI run is a plaintext secret written onto an ephemeral, often shared, frequently-logged machine, by an automated process that no human is watching. That’s the exact situation where the slow-motion liability becomes a fast one. CI environments inject secrets as environment variables already; there’s no good reason for a tool to be writing one to disk there, so go-tool-base simply won’t.

How it decides at runtime

A credential can be configured more than one way at once. You might have an env reference and an old literal key still lurking. So resolution follows a fixed precedence, highest to lowest:

The *.env reference. If that env var is set, use it.
Otherwise the *.keychain reference. If a keychain entry resolves, use it.
Otherwise the literal *.key / *.value, the legacy path.
Otherwise a well-known fallback env var (ANTHROPIC_API_KEY and friends), so a tool still picks up the ecosystem-standard variable with no config at all.

The useful property here is that adding a more secure mode transparently wins. Drop an env reference next to an old literal key and the next run uses the env var. You can migrate a credential to a better home without first removing it from its worse one, which makes the migration safe to do incrementally instead of as one nervous big-bang edit.

The tool tells on itself

A precedence rule is no use if nobody knows their config still has a plaintext key three layers down. So the built-in doctor command grew a check for exactly that. Run doctor, and if any literal credential is sitting in your config it reports a warning, names the offending keys (the key names, never the values) and points you at how to migrate.

It’s not an error. Literal mode is still legal. But the tool will quietly keep reminding you that you left the campsite messier than you could have, until you go and tidy it. (Old Scout habits die hard, and they’ve leaked all the way into the framework.)

The gist

A CLI tool that writes your API key into a plaintext config file isn’t doing anything wrong, exactly. It’s just handing you a liability that activates later, when the file travels somewhere the key shouldn’t. go-tool-base’s answer is three storage modes: an env-var reference by default, the OS keychain on request, and a plaintext literal only as a documented last resort that CI environments can’t use at all. Runtime resolution runs in a fixed precedence so a more secure mode always wins, which makes migrating a credential safe to do gradually. And doctor keeps an eye on the config so a stray plaintext secret doesn’t get to hide forever.

The secret should live in a secret store. The config file should just know its name.

A configurable AI endpoint is an attack surface

Sun, 19 Apr 2026 00:00:00 +0000

“Let users point at their own AI endpoint” is one of those config options that looks completely harmless on the way in. People want it, for perfectly good reasons. Then you sit with it for a minute and realise you’ve handed every user a loaded gun and pointed it vaguely at their own API key.

Why you offer it at all

There are real reasons to let someone set a custom base URL. They’re running a local model and want localhost:11434. They’re behind a corporate proxy that fronts the real provider. They’re on Azure’s flavour of OpenAI, which lives at a different host. They’ve a self-hosted gateway doing rate-limiting. All reasonable, all things a framework should support rather than fight.

The bit that’s a loaded gun

Here’s what the config option quietly decides: the base URL is where your credential goes. The API key rides along in an Authorization header on every request, to whatever host that URL resolves to. So the moment the endpoint is user-configurable, the destination of your secret is user-configurable too.

And users do user things. They paste a URL from a gist that turned out to be a honeypot. They leave http:// on the front, so the key crosses the wire in plaintext. They copy https://user:token@host/v1 not realising the userinfo changes who they actually authenticate to. They never edit the https://api.example.com/v1 placeholder and wonder why the key’s been posted to a domain they don’t own. None of that is malice. It’s what happens when the destination of a secret is a free-text field.

Validate before the first byte leaves

So every chat.New routes through ValidateBaseURL before the provider is built. The threat model is written at the top of pkg/chat/baseurl.go: an operator who can influence config could “redirect chat-provider traffic to an attacker-controlled HTTPS host and capture the Authorization header.” The checks run cheapest-first: a length cap, no ASCII control characters, must parse, no userinfo, https only, a host must be present, and the host mustn’t be a placeholder.

The userinfo rule is the sharp one:

if parsed.User != nil {
	// Reject any userinfo, with or without password. Never log
	// the URL itself because it contains the credential.
	return errors.WithHint(ErrInvalidBaseURL,
		"base URL must not contain credentials; use the Token field instead")
}

The placeholder check rejects example.com and friends and any subdomain of them, so the unedited https://api.example.com/v1 from a setup wizard never reaches the wire and hits some typosquatted lookalike. And the HTTP escape hatch is test-only by construction: the AllowInsecureBaseURL field that permits plain http is tagged json:"-", so a config file physically cannot set it. This all came out of the 2026-04-17 security audit, finding M-3.

rust-tool-base enforces the same at its own boundary: validate_base_url rejects userinfo, any scheme but https (bar a test-only allow_insecure), and documentation placeholder hosts like example.com.

What it can and can’t do

It won’t stop a user who deliberately points the tool at a malicious HTTPS host they genuinely chose. If someone is set on sending their own key somewhere bad, validation can’t read their mind.

What it stops is the accidents: the plaintext slip, the userinfo confusion, the placeholder nobody changed. Those aren’t theoretical, they’re the ones that happen to careful people on ordinary days. Storing the key well is one job (where a CLI keeps it), stopping it leaking through a log is another, and this is the third side of the triangle: once you’ve stored it and stopped it leaking, make sure you don’t send it somewhere daft.

Redacting the secret you didn't know was in the string

Sat, 18 Apr 2026 00:00:00 +0000

Dammit! How did that get there?

A log line that should never have existed. Not a password I’d carelessly printed, nothing as obvious as that. An upstream API handed me back an error, and it had quoted my own bearer token inside the message, and that error went straight into the logs the way errors do. I didn’t put the secret there. The error did. And I’d never have caught it by being careful, because being careful only protects you from the secrets you know you’re handling.

The easy half of redaction

Hiding the secrets you know about is the part everyone does. You’ve got an API key field, a password flag, so you mask them at the point you print them. key=****. Done, and it feels like you’ve solved redaction, when really you’ve solved the half that was never going to bite you.

The half that bites

The secrets that escape are the ones that arrive inside strings you don’t control. An upstream service echoes your token back in a 401 body. A connection string with the password in the userinfo, https://user:pass@host, lands in a debug line. A library stringifies a whole request, headers and all, for a “helpful” trace. You cannot field-mask a secret you didn’t know was in the string, because you never watched it go in.

You can’t register a value you never had, so match the shape

This is the bit I got wrong in my own head at first. I assumed redaction meant handing it the secrets I was holding so it could watch for them. But the dangerous secrets are exactly the ones I’m not holding a copy of. So pkg/redact doesn’t keep a registry of your values at all. It knows what secrets look like.

pkg/redact/redact.go carries a set of RE2 patterns: a credential in URL userinfo, an Authorization: header sitting in free text, query-string credentials, and the well-known provider prefixes:

prefixPatterns = []*regexp.Regexp{
	regexp.MustCompile(`sk-[A-Za-z0-9_\-]{16,}`), // OpenAI / Anthropic-style
	regexp.MustCompile(`ghp_[A-Za-z0-9]{30,}`), // GitHub PAT classic
	regexp.MustCompile(`github_pat_[A-Za-z0-9_]{30,}`), // GitHub fine-grained PAT
	regexp.MustCompile(`xox[baprs]-[A-Za-z0-9-]{10,}`), // Slack
	regexp.MustCompile(`AIza[A-Za-z0-9_\-]{30,}`), // Google API key
	regexp.MustCompile(`AKIA[A-Z0-9]{16}`), // AWS access key ID
}

Run any string through redact.String and an OpenAI key, a GitHub token or an AWS access key ID gets caught wherever it’s hiding, in an error you didn’t write, in a URL, in a stack trace, because each has a recognisable shape. For the secrets that don’t announce themselves with a prefix there’s a fuzzy fallback: any opaque alphanumeric run of 41 characters or more. The 41 is chosen on purpose, to clear UUIDs (36), MD5 (32) and git SHA-1 (40) without flagging them, while accepting that a SHA-256 (64) will trip it. A deliberate, documented trade rather than a magic number.

Where it runs

At the boundary where a string leaves for somewhere you can’t reach back into. The telemetry backend runs every event argument and error message through redact.String before it emits anything (pkg/telemetry/telemetry.go), and both telemetry and HTTP logging drop the value of any header redact flags as sensitive. It doesn’t matter which code path produced the string, or whether you even wrote that path; everything goes through the same gate and gets the same scrub.

rust-tool-base’s rtb-redact crate takes the same shape-matching approach: regex patterns, the same family of well-known provider prefixes, and an is_sensitive_header check for header values.

A realistic limit

It isn’t a force field. A secret with no recognisable shape, shorter than the fallback threshold, will sail through. You cannot redact what you cannot recognise. But the leak that actually keeps happening isn’t some exotic unknown, it’s a well-known token turning up in a place you didn’t expect, and a shape-matcher sitting at the edge catches exactly that, including secrets you never told it about. Which is the one thing registering your own values could never have done. Storing the key safely is a separate job, where a CLI keeps it; this is about making sure that, having stored it, it doesn’t quietly fall out through a log.

I had the framework audited: every finding was the same shape

Fri, 17 Apr 2026 00:00:00 +0000

When a real security audit lands back in your inbox, the temptation is to read it as a shopping list of unrelated mistakes. Fix one, fix the next, tick them off, move on. I did exactly that the first time. The second time, I noticed something far more useful: the findings weren’t scattered at all. They clustered. Almost every one was the same sentence with the nouns swapped out.

Findings cluster, they don’t scatter

When you get a real security audit back, the instinct is to read it as a list of unrelated mistakes. Finding 1, unrelated to Finding 2, unrelated to Finding 3. Triage each, fix each, move on.

That’s not what the go-tool-base audits looked like once I stopped reading them as a list. The findings clustered. Strip away the specifics and almost every one was the same sentence with the nouns swapped: untrusted input reaches a powerful operation, and nothing checks it in between.

That reframe is worth more than any individual fix, because it turns “we patched some bugs” into “we know where to look next time”. A framework’s attack surface isn’t spread evenly. It’s concentrated at the boundaries: the handful of points where data from outside (a config file, a command-line flag, something typed into a TUI, an HTTP response) flows into machinery that can be made to misbehave. Audit the boundaries and you’ve audited most of the risk. Three examples make the pattern obvious.

Boundary one: a regex compiler

Somewhere in the tool, a user-supplied string gets compiled into a regular expression. A search pattern typed into the docs browser, a filter from a config file. Feeding user input to regexp.Compile feels harmless. It’s just pattern matching, after all.

It isn’t quite harmless. A regular expression is a tiny program, and some tiny programs are catastrophically slow. A pattern with the wrong kind of nested repetition can take exponential time to evaluate against a modestly hostile input. That’s the class of bug known as ReDoS. A user, or something feeding the user’s config, hands you a pathological pattern and your tool wedges, burning a whole core, on what looked for all the world like a search box.

The fix isn’t to ban user-supplied regexes. It’s to stop treating “compile this string” as free. go-tool-base routes any regex whose pattern came from outside the binary through a regexutil.CompileBounded helper. It caps the pattern length and puts a hard timeout on compilation. A pattern known at build time can still use plain regexp.MustCompile, because that isn’t a boundary, it’s a constant. The discipline only applies where the input genuinely crosses in.

Boundary two: a URL opener

The tool needs to open a URL in the user’s browser, a docs link or an OAuth flow. Under the hood that’s the OS handler: xdg-open, or open, or rundll32.

Now ask where the URL came from. If any part of it is influenced by config, by a server response, by user input, then “open this URL” has quietly become “ask the operating system to do something with an attacker-influenced string”. A file:// URL. A javascript: URL. Something with control characters smuggled into it. The browser-open was never the dangerous part. The unvalidated string was.

So go-tool-base funnels every URL-open through one package, pkg/browser, and that package is a gate. It enforces an allowlist of schemes (https, http, mailto, and nothing else), bounds the length, and rejects control characters before the OS ever sees the string. The rule that makes it stick is that nothing else is allowed to call the OS handler directly. One door, and the door has a lock. A scattered capability with no chokepoint can’t be secured; a capability that has a chokepoint can. (You’ll have spotted the “one door out” idea by now… it’s the same instinct as the single error handler, pointed at security instead of consistency.)

Boundary three: a log sink

This one’s the sneakiest, because it runs the wrong way round. The first two boundaries are about dangerous input coming in. This one is about sensitive data leaking out.

The tool handles credentials. It also logs, emits telemetry, and reports errors, and all three of those are exit boundaries: places where strings leave the process for somewhere more persistent and more public, like a log aggregator, an analytics backend, an error tracker. If a token ever ends up in a string that flows to one of those, you haven’t logged an event, you’ve published a secret.

The defence is pkg/redact. Any free-form string heading for an observability surface goes through it first, and it strips the usual suspects: credentials in URL userinfo, sensitive query parameters, Authorization headers, the well-known provider key prefixes (sk-, ghp_, AIza and friends), long opaque tokens. The places most likely to leak, command arguments and error messages in telemetry, get it applied automatically rather than relying on every caller to remember.

Same pattern as the other two. A boundary, and something standing on it checking what goes through.

The unglamorous part

None of these fixes is clever. There’s no exploit demo, no neat trick to show off. Bound a length. Check a scheme against an allowlist. Run a string through a redactor. The work was almost entirely in noticing the boundary existed, and then making sure everything routes through the one checked path instead of dotting raw calls all over the codebase.

That’s the actual lesson of a security audit, and it’s why the cluster reframe matters. The value wasn’t the dozen-or-so individual fixes. It was learning that the next risk will be at a boundary too, the next place untrusted input meets a powerful operation with nothing in between, and that the job is to find those points and put a single, mandatory, checked door on each.

To sum up

A security audit of a CLI framework reads like a list of unrelated bugs and isn’t one. go-tool-base’s findings nearly all reduced to the same shape: untrusted input reaching a powerful operation unchecked. A regex compiler that needed a length and time bound (regexutil.CompileBounded). A URL opener that needed a scheme allowlist and a single chokepoint (pkg/browser). Log and telemetry sinks that needed credentials redacted on the way out (pkg/redact).

The fixes were structural and dull, which is exactly right. Find your boundaries (config, flags, TUI input, network responses, log and telemetry sinks), give each one a single mandatory checked path, and you’ve spent your audit effort where the risk actually lives.

A mutex on a flag nobody writes twice

Thu, 16 Apr 2026 00:00:00 +0000

“Why is there a mutex around a boolean that only ever gets set once?”

It’s a fair question, and I’d half-asked it of myself before someone asked it of me. The answer turns out to be written, in as many words, in a code comment I’ve grown rather fond of.

The registry and its one-way latch

go-tool-base keeps a feature registry: the initialisers, sub-commands, flags and checks that each feature adds to the CLI. Features register themselves into it at startup, from init(), before main runs. Once everything’s wired, the framework calls SealRegistry() and the registry latches shut. Any Register call after that point panics, on purpose, because a sub-command or flag that turns up after the CLI has parsed its arguments is a bug I want to hear about at once, not discover three releases later.

So there’s a registrySealed bool. It starts false, SealRegistry flips it to true exactly once in normal operation, nothing flips it back outside of tests, and it’s read on every registration attempt. Written once, read many. The textbook shape of “you don’t need a lock for this.”

Except the comment disagrees, on purpose

Here is the actual declaration, in pkg/setup/registry.go:

// registryMu protects globalRegistry and registrySealed. Acquired for write
// by all Register* and Reset/Seal helpers; acquired for read by all Get*
// accessors. The mutex is required for memory visibility of registrySealed
// across goroutines, not only mutual exclusion on the maps.
var (
	registryMu sync.RWMutex
	registrySealed bool
)

That last sentence is the entire post. The mutex has an obvious day job: the registry is a clutch of maps that get appended to during registration, and concurrent appends need genuine mutual exclusion. registrySealed could have just hitched a ride on that lock and nobody would have thought twice. But the comment goes out of its way to say the lock is also required for the flag, for visibility, not only exclusion.

Why a write-once bool still needs the lock

The Go memory model makes no promise that a goroutine reading registrySealed will ever see the write SealRegistry made, unless there is a happens-before relationship between them. No synchronisation, no guarantee. A reader can sit there seeing false long after the seal happened on another goroutine, because the compiler may cache the read and the CPU may serve it from a core-local view. And a concurrent read and write of the same variable, with nothing ordering them, isn’t “probably fine”; it’s a data race, which Go defines as undefined behaviour.

“But registration is single-threaded, it’s all init().” It was, right up until we wanted the tests to run in parallel. This lock exists because of a deliberate campaign to restore t.Parallel() across the codebase after a stack of races forced us to drop it (the same campaign that retired the package-level mocking hooks). Tests build, register, seal and reset this registry from parallel goroutines. The instant that’s true, the seal check has to stay correct while racing, because the very thing it guards against is concurrency. So reads take registryMu.RLock, the write takes registryMu.Lock, and now there’s a happens-before edge: anyone who acquires the lock after SealRegistry released it is guaranteed to see true.

What the lock is actually for

It isn’t there to stop two goroutines both sealing the registry. There’s only ever the one seal. It’s there so that every reader can trust what it reads. A value written exactly once is precisely the case where you’re most tempted to skip the synchronisation, and precisely the case where skipping it can leave a reader legally staring at the stale value for good. The comment spells it out so that the next person to glance at registrySealed, think “that clearly doesn’t need a lock,” and reach for the delete key, reads the sentence first.

(There’s a sibling sealed flag in the middleware registry that follows the identical pattern, for the identical reason.)

The test-mocking pattern that races

Thu, 16 Apr 2026 00:00:00 +0000

I’m going to tell you about a bug go-tool-base shipped, because it’s one of those bugs that’s so reasonable-looking you’ll find it in textbooks, conference talks, and an awful lot of otherwise excellent Go code. We had it too. It passed every test on my laptop, every single time, and then quietly fell over on CI while blaming an innocent bystander.

It’s the classic Go trick for mocking a dependency, and it races.

A pattern that looks completely reasonable

Here’s a thing you need to do constantly in Go tests: stop a function from really shelling out. It calls exec.LookPath to find a binary, or exec.Command to run one, and your test very much does not want it touching the real $PATH or spawning a real process.

The Go community has a well-worn answer. Hoist the function into a package-level variable, call that, and let tests reassign it:

// production code
var execLookPath = exec.LookPath

func findTool() (string, error) {
 return execLookPath("sometool")
}

// test
func TestFindTool(t *testing.T) {
 old := execLookPath
 defer func() { execLookPath = old }()
 execLookPath = func(string) (string, error) {
 return "/fake/path", nil
 }
 // ...assert...
}

It’s tidy. No interface to thread through, no constructor to change. You’ll find it in a great deal of Go code, including some very respectable Go code indeed. go-tool-base had it too.

And it works. It works on your machine, it works in code review, it works the first hundred times CI runs it. Which is precisely what makes it dangerous, because it’s wrong, and it’s just been biding its time.

Add one line and it detonates

Go’s t.Parallel() is more or less free performance. Mark your tests with it and the runner overlaps them instead of plodding through one at a time. On a package with a few hundred tests it’s a real, worthwhile speed-up, so naturally you reach for it.

Now picture two tests, both using the pattern above, both marked t.Parallel(). They run concurrently. Test A assigns its fake to execLookPath. Test B assigns its fake to execLookPath. Test A reads execLookPath expecting its own fake. Two goroutines, one variable, writes and reads with nothing synchronising them. That’s a textbook data race, and the textbook is right: the behaviour is undefined. Test A might see B’s fake. The deferred restore might land in the wrong order and leave the variable pointing at a fake after both tests have finished, poisoning a third one for good measure.

The truly nasty part is the intermittency. Whether the race actually bites depends on goroutine scheduling, which depends on machine load and core count. Your laptop running eight tests at once might never lose the coin-toss. A CI runner under load, scheduling differently, loses it and fails a test that has nothing obviously to do with the change in the commit. You re-run the pipeline, it passes, everyone shrugs and moves on. A test suite that fails one run in twenty trains your team to ignore it, and an ignored CI failure is worse than no CI at all.

I can tell you this one from direct, slightly embarrassed experience, because go-tool-base shipped exactly this bug and CI caught it the honest way: green on the laptop, red on the runner, with the failure cheerfully pointing at innocent bystander tests rather than the global that was actually the culprit. go test -race will name it for you if you crank the parallelism up high enough to lose the toss reliably… but you have to go looking, and you only go looking once it’s already ruined an afternoon.

The fix isn’t synchronisation, it’s structure

The instinct is to slap a mutex around the variable. Resist it. A mutex makes the race defined, but it doesn’t make the design any good. You’ve still got global mutable state, you’ve just queued the fight instead of cancelling it. And tests that serialise on a shared lock aren’t really parallel any more, so you’ve also handed back the speed-up you came for in the first place.

The real fix is to not have a shared variable at all. The dependency was always an input to the code; the package-level var was just a way of avoiding saying so out loud. So say it. Inject it.

A struct field:

type Finder struct {
 lookPath func(string) (string, error) // defaults to exec.LookPath
}

func (f *Finder) find() (string, error) {
 return f.lookPath("sometool")
}

Or a functional option, if you’d rather keep the zero value clean. Either way, each test constructs its own Finder with its own fake. There’s no shared variable, so there’s no race, and t.Parallel() is free again because the tests genuinely don’t touch each other.

go-tool-base wrote this straight into its standing rules: no package-level mocking hooks, full stop. Dependencies come in through struct fields, functional options, or config fields. (The same injection discipline that makes Props so testable, applied one rung further down.) And to stop everyone hand-rolling the same exec fakes, there’s a small internal package, internal/exectest, with ready-made LookPath and CommandContext doubles you construct per-test. The pattern is gone, and the door it came in through is shut.

The rule worth taking away

A package-level variable that tests reassign is shared mutable state. It reads as a harmless convenience because in a single-threaded test run it behaves like one. t.Parallel() is the thing that reveals it was never harmless, only unobserved.

The general lesson is older than Go: if a value is an input to your code, make it an input. Smuggling it in as a global is borrowing test-time convenience against a debt that comes due, with interest, the day someone wants their tests to run in parallel. Pay cash. Inject the dependency.

Worth remembering

Mocking via a reassignable package-level variable is a beloved Go shortcut and a latent data race. It survives because single-threaded test runs hide it; t.Parallel() exposes it as intermittent, bystander-blaming CI flake that’s miserable to trace. A mutex only makes the bad design defined. The fix is structural: inject the dependency as a struct field or functional option, so each test owns its own double and there’s no shared state to race over. go-tool-base banned the global-hook pattern outright and ships internal/exectest so nobody’s tempted back to it.

If a piece of code depends on something, let it say so in its signature. Your future self, staring at a CI failure that flatly refuses to reproduce, will thank you.

OpenSSF Scorecard graded my supply chain

Tue, 14 Apr 2026 00:00:00 +0000

I turned OpenSSF Scorecard on expecting a pat on the head. go-tool-base is a security-minded project, I’m careful, surely the robot would agree. The robot did not agree. It handed back a report card with a fair bit of red ink, and the most pointed finding on it wasn’t about my code at all. It was about me.

A linter for the things you don’t call code

Scorecard is an automated set of checks that grades a repository’s supply-chain hygiene: are your CI dependencies pinned, are your workflow tokens least-privilege, is your branch protected, do commits get reviewed. It’s a linter, but pointed at the part of the project you don’t usually think of as code, the build and release machinery and the practices around them. And like any good linter, its value is mostly in catching the things you’d swear you’d already got right.

Three of its findings were worth the price of admission on their own.

Pin the actions you don’t control

The first was about how go-tool-base’s GitHub Actions referenced other actions. Like nearly everyone, I’d written uses: actions/checkout@v6. Scorecard doesn’t like that, and it’s right not to.

@v6 is a tag, and a tag is mutable. Whoever controls that action can move v6 to point at different code tomorrow, and your CI will pick it up silently on the next run. For an action that runs in a job holding your repository token, that’s a supply-chain hole the width of a barn door: compromise the tag, compromise every pipeline that trusts it. The fix is to pin to an immutable commit SHA, with the human-readable version left as a comment, which is exactly what I changed:

-  - uses: actions/checkout@v6
-  - uses: actions/setup-go@v6
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+ - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6

Now the action is frozen at bytes I reviewed. Dependabot still bumps the SHA when a real new version lands, so I get updates as reviewable pull requests rather than as silent tag movements. The pin doesn’t stop me updating. It stops me updating without noticing.

Give the workflow token the least it can do

The second finding was about permissions. My workflow declared its token permissions at the top, once, for the whole file:

permissions:
 contents: read
 security-events: write
 id-token: write

That reads as careful, and it’s still too broad, because top-level permissions apply to every job in the workflow. A job that only needs to read the repo is now also holding id-token: write and security-events: write, for no reason other than that some other job in the same file needed them. Scorecard rejects exactly this, and the fix is to default the whole workflow to read-only and grant write narrowly, in the job that actually needs it:

permissions: read-all

Write permissions moved down into the single job that uses them. It’s the same least-privilege instinct that runs through everything else in these projects, just applied to a CI token instead of an IAM role: a credential should be able to do the one thing it’s for, and nothing else, no matter how convenient the broad grant looked.

The finding that was about me

The third one stung, because there was no YAML to fix. Scorecard’s Code-Review check scores how consistently changes are reviewed before they land, and mine scored badly for the most embarrassing possible reason: I’d set up branch protection on main, and then, being the solo maintainer in a hurry, I’d been merrily bypassing it to push straight to main whenever it suited me.

So I had a rule, written down and enforced by the platform, that I was personally and routinely ignoring. Scorecard noticed, totted up the unreviewed commits, and graded me on it. There’s something properly humbling about a robot reading your git history and pointing out that the person breaking your security policy most often is you. The fix wasn’t code. It was going through a pull request like everyone else, even when “everyone else” is just me on a different day.

The bottom line

OpenSSF Scorecard is a linter for your supply chain, and like any linter it’s most useful when it tells you something you were sure you’d already handled. It dinged go-tool-base for referencing actions by mutable tag instead of pinned SHA, for granting workflow-token write permissions at the top level where every job inherited them, and for a Code-Review score I’d earned fair and square by bypassing my own branch protection.

The first two were quick, satisfying changes with a clear security story. The third was the one that stuck, because the tool I’d added to grade the project ended up grading the maintainer, and was entirely right to. Turn it on. Brace yourself a little.

Testing code that calls an LLM: yes, you actually can

Wed, 08 Apr 2026 00:00:00 +0000

“You can’t test code that calls an AI.” I’ve heard it said with great confidence, and it’s half right, which is the most dangerous kind of right. You genuinely can’t assert on what a non-deterministic model says. But the model isn’t your code, and the bits sitting either side of it most certainly are.

“You can’t test AI code”

It’s a fair worry. Your command calls an LLM. The LLM returns something slightly different every run. A test that asserts response == "..." is broken before you’ve finished typing it. So the conclusion arrives quickly: the AI path can’t be tested, leave it uncovered.

Which is a shame, because the AI call is usually the riskiest line in the whole command.

The conclusion is also wrong. It mistakes “I can’t test the model” for “I can’t test my code”. The model is not your code. Your code is the two pieces sitting on either side of it.

Your code is a prompt and a handler

Strip the command down to what it actually does:

It builds a prompt. It assembles a system prompt, the user’s input, perhaps some context, and sends it.
The model does something. This is not your code.
It takes the response and does something with it. It parses it, branches on it, prints it, stores it.

Steps one and three are entirely yours, and entirely deterministic. The same inputs build the same prompt and handle the same response the same way, every single time. That’s testable. Step two is the only part that isn’t, and step two was never yours to test in the first place.

So the job is to pin step two to a known value, and then test one and three properly.

Test the prompt: snapshot it

Step one produces a prompt, and a prompt is just a string, which means you can pin it.

Both frameworks lean on snapshot testing here. go-tool-base uses a golden-file approach: the prompt your code generates is recorded to a file, and the test re-generates it and compares against that file. rust-tool-base does the same with insta, snapshotting the request body the client would send.

The reason this matters is that the prompt is load-bearing and quietly easy to break. You refactor how context gets assembled. Without noticing, you’ve changed the wording, or the ordering, or dropped a line the model was leaning on. Nothing fails to compile. The behaviour just drifts, silently.

A snapshot test catches exactly that. It fails, it shows you the diff between the old prompt and the new one, and it makes you stop and make a decision. Was this change intended? If yes, you accept the new snapshot and move on. If no, you’ve just caught a bug before it shipped. Either way the prompt never changes by accident, which for AI code is most of the battle.

Test the handler: mock the response

Step three needs a response to handle, and in a unit test you don’t get that response from the real model. You supply it.

go-tool-base ships generated mocks for the ChatClient interface. A test builds a mock client, tells it “when Ask is called, return this canned value”, and runs the command against it:

mockClient := mock_chat.NewMockChatClient(t)
mockClient.EXPECT().
 Ask(mock.Anything, mock.Anything, mock.AnythingOfType("*main.Analysis")).
 RunAndReturn(func(_ context.Context, _ string, target any) error {
 *(target.(*Analysis)) = Analysis{Severity: "critical"}
 return nil
 })

Because the interface is only four methods, that mock is trivial to set up and complete by construction. rust-tool-base takes the same idea one layer down: HTTP-bound tests use wiremock, which stands up a fake server returning a canned response body. The client makes a real HTTP request; it just goes to a fake endpoint the test controls.

Either way, step two is now fixed to a value you chose, which makes step three deterministic. And that unlocks the tests that actually matter: given a malformed response, does the command fail gracefully? Given a rate-limit error, an empty answer, a field missing? Those are the cases a live model almost never hands you on demand, and a mock hands you every time, on the first run.

This is, incidentally, the same discipline as the test-mocking work elsewhere in the framework: the dependency is injected, so the test gets to decide what it does.

What you deliberately don’t test

One boundary worth stating. None of this tests whether the model gives good answers. That question is real, but it’s a different activity (evaluations, run as their own suite) and not something to mix into the unit tests.

The unit suite’s job is your code: that it builds a sound prompt, and that it handles every shape of response correctly, including the ugly ones. Keep that well away from “is the model clever today”. A unit test that depends on the model being clever is a unit test that fails when the weather changes, and a flaky test just teaches people to ignore the whole suite.

What it comes down to

Code that calls an LLM is testable; the model is not, and those are different statements. Your code is a prompt builder and a response handler, both deterministic, with the model sat in between.

go-tool-base and rust-tool-base converge on the same approach. Snapshot the prompt, with golden files or insta, so a refactor can’t change what you send without a test noticing. Mock the response, with generated ChatClient mocks or a wiremock server, so tests run with no network and you can feed in the malformed and error cases a real model won’t reliably produce. Leave “are the answers any good” to a separate evaluation suite. Test the two halves you own, and the non-determinism in the middle stops being an excuse to leave the riskiest line uncovered.

The AI provider that isn't an API

Mon, 06 Apr 2026 00:00:00 +0000

go-tool-base’s chat package puts five AI providers behind one interface. Four of them are exactly what you’d guess: HTTP calls to OpenAI, Claude, Gemini, and anything OpenAI-compatible. The fifth one isn’t an API at all. It shells out to a binary.

That sounds like a slightly mad thing to want, right up until you’ve worked somewhere the network says no.

The fifth provider shells out

The chat package speaks to five providers through one ChatClient interface. Four of them are what you’d expect: HTTP requests to OpenAI, to Claude, to Gemini, to any OpenAI-compatible endpoint. The tool author picks one in config, and the rest of the code never knows the difference.

The fifth, ProviderClaudeLocal, is different in kind. It doesn’t make an HTTP request at all. It shells out. It runs the claude CLI binary as a child process, passes the prompt in, and reads the answer back from the binary’s output.

That sounds like an odd thing to want until you’ve been stuck in the environment it was built for.

Why you’d want that

Picture a corporate network with its egress locked right down. Outbound HTTPS to api.anthropic.com is blocked by policy. A tool built on go-tool-base that uses AI would simply fall over there. It tries to reach the API, there’s no route, and that’s the end of the feature.

But the developer at that machine has the claude CLI installed, and has run claude login. That binary is permitted. It’s an approved, managed tool, and it has its own sanctioned path out. The direct API call is blocked; the claude command is not.

ProviderClaudeLocal is what bridges those two facts. If your tool’s AI calls go through that already-blessed binary instead of straight at the API, they work, in an environment where the direct call cannot. That’s the whole reason the provider exists. It isn’t faster (a real API call has lower latency) and it isn’t more capable. It’s for the place where the API call simply isn’t an option, and “isn’t an option” is a surprisingly common place to find yourself inside a large organisation.

What it costs

It’s worth being straight about the trade, because ProviderClaudeLocal is the reduced-capability provider.

It doesn’t do tool calling. It doesn’t do parallel tools. It doesn’t stream. Those need a live, structured connection to the model’s API, and a subprocess that runs once and prints an answer is not that. What it does support is plain chat and structured output, the latter through the binary’s own --json-schema flag.

So the positioning, and the package’s documentation says exactly this, is: prefer the API providers when you can reach them, because they’re lower latency and feature-complete. Reach for ProviderClaudeLocal when API access is restricted. You accept the narrower capability set as the price of working at all. For a tool whose AI feature is “answer a question” or “return a structured analysis”, that price is often nothing you’d even notice. For one built on an agentic tool-calling loop, it’s a real limitation, and you’d know to expect it.

How it stays behind the same interface

Here’s the part that makes it pleasant rather than a special case to maintain. Despite being a subprocess and not an API, ProviderClaudeLocal is still a ChatClient. Your feature code calls Chat and Ask exactly the way it would for any other provider.

Everything that makes a subprocess provider awkward stays inside the provider. Spawning the binary, feeding it the prompt, parsing its output, capturing stderr and surfacing it when the binary exits non-zero, and threading multi-turn continuity through session identifiers passed back on the next call with --resume: all of that is the provider’s problem, and all of it sits behind the interface. The code in your tool that uses AI doesn’t know, and has no way to find out, that this particular provider is a child process rather than an HTTPS call.

That’s a unified interface genuinely earning its place. It’s easy to put a uniform face on four things that already work the same way underneath. The real test of the abstraction is whether something that works in a completely different way, a subprocess instead of a socket, can still slot in without the caller changing a line. Here it can. You swap one config value, and a tool that talked to an API now talks through a binary, and nothing downstream so much as blinks.

The bottom line

go-tool-base’s chat package puts five providers behind one ChatClient interface, and ProviderClaudeLocal is the one that isn’t an API. It runs the locally installed, pre-authenticated claude CLI as a subprocess.

It exists for the locked-down environment where outbound HTTPS to the AI API is blocked but the claude binary is allowed: there, AI features keep working where a direct call would fail. The trade is a narrower capability set (no tool calling, no streaming, plain chat and structured output only) so you prefer the API providers when you can reach them and fall back to this when you can’t. And because it’s still a ChatClient, all the subprocess machinery stays hidden, and your code uses it without knowing it’s there. That last part is the real test of an abstraction: a provider that works in an entirely different way still slots in unchanged.

AI conversations you can resume

Sat, 04 Apr 2026 00:00:00 +0000

An AI conversation is, fundamentally, its own history. The model’s next answer depends on everything said so far. And a CLI tool, by its very nature, forgets everything the moment it exits. Put those two facts together and you get the problem: run an AI command, exit, run it again, and you’re talking to someone who’s never met you.

A CLI forgets everything

A long-running service keeps its state in memory for as long as it runs. A CLI tool doesn’t get that luxury. It starts, does one thing, exits. The next invocation is a brand-new process with no memory of the last one.

For most commands that’s exactly right, and you wouldn’t want it any other way. But an AI conversation is a different kind of beast, because a conversation is its history. The model’s next answer depends on everything said so far. Run an AI command, exit, run it again, and you’ve started a fresh conversation with someone who’s never met you. For an interactive assistant, or any AI workflow that unfolds across several invocations, that’s plainly the wrong behaviour. The user expects to pick up where they left off.

Save and restore

The chat package handles this through a PersistentChatClient interface. Like streaming, it’s an optional capability discovered with a type assertion, sitting beside the four-method core rather than bloating it. A client that supports persistence also satisfies this interface:

if pc, ok := client.(chat.PersistentChatClient); ok {
 snapshot, err := pc.Save()
 // store the snapshot somewhere
}

A snapshot is a serialisable value that captures the conversation. You store it. Next run, you load it, Restore it onto a fresh client, re-register your tools, and call Chat again. “Where were we?” works, because the model is handed back the whole history.

A snapshot is opinionated about what it carries

The interesting part is what a snapshot does and doesn’t contain, because that’s a series of deliberate decisions.

It carries the messages, the system prompt, the model name, and tool metadata: the names, descriptions and parameter schemas of the tools that were registered.

It does not carry tool handlers. Handlers are code, not data; you can’t serialise a function meaningfully, so after a restore you re-register them with SetTools. The snapshot remembers that a tool called read_file existed and what its shape was; it doesn’t try to remember the Go function behind it.

And it does not carry API tokens. This is the one to dwell on. A snapshot is a file. A file gets synced, backed up, copied between machines, attached to a support ticket by a user trying to be helpful. A snapshot that carried the API key would be a credential leak the moment it left the laptop it was made on. So the snapshot never contains a token, at all. On restore, the client picks the credential up again the ordinary way, from the environment or the keychain. The conversation and the secret are kept in separate places on purpose, and only one of them is ever in the file.

Encrypted at rest, if you want it

The package ships a FileStore that writes snapshots as JSON files, with 0600 permissions in a 0700 directory, and it can encrypt them. Pass WithEncryption a 32-byte key and snapshots are written with AES-256-GCM.

That option exists because a conversation can hold sensitive content even when it holds no credential. The log a user pasted in for analysis, the source file they asked the model to review, the internal details tucked into their questions: none of that is an API key, and all of it might be something you’d rather not have sitting in plain JSON in a backup somewhere. Encryption at rest covers it.

The FileStore is also careful about the snapshot identifiers it’s handed. An ID has to be a canonical UUID, and the resolved file path is checked to lie inside the store directory, so a snapshot ID arriving from an untrusted source (a CLI flag, a request payload) can’t be bent into a path-traversal that reads or writes somewhere it shouldn’t. Persisting conversations adds a small filesystem surface, and the store treats it as exactly that.

The short version

A CLI tool forgets everything between invocations, which is correct for most commands and wrong for an AI conversation, because a conversation is its history.

go-tool-base’s chat package lets you persist one. PersistentChatClient saves a snapshot you can store and restore later, picking the conversation back up where it ended. The snapshot is deliberate about its contents: messages, system prompt and tool metadata yes; tool handlers no, because they’re code you re-register; API tokens never, because a snapshot is a file and a file travels. The built-in FileStore can encrypt snapshots at rest with AES-256-GCM and validates snapshot IDs against path traversal. Resumable conversations, without the conversation file turning into a place secrets leak from.

An AI agent that has to make the build pass

Thu, 02 Apr 2026 00:00:00 +0000

Most AI code generation works on a charming little principle I’ll call generate-and-hope. The model writes the code, the model stops at the closing brace, and whether the thing actually compiles is left as an exercise for you. For a snippet you paste into an editor, fine. For a whole generated command, that’s just outsourcing the disappointment.

go-tool-base does something I’m rather happier with: the AI has to make the build pass before it’s allowed to claim it’s done.

Generate and hope

The usual shape of AI code generation is this. You ask for code, the model produces it, and the model’s job ends at the closing brace. Whether it compiles, whether the tests pass, whether the imports even resolve, none of that has been checked. The model produced something that looks right. You find out whether it is right when you build it.

For a snippet you paste into an editor, that’s perfectly fine. The compiler tells you in a second. But go-tool-base’s generator, driven by gtb generate command --script or --prompt, produces a whole command: the implementation, its tests, the lot. “Generate and hope” at that scale means handing the user a project that may or may not build, and quietly making them the one who finds out which.

Drafting is only step one

So the generator doesn’t stop at drafting. Writing the first version of the implementation and its tests is step one of two. Step two is an autonomous repair agent.

Once the draft is on the filesystem, a separate agent takes over. It’s an LLM running in a loop, but a loop aimed at one narrow, checkable job: make this project build and pass its tests. It isn’t asked to be creative. It’s asked to get to green.

A fixed set of tools, and no shell

The agent is not handed a shell. It’s given a fixed, defined set of tools and nothing else. Three of them let it explore and edit the project: list_dir, read_file, write_file. Four of them let it verify the project:

go_build runs the build and captures the compiler errors.
go_test runs the tests and captures the failures.
go_get resolves a missing dependency.
golangci_lint runs the project’s linter.

That restriction is the design, not a limitation of it. The agent can’t delete arbitrary files, can’t reach the network, can’t run anything that isn’t on the list. It has exactly what it needs to make code compile and nothing it would need to do damage. Its file writes are confined to the project directory by an explicit path check, so even write_file can’t go wandering up into /etc. A coding agent you’d actually let near a filesystem is one whose abilities are an allowlist, not a denylist. (I keep coming back to that principle through this series… safety as a boundary you draw, not a behaviour you hope for.)

The loop

The repair loop is a ReAct loop, the same reason-act-observe shape as the tool-calling loop, only this time pointed at a goal:

The draft is on disk.
Verify: run go_build and go_test.
If verification failed, read the error logs, the compiler error or the failing test.
Reason about the cause: an undefined variable, a missing import, a wrong signature.
Act: call write_file to patch the code, or go_get to add the dependency.
Loop. Steps two to five repeat until the project is green, or the agent hits its bounded step limit.

What makes this work is treating the error output as feedback rather than as a failure to log and walk away from. A compiler error is the single most useful sentence you can hand a model that’s trying to fix code. It says what’s wrong, and usually where. The loop feeds it straight back in, and the model fixes against it.

Verification changes what “done” means

Here’s the real shift, and the agent’s own documentation puts it well: the agent “doesn’t just say it fixed a bug; it uses a Test tool to verify the fix before reporting success.”

A generate-and-hope model reports success when it finishes writing. It has no idea whether the code works, and it isn’t really claiming otherwise. “Done” means “I produced text”. The repair agent reports success when go_build and go_test actually pass. “Done” means “the build is green”. Those are two completely different claims, and only the second is worth anything to the person who asked for the command.

That’s the line between an AI that’s a creative writer and an AI that’s a collaborator you can hand a task to. And when the agent can’t reach green, when it spends its whole step budget and the project is still broken, the generator fails safely: it leaves the best-attempt code in place, commented out so the project still compiles, and tells the user what to finish by hand. There’s also an --agentless flag for anyone who’d rather have a plain single-shot retry than the multi-step agent. The default, though, is the agent, because the default should be code that’s been checked.

Where this leaves us

Most AI code generation generates and hopes: the model writes code and the user discovers whether it works. For a whole generated command, that pushes a may-or-may-not-build project onto the user.

go-tool-base’s generator drafts the command and then hands it to an autonomous repair agent. The agent has a fixed set of tools (explore and edit the project, build it, test it, lint it, fetch dependencies) and no shell at all, with file writes confined to the project directory. It runs a ReAct loop, reading each error and patching against it, until the build is green or it exhausts its steps. The point is what “done” comes to mean: not “the model finished writing”, but “the build passes”. Only one of those is a claim worth trusting.

Stop regex-ing the LLM's prose

Tue, 31 Mar 2026 00:00:00 +0000

Ask an LLM a question and it hands you back prose. Lovely to read, miserable to program against. You wanted the one number buried in the middle of it, and now you’re writing a regular expression to fish a word out of three well-written paragraphs that phrase themselves slightly differently every single time you run them.

There’s a much better way, and it’s the difference between forever interpreting an LLM and actually building on one.

The problem with a paragraph

You ask an LLM to analyse a log file and tell you the severity of what it found and a suggested fix. It comes back with three well-written paragraphs. Somewhere in there is the word “critical”, and somewhere is the fix.

Your program now has to extract those two facts from prose, and prose has no contract. The next run, the model phrases it differently. It leads with a caveat. It says “severe” where last time it said “critical”. It puts the fix first. Anything that worked by finding “critical” in the text is now quietly wrong, and you didn’t change a line. Parsing free text for structured facts is a game you lose slowly.

What you actually wanted was never a paragraph. It was a value: a thing with a severity field and a fix field, that you can branch on and store and pass around like any other.

Ask for the struct, not the prose

go-tool-base’s chat package draws the line with two methods. Chat gives you text. Ask gives you a struct.

You define the Go type you want back:

type Analysis struct {
 Severity string `json:"severity"`
 Fix string `json:"fix"`
}

var result Analysis
err := client.Ask(ctx, "Analyse this log file: "+logText, &result)

The framework generates a JSON Schema from that struct, sends it to the model as the required response format, and unmarshals the reply straight into result. You never lay a finger on the prose. You get result.Severity and result.Fix, typed, ready to use. If you want the model’s answer to drive a switch statement, this is the method that lets it.

The struct is the schema is the contract

The detail that makes this hold up over time: you don’t write the schema. The struct is the schema.

The framework derives the JSON Schema from your type. In go-tool-base that’s GenerateSchema[T](); in rust-tool-base the schema comes from your Rust type through schemars. (Yes, there’s a Rust sibling now. I’ll introduce it properly in a few weeks, but it keeps gatecrashing these posts because the two frameworks deliberately share ideas.) Either way there’s one definition, your type, and the schema is just a projection of it.

That matters, because otherwise two things have to agree. There’s the schema you tell the model to obey, and there’s the type you unmarshal the answer into. Hand-write the schema and those two can drift: add a field to the struct, forget to add it to the schema, and the model is never told to produce it, so it silently never appears. Deriving the schema from the type collapses the two into one. They can’t disagree, because there’s only one of them.

Both frameworks, with one extra step in Rust

go-tool-base does this with Ask and a ResponseSchema set on the client config. rust-tool-base does it with chat_structured::<T>, where T is any type that’s both deserialisable and JsonSchema.

rust-tool-base adds one step worth calling out. Before it deserialises the model’s reply into your T, it validates the raw response against the schema with a JSON Schema validator. That splits the failure into two distinct, named cases: the response didn’t match the schema, or it matched the schema but still wouldn’t deserialise. A model that returns subtly wrong JSON fails loudly and specifically, with an error that tells you which of those happened, instead of quietly handing you a zero-valued struct that you end up debugging an hour later.

When you’d reach for it

The line is simple, and it’s about who reads the answer.

If a human reads the answer, prose is right. Chat, free text, let the model write well. A summary, an explanation, an interactive reply: leave all of those as prose.

If a program consumes the answer, you want a value. Classification, extraction, a code review scored out of a hundred with a list of issues, a yes-or-no with reasons: anything where the next thing that happens is your code branching on the result. There, Ask and chat_structured turn the LLM from something you have to interpret into something that returns a value, and a typed value is a thing you can actually build on.

To sum up

An LLM returns prose by default, and prose has no contract, so a program that picks structured facts out of it breaks the moment the model rephrases.

Structured output asks for the value instead. You define a struct, the framework derives a JSON Schema from it, the model is constrained to that shape, and you get a typed result. go-tool-base’s Ask and rust-tool-base’s chat_structured both work this way, with the schema derived from your type so the schema and the type can’t drift; rust-tool-base additionally validates the response against the schema before deserialising. Use it whenever the answer feeds code rather than a human. It’s one of the four methods that make up go-tool-base’s small chat interface, and it’s the one that makes an LLM safe to program against.

Telemetry that asks first

Mon, 30 Mar 2026 00:00:00 +0000

Usage telemetry is genuinely useful. Knowing which commands people actually run, where the errors cluster, whether anyone ever touched the feature you spent a fortnight on… that’s the stuff that makes you a better maintainer. Wanting it is completely legitimate.

The trouble is that the usual way of getting it, on by default and quietly hoovering up everything, is a small betrayal of the people who installed your tool to get a job done. I wasn’t willing to build that, so go-tool-base’s telemetry starts from a different question.

The data you want, and the line you shouldn’t cross

If you maintain a tool, you want to know how it’s actually used. Which commands matter and which are dead weight. Where the error rate spikes. Whether anyone touched the feature you spent that fortnight on. That information makes you a better maintainer, and, to say it again, wanting it is completely legitimate.

The trouble is the standard way of getting it. Telemetry on by default. An opt-out buried three levels down in a settings file nobody reads. And once it’s running, it quietly collects far more than it ever admitted to: the arguments people passed, the paths they were working in, an IP address for good measure.

Every one of those is a small betrayal of someone who installed your tool to get a job done, not to become a data point. And the cost when users notice isn’t a slap on the wrist. It’s trust, and trust in a developer tool does not grow back quickly. A tool that surprises you once with what it was quietly collecting is a tool you uninstall and warn your colleagues about.

So go-tool-base’s telemetry started from a different question. Not “how do we collect the most data” but “how do we collect useful data without ever putting the user in a position they didn’t choose”.

Rule one: it is off until you say otherwise

The foundation is the simplest possible rule, and it’s absolute. Telemetry is never enabled by default. A freshly installed tool built on go-tool-base sends nothing. Not a heartbeat, not a ping, nothing at all.

It only starts collecting when the user makes an explicit, visible choice to let it. Three honest doors: they run telemetry enable, they say yes to a clear prompt during init, or they set TELEMETRY_ENABLED themselves. All three are deliberate acts. None of them is a pre-ticked box or a default they have to discover and then undo.

This is opt-in, and the distinction from a well-hidden opt-out is the entire point. Opt-out telemetry treats consent as something to be assumed and grudgingly reversed. Opt-in treats it as something that has to be given. Only one of those is actually consent.

Rule two: no personally identifiable information, full stop

Consent to “some telemetry” is not consent to “any telemetry”, so the second rule constrains what can ever be collected, even from a user who’s opted in.

No personally identifiable information. The framework does not record command arguments (they routinely contain paths, hostnames, the occasional secret someone’s pasted in). It does not record file contents. It does not record IP addresses.

It does need some notion of “distinct installations” for the numbers to mean anything, so it derives a machine ID from a handful of system signals and runs it through SHA-256. What leaves the machine is a hash. It tells you “this is the same install as last week” and tells you precisely nothing about whose install it is, and the hash can’t be walked backwards into the signals it came from.

The events themselves are deliberately thin. Which command ran, roughly how long it took, whether it errored. The shape of usage, not a transcript of it.

Rule three: the author picks the destination

Even with consent given and PII excluded, there’s a third question: where does the data actually go? go-tool-base doesn’t answer that for you, because it can’t. A corporate internal tool, an open-source CLI and an air-gapped utility have completely different right answers.

So the backend is the tool author’s choice. The framework ships several (a noop backend, stdout, a file, plain HTTP, and OpenTelemetry over OTLP) and supports custom ones. The noop backend matters more than it looks: it lets a tool wire up the whole telemetry surface, commands and all, while sending data precisely nowhere. A perfectly reasonable, fully supported configuration.

Pluggable backends also mean the data never has to touch any infrastructure I run. It goes where the tool’s author decides, on their terms. The framework provides the plumbing and stays well out of the destination.

And a way back out

One last thing, because it’s the part that makes the opt-in real rather than decorative. A user who opted in can opt straight back out, and the package includes a GDPR-aligned deletion path, so “stop, and remove what you have” is an actual supported request rather than a polite fiction.

Consent you can’t withdraw isn’t consent. It’s a one-way door with a friendly sign on it. The deletion path is what keeps the front door an actual door.

The bottom line

Telemetry is genuinely useful to a maintainer and genuinely dangerous to the trust of the people running the tool, and the usual implementation (on by default, opt-out buried, collecting everything) spends that trust recklessly. go-tool-base’s telemetry holds three lines: never enabled without an explicit user action, never collecting personally identifiable information even once enabled, and always sending data to a destination the tool’s author chose, up to and including nowhere. A real deletion path makes the opt-in something you can take back.

You can have your usage numbers. You just have to ask for them, the way you would for anything else that wasn’t yours to begin with.

Letting the AI call your Go functions

Sun, 29 Mar 2026 00:00:00 +0000

An AI that can only produce text can describe your system. An AI that can call your Go functions can actually operate it. That gap, between describing and doing, is the difference between a chatbot and something genuinely useful, and crossing it comes down to one fiddly mechanism: tool-calling, and the loop that drives it.

Talking about the system versus operating it

Wire an AI provider into a CLI command and you get something that can talk. Ask it a question, get a paragraph back. Useful, up to a point.

But notice the ceiling. An AI that can only generate text can describe things. It can tell you what it would do. What it can’t do is look at the actual current state of your system, or take a real action, because it has no hands. It’s reasoning in a vacuum about a world it can’t reach out and touch.

The thing that gives it hands is tool-calling. You hand the AI a set of functions it’s allowed to call. Now, mid-conversation, it can decide it needs to read that file before it can answer, or run that query, or check that status, and actually go and do it, and then reason about the real result. The AI stops describing your system and starts operating it.

The loop is the hard part

Tool-calling has a shape, and the shape is a loop. The literature calls it ReAct: Reason, Act, Observe.

The AI reasons about the prompt and decides whether it needs a tool.
If it does, it acts, asking for a specific tool with specific arguments.
Your code runs the tool and feeds the result back. The AI observes that result.
Round again. Reason about the new information, maybe call another tool, maybe several. Keep going until the AI has what it needs and produces a final text answer with no more tool calls.

Conceptually simple. Tedious and error-prone to implement by hand every single time: parsing the model’s tool-call requests, dispatching to the right function, marshalling arguments in and results out, feeding observations back in the exact format the provider expects, knowing when to stop, and not looping forever if the model gets itself stuck.

That orchestration is pure plumbing, and it’s identical for every tool and every command. So you can probably guess what’s coming: go-tool-base’s chat package owns it. You don’t write the loop. You write the tools.

Defining a tool

A chat.Tool is four things: a name, a description, a parameter schema, and a handler. The description is what the AI reads to decide whether to use the tool, so it’s worth writing well. The schema describes the arguments, and you don’t hand-write it. You write a tagged Go struct and let it generate:

type ReadFileParams struct {
 Path string `json:"path" jsonschema_description:"Relative path to the file"`
}

The struct is the contract. The framework derives the JSON Schema the AI is given straight from those tags, so the schema and the Go type the handler receives can’t drift apart, because they share a single source. The handler is then just an ordinary Go function that takes those parameters and returns a result.

You register your tools with SetTools, call Chat, and that’s the whole of your involvement. The framework runs the ReAct loop and Chat returns the AI’s final text answer once the loop settles.

Two details that show it was built for real use

A couple of decisions in the loop tell you it’s meant for production, not a demo.

Tool errors don’t abort the conversation. When a handler returns an error, the framework doesn’t crash the loop. It hands the error back to the AI as a string, as just another observation. That’s deliberate, and it’s right. A real agent should be able to call a tool, watch it fail, and react: try different arguments, take a different route, or tell the user it couldn’t manage it. A loop that aborted on the first tool error would be far more brittle than the model driving it.

The loop is bounded. There’s a MaxSteps limit, default 20. An AI that gets confused could otherwise call tools forever, and a CLI command that never returns is a worse failure than a wrong answer. The cap guarantees the command terminates. The agent gets room to genuinely work a problem across many steps, but not infinite room to flail about in.

There’s also parallel tool execution: when the model asks for several tools in a single step (three independent file reads, say) the framework runs them concurrently rather than one after another, because there’s no reason to make the AI sit and wait out a sequence of things that don’t depend on each other.

Boiling it down

A text-only AI can describe your system; an AI that can call your functions can operate it. Bridging that gap means tool-calling, and tool-calling means the ReAct loop (reason, act, observe, repeat) whose orchestration is fiddly, identical every time, and not a problem worth solving twice.

go-tool-base’s chat package runs the loop for you. You define chat.Tool values (name, description, a tagged parameter struct that generates its own schema, a handler), call SetTools and Chat, and get the final answer. Tool errors go back to the AI as observations so it can recover, and a MaxSteps cap guarantees the command always terminates. You write Go functions. The framework turns them into things an agent can reach for.

Nobody reads the manual

Sun, 29 Mar 2026 00:00:00 +0000

Let me describe the actual lifecycle of a user meeting your CLI tool, because it’s a bit humbling. They run it. It doesn’t quite do what they expected. They run it again with --help. They get a wall of monospaced flag descriptions, skim it, don’t find the thing they wanted, and either give up or go and ask a human who already knows.

Your documentation might be magnificent. It doesn’t matter, because the user never reached it.

The manual loses on location, not quality

That’s the lifecycle, and notice exactly where it breaks. The documentation might be excellent. It might answer their precise question in full. It doesn’t matter, because it’s on a website, in another window, behind a search box, and the user is here, in the terminal, mid-task. The docs lost not on quality but on location. They simply weren’t where the work was.

go-tool-base’s answer starts with a decision about location: the documentation gets embedded into the binary itself. Your docs/ folder ships inside the tool, the same way its default config does. Wherever the tool is installed, the docs are right there alongside it, no network, no browser. That embedding is what makes everything else possible, and there are two things built on top of it.

A browser, in the terminal

The first is the docs command, and it’s not --help with extra steps. It launches a proper Terminal User Interface, built on Bubble Tea.

It has a sidebar, structured from the project’s own mkdocs.yml, so the docs are a navigable tree rather than one flat scroll. Markdown renders with real formatting through Glamour (colour, tables, lists, headings) instead of collapsing into monospaced soup. There’s live search across every page, regex included.

Compared with man and --help, the difference isn’t a nicer coat of paint. man gives you linear scrolling and grep; this gives you a structured tree, rich rendering and real search. It’s the documentation experience a modern developer expects, except it followed the tool into the terminal instead of demanding the user leave it.

A documentation assistant that won’t make things up

The second thing built on the embedded docs is the one I find genuinely transformative: docs ask.

The user doesn’t navigate anything. They just ask:

mytool docs ask "how do I point this at a self-hosted server?"

and get a direct, specific answer. Under the hood, the framework collates the tool’s embedded markdown and hands it to the configured AI provider (Claude, OpenAI, Gemini, Claude Local, any OpenAI-compatible endpoint) as the context for the question.

Now, “an AI answers questions about my tool” should immediately make you nervous, and the correct thing to be nervous about is hallucination. An AI that confidently invents a flag that doesn’t exist, or describes behaviour the tool simply doesn’t have, is worse than no assistant at all, because the user trusts it.

This is where embedding the docs pays off a second time, and it’s why I keep stressing that the corpus is closed. The model is instructed to answer only from the tool’s actual documentation, and the context it’s handed is exactly that documentation and nothing else. It isn’t drawing on a vague memory of similar tools from its training data. It’s answering from this tool’s real, shipped, version-matched docs. The corpus is small, closed and authoritative, which is the combination that keeps the answers honest. “Zero hallucination by design” isn’t a slogan about the model. It’s a property of bounding what the model is allowed to look at, which is the same instinct I leaned on with the mcp command: the safety comes from the boundary you drew, not from trusting the AI to behave itself.

There’s a nice second-order effect, too. The answer is always about the version of the tool the user actually has, because the docs were embedded into that build. No mismatch between a website documenting the latest release and the slightly older binary sitting on the user’s machine.

The upshot

Documentation usually loses to --help not on quality but on location: it’s in a browser, and the user is in the terminal. go-tool-base embeds the docs into the binary and surfaces them two ways: a docs command that’s a real TUI browser with a sidebar, rich markdown and search, and docs ask, which answers natural-language questions using the embedded docs as context.

Because that context is the tool’s own closed, shipped documentation and the model is told to use nothing else, the assistant stays grounded, and it’s always describing the exact version the user is holding. The fix for unread documentation was never to write more of it. It was to put it where the work happens and let it answer back.

BDD where it earns its place, and nowhere else

Sat, 28 Mar 2026 00:00:00 +0000

I have a slightly complicated relationship with BDD. I’ve watched it turn a tangled test suite into something the whole team could read and reason about, and I’ve watched it turn a perfectly good unit test into a paragraph of ceremonial English that nobody benefits from. So when go-tool-base brought in Cucumber-style BDD, the interesting decision wasn’t adopting it. It was being ruthless about where not to.

Two tests that hurt for different reasons

Most of go-tool-base’s tests are ordinary table-driven Go tests, and they’re absolutely fine. A function, a slice of input/expected pairs, a loop. Nobody needs Gherkin to understand a parser test.

But two areas were genuinely painful, and they were painful in the same way: the test had become harder to understand than the thing it was testing.

The first was pkg/controls, the service-lifecycle package. It runs a small state machine (Unknown, Running, Stopping, Stopped) with signal handling, health monitoring, restart policies and graceful shutdown all woven through it. The integration tests for graceful shutdown had grown to over three hundred lines of imperative goroutine and channel coordination. They worked. But reviewing them was a slog, and a test you can’t review with confidence is a test you can’t trust when it fails. The behaviour being checked, “when a shutdown signal arrives mid-startup, the controller stops cleanly”, was a simple sentence buried under a heap of synchronisation scaffolding.

The second was the CLI itself. init, update, doctor are user workflows. “Given a config file with a custom value, when I run init, then the custom value survives the merge.” That’s already a Given/When/Then; it just happened to be written out as Go.

Godog, and the line I drew

Godog is the official Go implementation of Cucumber. You write .feature files in plain Gherkin and bind each step to a Go function. The shutdown scenario stops being three hundred lines of channels and becomes this:

Scenario: graceful shutdown completes within the deadline
 Given a controller with two registered services
 When a shutdown signal is received
 Then both services stop in registration order
 And the controller reports a clean shutdown

The goroutine choreography doesn’t vanish, of course. It moves into the step definitions, written once and reused. What changes is that the scenario is now readable by someone who’s never opened the file before, including someone from an ops team who’ll never write a line of Go but absolutely has opinions about how shutdown should behave.

Here’s the part I want to dwell on, because it’s the part most BDD adoptions get wrong. The first design decision written down for this work was: strategic, not universal. Use Godog only where BDD adds clarity. Keep table-driven Go tests as the baseline everywhere else.

That sounds obvious written down. It is not obvious in practice, because BDD has a gravitational pull. Once a team has feature files, there’s a powerful urge to express everything as feature files, for consistency. And that’s how you end up with Gherkin scenarios for a pure function (Given the number 2, When I double it, Then I get 4) which is pure ceremony. You’ve wrapped a one-line table test in a paragraph of English and a step-definition indirection, and made it actively worse.

The test for whether BDD belongs is this: is this test a narrative, or is it a matrix?

A matrix is the same logic with many input/output pairs. That’s a table-driven test, that’s most unit tests, and Gherkin actively harms them. A narrative is a sequence of steps where the ordering and the state between steps is the thing under test, and that’s where Gherkin pays for itself. Lifecycle transitions are narratives. A user running three commands in sequence is a narrative. Doubling a number is not.

go-tool-base drew that line and stuck to it. Feature files live in features/ at the project root, where a non-Go developer can find and read them. Step definitions live in test/e2e/, kept well away from the unit tests. And the unit tests stayed exactly what they were, because they were already the right tool.

Made to fit, not bolted on

A couple of smaller decisions kept the BDD layer from feeling like a foreign object.

It runs under go test. There’s no separate Cucumber runner to install or remember. A godog.TestSuite is invoked from an ordinary TestFeatures(t *testing.T), so the BDD scenarios run in the same go test ./... as everything else. CI didn’t need a new concept bolted onto it.

And the CLI end-to-end tests build the gtb binary once and reuse it across every scenario. Compiling a binary per scenario would make the suite slow enough that people would quietly start skipping it, and a test suite people skip is just decoration. Build once, test many.

Stepping back

go-tool-base brought in Godog for BDD, but the decision worth writing about is the restraint. BDD was applied to exactly two things: the service-lifecycle state machine, where a 300-line goroutine tangle became a four-line scenario anyone can review, and CLI workflows, which are Given/When/Then by their very nature. Everywhere else, table-driven Go tests remained the baseline, because wrapping a matrix test in Gherkin makes it worse, not better.

The useful rule: BDD fits a narrative, ordered steps with meaningful state in between, and fights a matrix. Adopt it as a scalpel for the narratives. Resist the pull to turn it into a religion.

An AI interface that fits on one screen

Fri, 27 Mar 2026 00:00:00 +0000

The moment you decide a CLI tool should talk to an LLM, there’s a strong gravitational pull towards reaching for LangChain, or one of its many relatives. It’s the obvious move. It’s also, for most CLI work, a bit like hiring a removals firm to carry a single box up the stairs.

Let me explain why go-tool-base went the other way, and what “the other way” actually looks like.

The instinct, and why it overshoots

When you add AI to a tool, the instinct is to reach for the big general-purpose framework. LangChain and its relatives are capable, and they exist for a real need: orchestrating complex multi-step AI applications, with retrieval pipelines, memory stores, chains of calls, whole fleets of agents.

Now look at what a CLI tool actually needs from an LLM. It needs to send a prompt and get text back. Sometimes it wants structured data back instead of prose. Sometimes it wants to let the model call a few of the tool’s own functions. That’s pretty much the whole list.

Pulling in a framework built to orchestrate retrieval and agent swarms in order to do that is a poor trade. You take on a large new vocabulary of concepts, a wide dependency surface, and a great deal of abstraction you’ll never touch, all to perform three or four operations. The framework isn’t wrong. It’s just answering a far bigger question than the one a CLI tool is asking.

What go-tool-base chose instead

go-tool-base didn’t reach for a framework. The decision is on the record in its own design notes: before a single line was written, LangChain Go, go-openai, Vercel’s AI SDK and around ten other options were evaluated, and not one of them matched what a CLI framework actually needs. So the chat package was built deliberately small.

How small? The entire core ChatClient interface is four methods:

type ChatClient interface {
 Add(ctx context.Context, prompt string) error
 Chat(ctx context.Context, prompt string) (string, error)
 Ask(ctx context.Context, question string, target any) error
 SetTools(tools []Tool) error
}

Add appends a message to the conversation. Chat sends a prompt and returns text. Ask sends a prompt and returns a typed Go struct, the model’s answer unmarshalled straight into a value you defined. SetTools hands the model a set of your own functions it’s allowed to call. That’s the whole surface. Downstream code that uses AI never holds anything larger than this, and never has to know which provider is behind it.

The package’s own documentation has a word for this: right-sized. Large enough to solve genuine provider-abstraction complexity, small enough that the full interface fits on a single screen.

“Thin” is not the same as “does little”

This is the part worth being precise about, because “four methods” can sound like “barely does anything”, and that’s the wrong read entirely.

Behind those four methods sits genuinely awkward work. Five providers (OpenAI, Claude, Gemini, a locally installed claude binary, and any OpenAI-compatible endpoint) each with a different wire API, all normalised behind the one interface. A tool-calling loop. Structured output via JSON Schema, made to behave consistently across providers that each express it differently. Error normalisation. Token chunking.

The point of a thin abstraction is not that there’s little underneath it. It’s that the interface stays small while the implementation quietly absorbs the complexity. Four methods on the surface; five provider integrations and a tool-calling loop below the waterline. The thinness is a property of what the caller sees, not of what the package does. A reach-for-LangChain decision gets that backwards: it exposes the caller to all the machinery, whether or not the caller will ever need it.

The core stays small even as features grow

There’s a neat detail in how chat keeps the interface from creeping. The package also supports streaming responses and conversation persistence, both of which are real features with real surface area. Neither of them is in the four-method core.

Instead they’re separate, optional interfaces. A streaming-capable client also satisfies StreamingChatClient; a persistable one also satisfies PersistentChatClient. Code that wants those capabilities does a type assertion to ask for them, and code that doesn’t simply never sees them. So the common path stays four methods forever. New capabilities arrive as opt-in interfaces alongside the core, not as new methods bolted onto it. The thing that fits on one screen keeps fitting on one screen.

Extensible without forking, testable without a network

Two more properties keep the package small without making it limiting.

It’s extensible. The provider list isn’t closed. A RegisterProvider call lets any package contribute a new provider, and chat.New will route to it. You add a backend without forking pkg/chat or sending a patch upstream.

And it’s testable. The package ships generated mocks. A downstream tool’s AI features can be tested against a mock ChatClient returning canned responses, with no network, no API key, and no flakiness. Because the interface is four methods, that mock is trivial to set up and complete by construction. A sprawling framework interface is a sprawling thing to fake; a four-method one is not. (I’ll come back to testing AI code properly in a later post, because it deserves a whole article of its own.)

The right size

When a CLI tool needs AI, the instinct is a large framework like LangChain. For orchestrating retrieval pipelines and agent swarms, that’s exactly the right tool. For sending a prompt, getting a struct back, and letting the model call a few functions, it’s enormous overkill.

go-tool-base’s chat package is the deliberate alternative, chosen only after LangChain Go and a dozen others were weighed up and rejected. Its core ChatClient interface is four methods. Underneath sit five normalised providers, a tool-calling loop, structured output and error handling, but the caller sees four methods and never learns which provider is active. Streaming and persistence are opt-in interfaces beside the core, not additions to it. It extends without forking and tests without a network. Right-sized: the complexity is real, but it lives under the interface rather than in it.

The config key that quietly did nothing

Fri, 27 Mar 2026 00:00:00 +0000

I once spent the better part of an hour convinced a timeout setting was broken. I’d set it in the config file, the tool ignored it, and the code that read it looked perfectly correct. The setting was tiemout. I’d typed it wrong, and not one thing in the entire stack had thought that worth mentioning.

Config loaders are too polite

Most config loaders have the same agreeable flaw: they’ll read whatever’s in the file and quietly ignore anything they weren’t expecting. Put a key the tool doesn’t know about and it sails straight past. No error, no warning, nothing. The loader assumes you meant it, or assumes some other layer will care, and neither turns out to be true.

That politeness costs you in two directions. A key you misspelled is silently dropped, so the setting you thought you’d changed keeps running on its old value. And a key you forgot leaves the field at its zero value, which you then discover at runtime, usually at the least convenient moment, when something downstream divides by a timeout of zero. The file looked fine. It parsed fine. It was just quietly wrong, and nothing was watching for that.

The struct already knows the answer

The thing is, the program already has a complete description of what valid config looks like. It’s the struct you unmarshal into. The field names, the types, which ones matter. That description exists; it’s just not being used to check anything.

go-tool-base’s config package puts it to work. You hand it a tagged struct and it derives a schema from the tags, in pkg/config/schema.go:

// WithStructSchema derives a schema from a tagged Go struct.
// Supported tags: `config:"key" validate:"required" enum:"a,b,c" default:"value"`.
func WithStructSchema(v any) SchemaOption { ... }

So a feature’s config type carries its own rules inline:

type ServerConfig struct {
 Host string `config:"host" validate:"required"`
 Port int `config:"port" validate:"required"`
 LogMode string `config:"log_mode" enum:"text,json"`
}

There’s no second artefact to keep in sync, which is the same instinct go-tool-base leans on for structured AI output: the type is the schema, and the schema is a projection of the type, so the two can’t drift apart because there’s only one of them. Each package describes its own slice of config on its own struct, and NewSchema composes them into the schema the loaded config gets checked against.

Strict mode turns the typo into an error

Deriving the schema is half of it. The half that actually catches tiemout is this one, also from schema.go:

// WithStrictMode treats unknown keys as errors instead of warnings.
func WithStrictMode() SchemaOption { ... }

By default a key the schema doesn’t recognise is a warning: surfaced, but not fatal, which is the right call when a config file might legitimately carry extra keys for tools other than yours. Turn on strict mode and an unknown key becomes an error. tiemout isn’t in the schema, so the tool refuses to start and tells me which key it didn’t recognise, instead of shrugging and using the default for an hour while I lose my mind. The validator walks every key actually present in the file and checks it against the known set, so a typo has nowhere to hide.

What it deliberately doesn’t do

There’s one decision in here I think is worth calling out, because the obvious feature is conspicuously absent. The schema knows each field’s default value. It would be the easiest thing in the world to have validation fill in missing fields from those defaults.

It doesn’t, on purpose. Validation validates. It tells you what’s wrong and what to do about it, and it stops there. Defaults are a separate job, handled by the embedded default config that every feature ships and merges in before validation ever runs. Keeping the two apart means the validator has exactly one responsibility, and the defaults live in one place rather than being half in an embedded file and half injected by a check. A field’s default tag is there for the documentation and the error hint, not as a sneaky second source of values.

Errors you can act on

The output isn’t a bare boolean. Validation returns a result that separates the fatal from the advisory: the missing required field and the wrong type are errors that stop the tool; the unrecognised-but-harmless key is a warning that informs you without blocking. And because each problem carries the offending key by name and a hint about the fix, the message tells you what to change, in the spirit of errors that tell you what to do next.

The short version

A config loader that silently ignores keys it doesn’t recognise will, sooner or later, ignore one you meant. go-tool-base derives a validation schema straight from your tagged config struct, so there’s no separate schema to maintain, and strict mode promotes an unknown key from a quiet shrug to a real error that names the typo. It validates without injecting defaults, because defaults are the embedded config’s job and a validator with one responsibility is easier to trust. Set tiemout now and the tool tells you, which is roughly fifty-nine minutes sooner than I found out.

One variadic, and I'd already spent it

Thu, 26 Mar 2026 00:00:00 +0000

I had a constructor I was rather pleased with. Hand go-tool-base’s root command its props and as many sub-commands as you like, and off it goes. Then I needed to thread some config file paths through it, reached for the obvious “just add a parameter,” and discovered I’d already spent my one variadic with no second one going spare.

The ergonomics I’d happily bought

NewCmdRoot ends in subcommands ...*cobra.Command. That trailing ... is a small luxury: callers write NewCmdRoot(props, build, deploy, status) and never have to think about slices. Variadics are lovely for exactly this, the “and as many of these as you fancy” argument.

The parameter I couldn’t add

Then config arrived, and the root command needed to know about some extra configuration file paths. The instinct is to add a parameter. The instinct is wrong, because there’s nowhere legal to put it.

You can’t write NewCmdRoot(props, configPaths ...string, subcommands ...*cobra.Command). Go allows a function exactly one variadic, and it must be the final parameter. Two variadics results in a compile error before you’ve finished the line (assuming your IDE does compile time checks for you), and fairly so: at the call site, how would Go ever know where the strings stopped and the commands began? So the variadic I’d spent on sub-commands was spent. There wasn’t another to hand.

The choices, and the one I took

You can demote the variadic. Make it subcommands []*cobra.Command and you’re free to add configPaths []string next to it. Correct, and it breaks every existing call: NewCmdRoot(props, build, deploy) becomes NewCmdRoot(props, []string{}, []*cobra.Command{build, deploy}). Uglier at every site, to solve a problem only some callers have.

You can reach for functional options, and for plenty of go-tool-base’s constructors that is exactly what happened. But the root builder is the one everybody calls first, with the simplest signature in the codebase, and I didn’t want the common case lugging option machinery around for the sake of the rare one.

What I actually did was add a second door. From pkg/cmd/root/root.go:

// NewCmdRoot creates the root command with Props wiring and optional subcommands.
func NewCmdRoot(props *p.Props, subcommands ...*cobra.Command) *cobra.Command {
	return NewCmdRootWithConfig(props, []string{}, subcommands...)
}

func NewCmdRootWithConfig(props *p.Props, configPaths []string, subcommands ...*cobra.Command) *cobra.Command {
	// ...
}

The new argument goes in as a plain []string, sat before the variadic, which is perfectly legal: one variadic, still last. Callers who care about config use NewCmdRootWithConfig explicitly, and NewCmdRoot becomes a one-line wrapper that delegates with an empty slice, so every existing caller compiles untouched and none the wiser. Two doors into the same room, granted, but the original door is exactly where everyone left it.

What it comes down to

A trailing variadic is a slot you fill once. It buys gorgeous ergonomics for the “as many as you like” argument, and in exchange it quietly forecloses on ever appending another parameter, because the next one has nowhere to stand. Once it’s spent, new arguments come in as ordinary parameters before the variadic, and the kind thing to do for your callers is to put that behind a second constructor and let the original keep delegating.

So spend the variadic deliberately. Give it to the argument that genuinely wants to be a loose list, not the first one that happens to be plural, because you don’t get a second.

Half your users don't have eyes

Wed, 25 Mar 2026 00:00:00 +0000

Run a command in your favourite CLI tool and look at what comes back. Colour. Neatly aligned columns. A friendly little summary sentence. Lovely… if you happen to be a human with eyes.

But a good half of any tool’s users aren’t people at all. They’re scripts, CI pipelines, bits of automation. And that pretty output you’re so proud of is, to them, actively hostile.

Your tool has two audiences and only serves one

I made more or less this same point about AI assistants when I argued that your CLI is already an AI tool. The machines are users too. Here it isn’t an AI doing the calling, it’s a humble shell script, but the principle is identical.

Run a CLI command and look at what comes back. Colour. Aligned columns. A friendly summary sentence. It’s designed for a person reading a terminal, and for a person reading a terminal it’s great.

Now picture the other half of your users. A deploy script that needs to know which version is installed. A CI job that runs doctor and wants to fail the build on one specific check. A bit of automation gluing your tool to three others. None of them have eyes. They have parsers.

So what do they do with your beautiful human output? They butcher it. They grep for a keyword, awk out the third field, sed off a prefix. It works in the demo. Then someone rewords a status line, or adds a column, or the colour codes shift, and every script downstream breaks at once. Silently, too, because a broken grep returns nothing rather than an error. You changed a sentence and quietly took out somebody’s pipeline without ever knowing.

The human-readable output was never the contract. It just got used as one, because it was the only output there was.

Give the machines their own channel

The fix is not to make the human output more parseable. That’s a trap. You’d be constraining prose meant for people in order to satisfy programs, and end up serving neither of them well. The fix is to give programs their own output format, declared and stable, kept well away from the prose.

So every command built with go-tool-base gets a --output flag. Leave it alone and you get the friendly human rendering. Pass --output json and you get something a parser can actually rely on.

And not just some JSON. JSON with a fixed shape.

One envelope, every command

The temptation with JSON output is to let each command emit whatever structure happens to suit it. Don’t. A consumer scripting against five of your commands then has to learn five shapes, and “where’s the actual payload?” has a different answer every single time.

go-tool-base wraps every command’s JSON in one standard Response envelope:

{
 "status": "success",
 "command": "deploy",
 "data": {
 "environment": "production",
 "version": "1.4.0",
 "replicas": 3
 }
}

status says how it went. command says what produced it. data holds the command-specific payload, and only the payload. Every built-in command (version, doctor, update, init) emits exactly this shape. So does every command you write, because pkg/output hands you the envelope rather than letting you freelance:

format, _ := cmd.Flags().GetString("output")
w := output.NewWriter(os.Stdout, output.Format(format))

return w.Write(output.Response{
 Status: output.StatusSuccess,
 Command: "deploy",
 Data: result,
})

The consumer-side payoff is the whole point. A script can check .status without ever touching .data. It can pull .data.version and know the field is there because it’s typed, not scraped. It learns the envelope once, and every command in your tool, and every tool built on the framework, honours it. The contract is explicit, versioned, and the same everywhere, which is precisely what the abused human output never was.

The human output gets to relax

There’s a quiet second benefit, and it’s my favourite kind: the sort you get for free. Once programs have their own reliable channel, the human output is freed. It no longer has to stay accidentally parseable. You can reword a status line, add colour, restructure a table, make it genuinely nicer to read, and not break a single script, because no script is reading it any more. They’re all over on --output json, where the real contract lives.

Two audiences, two formats, each one actually suited to its reader. That’s the deal a CLI tool ought to be offering, and most of them don’t.

In short

A CLI tool that only emits human-readable output is only half-built, because half its users are programs that end up grep-ing prose and shattering the moment that prose changes. go-tool-base gives every command a --output json flag and one standard Response envelope (status, command, data) used identically by every built-in command and by anything you write through pkg/output. Machines get a stable, explicit, learn-it-once contract; humans get output that’s now free to be properly readable, because nothing fragile depends on its wording any more.

If your tool will ever be called by another program (and it will), give that program a front door. Don’t make it climb in through the window.

Lifecycle management for when your CLI grows up into a service

Tue, 24 Mar 2026 00:00:00 +0000

There’s a moment in the life of a lot of CLI tools where they stop being a CLI tool. Nobody quite decides it. It just happens. Someone needs the thing to also expose a little HTTP endpoint, or poll a queue, or run a scheduler, so it grows a serve command… and the honest command-line utility you wrote is suddenly a long-running service wearing a CLI as a hat.

And a service needs a whole pile of production plumbing that a one-shot command never did.

The command that stops being a command

go-tool-base is CLI-first. It is not CLI-only, and the reason is a pattern I’ve watched play out more times than I can count.

A tool starts its life as an honest command-line utility. It runs, it does its thing, it exits. Then someone needs it to expose a small HTTP endpoint. Or poll a queue. Or run a scheduler. So it grows a serve command, or a run command, and the moment it does, the thing that was a CLI tool is now a long-running service that happens to have a CLI bolted on the front.

And a long-running service needs a whole category of plumbing a one-shot command never did. It has to start things up in a sensible order. It has to shut them down gracefully when someone sends a SIGTERM, finishing in-flight work rather than dropping it on the floor. It has to tell an orchestrator whether it’s alive, and whether it’s ready. It has to do something sensible when one of its internal services quietly falls over at 3am.

Hand-rolled, that’s a few hundred lines of goroutine choreography, channel-wrangling and signal handling that every such tool reinvents, slightly differently and slightly wrong each time. It’s the first-afternoon problem all over again, just turning up later in the project’s life. So go-tool-base ships it: pkg/controls.

A controller and the things it controls

The model is small. A Controller manages any number of services. You register each with Register(id, opts...) and describe it with functional options: WithStart takes a StartFunc, WithStop a StopFunc. An HTTP server, a background worker, a scheduler, anything with a “begin” and an “end”.

You register your services with the controller and it owns their collective lifecycle. They share a common set of channels (errors, OS signals, health, control messages) so the whole set can react together. A SIGTERM doesn’t get caught by one service off in a corner; it reaches the controller, and the controller takes everything down in order, each StopFunc handed a context with a deadline so that one sulking service can’t wedge the whole shutdown forever.

That ordering and timeout handling is the bit nobody enjoys writing and everybody needs. Centralising it means a tool that adds a second service later inherits correct coordinated shutdown for free, rather than discovering on its first production SIGTERM that it only half shuts down.

Probes, because something is usually watching

If the service ends up in Kubernetes (and a lot of them do) the orchestrator wants to ask two different questions, and they really are different questions.

Liveness: are you alive, or are you wedged and in need of a kill? Readiness: are you alive and able to take traffic right now? A service can quite easily be live but not ready… still warming a cache, still waiting on a dependency. Conflate the two and you get yourself killed during a slow startup, or sent traffic before you can actually serve it.

controls keeps them separate. You attach a WithLiveness probe and a WithReadiness probe to a service, each just a function returning a health report, and the controller exposes them. The tool answers Kubernetes honestly, in Kubernetes’ own terms, without you hand-wiring two more HTTP handlers.

Self-healing, but only if you ask

The last piece is what happens when a service fails. A worker’s StartFunc returns an error. Health checks start failing. In a hand-rolled setup this is where you either crash the whole process or write yourself a bespoke restart loop.

controls has a supervisor that can restart a failed service for you, and the important word in that sentence is can. It’s off by default. A service is only supervised if you hand it a RestartPolicy at registration:

controls.WithRestartPolicy(controls.RestartPolicy{
 MaxRestarts: 5,
 InitialBackoff: time.Second,
 MaxBackoff: 30 * time.Second,
 HealthFailureThreshold: 3,
})

With a policy in place, the controller restarts the service if its StartFunc errors out, or if it racks up more consecutive health-check failures than the threshold allows. Restarts back off exponentially, from InitialBackoff up to a MaxBackoff ceiling, so a service that’s failing because its database is down doesn’t sit there hammering that database flat with a tight restart loop. MaxRestarts caps the attempts, because a service that’s failed five times in a row is not going to be rescued by a sixth go, and at that point honest failure beats a thrashing pretence of health.

Opt-in matters here. Automatic restarts are exactly right for a resilient daemon and exactly wrong for a tool where a failure should stop the line and get a human’s attention. The framework doesn’t make that call for you. It gives you the supervisor and lets you point it at the services that genuinely want it.

The bottom line

A surprising number of CLI tools become long-running services the day they grow a serve command, and the day they do, they need coordinated startup, graceful ordered shutdown, real liveness and readiness probes, and a considered answer to a service falling over. That’s a few hundred lines of fiddly, easy-to-get-wrong plumbing.

pkg/controls provides it: a Controller over Controllable services with shared channels and deadline-bounded graceful shutdown, separate Kubernetes-style liveness and readiness probes, and an opt-in supervisor that restarts failed services with exponential backoff and a restart ceiling. Your tool can start as a command and grow into a daemon without that growth turning into a rewrite.

CLI-first, but not stuck there.

Middleware for CLI commands, not just web servers

Tue, 24 Mar 2026 00:00:00 +0000

Every CLI tool past a certain size grows a category of logic that doesn’t really belong to any one command, and yet has to happen for loads of them. Timing. An auth check. Panic recovery, so a crash becomes a clean error instead of a stack-trace all over someone’s terminal. A log line saying the command started and how it finished.

Web frameworks sorted this out years ago. CLIs, for some reason, mostly still copy-paste it around.

The logic that belongs to no single command

That category of logic doesn’t belong to any one command, yet needs to happen for many of them. Time how long the command took. Check the user is authenticated before a command that needs it. Recover from a panic so a crash becomes a clean error rather than a stack-trace vomited across the screen. Log that the command started and how it ended.

None of that is the command’s job. The deploy command’s job is to deploy. But timing and recovery and auth still have to happen around it, and around build, and around sync.

Put that logic inside each command’s RunE and you’ve copied the same six lines into thirty functions, which means thirty places to fix when the logging format changes and thirty chances to forget one of them. Cross-cutting concerns copied by hand don’t stay consistent. They drift, every time.

Web frameworks already solved this

This is not a new problem. It’s about the oldest problem in web frameworks, and they settled on an answer a long time ago: middleware. Gin has it, Echo has it, every HTTP stack you’ve ever touched has it. A middleware is a wrapper that sits around a handler, runs its cross-cutting logic, and calls through to the handler in the middle.

A CLI command is, structurally, just a handler too. So go-tool-base brings the same pattern to the Cobra command tree, with the same functional Chain shape:

type Middleware func(
 next func(cmd *cobra.Command, args []string) error,
) func(cmd *cobra.Command, args []string) error

A middleware receives the next handler in the chain and returns a new handler that wraps it. You compose a stack of them, and each command’s real RunE runs in the middle of the onion. Write the timing logic once, as one middleware, and every command in the chain is timed. Change the log format once and all thirty commands change with it, because there was only ever one copy. (The “write it once, in a place where everyone inherits it” drum again, which I will keep banging until the series runs out.)

“But Cobra already has PreRun”

It does, and this is the objection worth answering properly, because Cobra ships PersistentPreRun and PreRun hooks and they look, at a glance, like they cover this.

They don’t, and the reason is structural. A PreRun hook is a thing that happens before the command. That’s all it is. It can’t run anything after. It can’t wrap the command in a defer. It can’t catch a panic the command throws. It can’t measure how long the command took, because measuring a duration needs a start point and an end point, and the hook only owns the start.

A middleware wraps the entire execution. Because it’s a function that calls next() in its own body, it straddles the command (with the handler signature abbreviated to HandlerFunc here for readability):

func TimingMiddleware(next HandlerFunc) HandlerFunc {
 return func(cmd *cobra.Command, args []string) error {
 start := time.Now()
 err := next(cmd, args) // the command runs here
 log.Debug("command finished", "took", time.Since(start))
 return err
 }
}

Before, after, and around. A recovery middleware can put a defer recover() in place that a PreRun hook structurally cannot. An auth middleware can check a condition and return an error instead of calling next() at all, refusing to let the command run in the first place. PreRun can’t veto the command; it runs, and then the command runs regardless.

PreRun is a notification that the command is about to happen. Middleware is control over whether and how it happens. For genuine cross-cutting concerns you need the second thing, not the first.

To sum up

Timing, auth, recovery and logging are cross-cutting concerns: necessary for many commands, owned by none. Hand-copied into every RunE, they drift out of sync. Web frameworks fixed this with middleware years ago, and a CLI command is structurally just another handler.

go-tool-base brings the functional Chain middleware pattern to the Cobra command tree. A middleware wraps a command’s whole execution, so it acts before and after and can decide whether the command runs at all… strictly more than Cobra’s PreRun hooks, which only fire beforehand and can’t wrap, recover, time, or veto. Write the concern once, wrap the chain, and every command inherits it consistently.

A logging interface that doesn't leak its backend

Mon, 23 Mar 2026 00:00:00 +0000

The same tool, in two different lives, wants two completely different kinds of log.

On my laptop I want logs I can actually read: colour, alignment, friendly timestamps. The very same tool running as a daemon in a container wants none of that. It wants structured JSON, one object a line, ready for a log aggregator to swallow. And in a test I want the logger to shut up entirely. The interesting question is what it costs you to move between the three.

The same tool wants different logs

On a developer’s machine the tool is a CLI. You want logs that are pleasant to read in a terminal: colour, alignment, human-friendly timestamps. The charmbracelet logger does that beautifully.

Then the very same tool grows a serve command and gets deployed as a daemon in a container. Now coloured terminal output is worse than useless. The log aggregator wants structured JSON, one object per line, machine-parseable. slog does that.

And in tests you want neither. You want the logger to exist, satisfy the interface, and stay completely silent.

That’s three different logging backends, wanted by one tool across three different lives. So what does switching between them actually cost?

What it costs depends on what your packages imported

If your packages import a concrete logger, if pkg/config and pkg/setup and twenty others each have import "github.com/charmbracelet/log" and take a *log.Logger, then the backend is welded into the entire codebase. Switching to JSON for the container build means editing the import and the parameter type in every single one of those packages. The backend has leaked. A detail that should have been one decision has become a property of a hundred files.

go-tool-base doesn’t let it leak. Every package in the framework accepts a logger.Logger, an interface, and nothing else. No package anywhere imports a concrete logging library. A package states, in its types, “I need something I can log through”, and stops right there. It has no idea, and no way to find out, what’s actually on the other end.

// what every package depends on
type Logger interface {
 Debug(msg string, keyvals ...any)
 Info(msg string, keyvals ...any)
 Warn(msg string, keyvals ...any)
 Error(msg string, keyvals ...any)
 // ...
}

The backend gets chosen once, at the top, when the tool builds its Props. It travels down to every package as the interface, through the Props container. The packages underneath never see the concrete type, so the concrete type can change without a single one of them noticing. (There’s that “decide it once, in one place” theme again. I did warn you it runs through everything.)

Three backends, and the swap is one line

go-tool-base ships three implementations of that interface:

charmbracelet (logger.NewCharm(w, opts...)). Coloured, styled, for humans at a terminal. The CLI default.
slog JSON, a slog-backed backend emitting structured JSON, for daemons and containers feeding a log aggregator.
noop, which does precisely nothing, for tests that want a real Logger and total silence.

Switching the tool from a friendly CLI logger to container-ready JSON is a change to the one line in main() that constructs the logger. That’s the lot. pkg/config doesn’t change. pkg/setup doesn’t change. None of the twenty packages change, because none of them ever knew which backend they had. The decision was always one line; the interface is what kept it one line.

The noop backend deserves its own mention, because it’s the one people underrate. A test for a command shouldn’t be spraying log output all over the test run, but the command still needs a non-nil Logger to function. logger.NewNoop() gives you exactly that: interface satisfied, output binned, test quiet. And because it’s just another implementation of the same interface, no test needs any special logging machinery. It passes a different backend, exactly the way the container build does.

The general shape

There’s nothing exotic going on here. It’s “depend on interfaces, not implementations”, which every Go developer has had drilled into them at some point. The bit worth holding onto is where the rule actually pays out, and it’s at the seams between a stable core and a detail you know full well you’ll want to vary.

A logging backend is exactly such a detail. You will want it different in a terminal, in a container, and in a test. So the thing your code depends on has to be the interface, and the concrete backend has to be chosen at one well-known point and nowhere else. Get that boundary right and “we need JSON logs in production” is a one-line change. Get it wrong and it’s a refactor and a bad afternoon.

What it comes down to

One tool legitimately wants three different logging backends across its life: coloured output in a terminal, structured JSON in a container, silence in a test. The cost of moving between them is decided entirely by whether your packages imported a concrete logger or an interface.

go-tool-base’s packages depend only on logger.Logger, never a backend. Three implementations ship (charmbracelet, slog JSON, noop) and the backend is chosen once, in main(), then carried everywhere as the interface through Props. Switching is one line at the top, because the detail was never allowed to leak into the hundred files below it.

Errors that tell the user what to do next

Sun, 22 Mar 2026 00:00:00 +0000

Here’s an error message I’ve been on the receiving end of more times than I’d care to count:

error: failed to read config file

True. Also completely useless! I now know something is broken and I haven’t the faintest idea what to do about it. Which file? Why couldn’t it be read? Should I create it, run some init command, fix a permission, set an environment variable? The message states the problem and then abandons me at it, rather like a sat-nav cheerfully announcing “you have arrived” in the middle of a motorway.

A message is not a fix

The instinct, the moment you notice this, is to go and write a better message:

error: failed to read config file at ~/.config/mytool/config.yaml.
Run 'mytool init' to create one, or set MYTOOL_CONFIG to point at an existing file.

Better for the human, no question. But look at what you’ve just done to the error as a value. The recovery advice is now welded into the error string. Any code that wants to ask “is this the config-missing error?” is reduced to substring-matching English prose. Reword the advice and you break the check. So you’ve helped the user and quietly sabotaged the program at the same time, because you’ve made one poor little string do two completely incompatible jobs… being a stable identity for code, and being friendly guidance for people.

Why I changed error libraries

go-tool-base started out on github.com/go-errors/errors. It’s a perfectly fine library and it gave us stack traces. What it didn’t give us was any way to attach human guidance to an error without shoving it into the message string. So the codebase did exactly the daft thing I just described: multi-line suggestion text baked straight into errors.Errorf calls, user-facing content and programmatic identity all mashed into one value.

That’s the whole reason for the migration to github.com/cockroachdb/errors. Not novelty, and not because I fancied a weekend of find-and-replace. One specific capability: cockroachdb/errors lets you attach a hint to an error as a separate, structured field.

return errors.WithHint(
 errors.New("failed to read config file"),
 "Run 'mytool init' to create one, or set MYTOOL_CONFIG to point at an existing file.",
)

Now there are two things, cleanly apart. errors.New("failed to read config file") is the identity… stable, matchable, the program’s handle on the error. The hint is the guidance… for the human, and rewordable as much as you like without breaking a single check, because no check ever looks at it. errors.Is and errors.As work properly through every wrapper layer, so code matches on identity and never has to read prose.

The migration brought a few other things worth having. Stack traces print with a plain %+v instead of a type assertion. Errors can carry structured, machine-readable metadata. Multiple errors from concurrent work can be combined as a first-class value. But the hint is the one that actually changed the user’s day, because the hint is the recovery step, stored where it belongs.

One door out, and it knows where the help is

Separating the hint is only half of it. The other half is making sure those hints actually reach the user, every time, and that comes down to having a single way out.

Every go-tool-base command returns its errors the idiomatic Cobra way, through RunE. They all funnel into one Execute() wrapper at the root, which routes every error (runtime failure, flag parse error, pre-run failure) through one ErrorHandler. One door out. So error presentation gets decided in exactly one place, and no command can render an error differently from the command sat next to it.

And because there’s one handler, it can pull off something the individual commands never could. The framework knows your tool’s metadata, including its configured support channel, be it a Slack workspace or a Teams channel. So the error handler can finish a fatal error not just with the what and the recovery hint, but with where to go if the hint didn’t help:

error: failed to read config file
hint: Run 'mytool init' to create one, or set MYTOOL_CONFIG.
 Still stuck? Ask in #mytool-support on Slack.

The user is never left at a dead end. The error tells them what broke, the hint tells them the most likely fix, and if that’s still not enough the handler tells them which door to go and knock on. A failure becomes a signpost instead of a full stop.

The short version

An error that only reports what went wrong leaves the user stranded, and the obvious fix (writing the recovery advice into the message) quietly wrecks the error as a value, because now your code has to substring-match prose just to work out what it’s looking at.

go-tool-base moved from go-errors to cockroachdb/errors to get hints: a structured, separate field for human guidance that leaves the error’s identity clean for errors.Is and errors.As. Every command’s errors leave through one Execute() wrapper and one ErrorHandler, so presentation stays consistent, and because that handler knows the tool’s support channel it can point a stuck user at real help.

State the problem for the program. Give the fix to the human. And for pity’s sake, keep the two in different fields.

Many embedded filesystems, one merged view

Sat, 21 Mar 2026 00:00:00 +0000

Go’s embed package is one of those features that makes you slightly giddy the first time you use it. One //go:embed directive and your default config, your templates, your docs are all baked into the binary. The tool just works the moment it’s installed, with nothing external to lose or forget to ship.

And then you go and build something modular on top of it, and you discover the catch nobody warned you about.

`embed.FS` is an island

An embed.FS has a property that’s easy to miss until it bites: it’s local to the package that declared it. The //go:embed directive can only see files at or below its own source file. So in any project bigger than a toy, you don’t have an embedded filesystem. You have many. The root package embeds one. Each feature, each subcommand that ships its own templates or defaults, embeds another. They’re islands, one per package, and Go gives you no native way to make them behave as a whole.

For most files that’s perfectly fine. A feature’s templates can stay on the feature’s island; nothing else needs them.

It stops being fine the moment features need to contribute to something shared.

The shared-config problem

Here’s the case that forces the issue. A go-tool-base tool has a global config.yaml of defaults, embedded at the root. Now you add a feature, and that feature has its own configuration keys, with their own sensible defaults.

Where do those defaults go?

The naive answer is: edit the root config.yaml and add the feature’s section. And that’s a genuinely bad answer, because it inverts the dependency. The root config now has to know about every feature. Add a feature, edit the centre. Remove one, edit the centre again. The central file becomes a pinch point that every feature has to reach into, and a modular architecture where you can’t add a module without editing the core isn’t really modular at all… it just has more files.

What you actually want is for the feature to ship its own slice of default config, on its own island, and for the global config the tool reads to somehow already contain it. The feature contributes; the centre doesn’t budge.

`props.Assets`: merge the islands

That’s the job of props.Assets. (Yes, it lives on Props, the load-bearing container I keep going on about. Most of the good stuff does.) It’s a layer that implements the standard fs.FS interface, and into it you Register each embed.FS under a name:

// root main.go
Assets: props.NewAssets(props.AssetMap{"root": &assets}),

// a feature's command constructor
//go:embed assets/*
var assets embed.FS

func NewCmdFeature(p *props.Props) *cobra.Command {
 p.Assets.Register("feature", &assets)
 // ...
}

Now Props carries one Assets value that represents all the islands as a single filesystem. The root’s files and every registered feature’s files, addressable through one fs.FS. Each registration is named, so the islands stay individually identifiable, but they read as one.

That alone solves the addressing problem. The genuinely clever part is what happens for structured files.

Opening a file that exists in several places

When you Open a path through props.Assets and that path has a structured extension it recognises (.yaml, .yml, .json, .toml, .csv, and a few more) it doesn’t simply return the first match it stumbles across. It does this:

Discovery. It finds every instance of that path, across every registered filesystem.
Parsing. It unmarshals each one.
Merging. It deep-merges the parsed data, using mergo.
Re-serialisation. It hands you back a single fs.File whose contents are the combined, merged result.

So picture the shared-config problem again, only solved this time. The root ships a config.yaml with the base defaults. Each feature ships a config.yaml on its own island carrying only its own keys. Nobody edits anybody else’s file. When the init command opens config.yaml through props.Assets, it doesn’t get the root’s copy. It gets the deep-merge of the root’s copy and every registered feature’s copy: one config.yaml that contains every default in the tool, assembled at runtime from contributions that never knew about each other.

A feature contributes its defaults simply by existing and registering. The centre never changes. That’s the modular property the naive approach couldn’t give you, and it generalises well beyond config… the same merge applies to a shared commands.csv, or any structured file features want to add rows or keys to.

There’s also a Mount method for attaching an arbitrary fs.FS at a virtual path, which is handy for surfacing something external (a temp directory, say) as part of the same tree. But the structured merge is the feature that really earns Assets its place.

Boiling it down

embed.FS is per-package by design, so a modular CLI ends up with many embedded filesystems, one island per feature. Most of the time that’s fine. It fails specifically when features need to contribute to a shared resource like the global config.yaml, because the naive fix forces every feature to reach in and edit a central file.

props.Assets merges all the registered islands into a single fs.FS, and for structured files it goes further: opening a .yaml, .json or .csv discovers every copy across every island, deep-merges them, and returns the combined whole. A feature drops its own defaults onto its own island, registers, and the merged config the tool reads already includes them. Contribution without coupling, which is rather the whole point of being modular in the first place.

Props: the container that does the heavy lifting

Sat, 21 Mar 2026 00:00:00 +0000

I name-dropped Props back in the introduction and then rather glossed over it, which was a bit unfair of me, because it’s the single most important design decision in the whole framework. So let’s give it the attention it actually deserves.

And the best place to start, oddly enough, is the name.

Start with the name

The container at the centre of go-tool-base is called Props, and the name is doing real work, so we’ll start there.

It is not short for “properties”, though it does hold a few. A prop is the heavy timber or steel beam that stops a structure quietly collapsing in on itself. And for anyone who follows the rugby: a prop is the position in the scrum, the broad-shouldered forward whose entire job is to provide structural support so everyone else can get on with the game.

That’s the design brief, in a single word. Props is not where the clever, flashy work happens. It scores no tries. It’s the unglamorous, load-bearing thing that holds the framework up so that your actual command logic gets to be the interesting part. Understand the name and you understand what the struct is for.

What it carries

Props is the single object passed to every command constructor in a go-tool-base tool. It holds the dependencies a command might need:

Tool, metadata about the CLI (name, summary, release source).
Logger, the logging abstraction.
Config, the loaded configuration container.
FS, a filesystem abstraction (afero), so a command never touches the real disk directly.
Assets, the embedded-resource manager.
Version, build information.
ErrorHandler, the centralised error reporter.
Collector, the telemetry collector (always present, a no-op when telemetry is off).

A command constructor’s signature is, accordingly, boring on purpose:

func NewCmdExample(p *props.Props) *cobra.Command { ... }

One parameter. Everything the command could possibly need is reachable through it. No globals, no init()-time wiring, no twelve-argument constructor that quietly grows a thirteenth argument next month.

Why a struct, and not `context.Context`

Here’s the design decision I actually want to defend, because it’s the one Go developers tend to raise an eyebrow at. Go already has a well-known way to carry things through a call tree: context.Context. So why not just put the logger and the config in the context and pass that around?

Because context.Context carries its values as interface{}, and that’s the wrong trade for dependencies.

Pull a dependency out of a context and you get this:

l := ctx.Value("logger").(logger.Logger) // a runtime type assertion

That one line has two separate ways to hurt you. The key is a bare string, so a typo compiles perfectly happily and then fails at runtime. The type assertion is unchecked, so if the wrong thing is sitting under that key, your tool panics in front of a user. Neither failure is visible to the compiler. Neither is visible to your IDE. You find out when it breaks, which is to say at the worst possible time.

Pull the same dependency out of Props and you get this:

p.Logger.Info("starting") // a field access

p.Logger is a typed field. If it doesn’t exist, or you’ve used it wrong, the code simply doesn’t compile. Your IDE autocompletes it. Refactor the Logger interface and every misuse lights up at build time. There’s no runtime type assertion, because there’s no interface{} to assert from in the first place.

context.Context is the right tool for what it was designed for: cancellation, deadlines, request-scoped signals that genuinely cross API boundaries. It’s the wrong tool for “here are my program’s services”, because it trades away the compiler’s help for a flexibility you really don’t want here. Dependencies should be declared, somewhere the compiler checks them. Props is that somewhere.

What you get back for it

That one decision pays out in three currencies.

Testability. A command is now a pure function of its Props. To test it, you build a Props with the doubles you want (an in-memory FS instead of the real disk, a no-op Logger, a config you’ve populated by hand) and call the constructor. No global state to reset between tests, no monkey-patching, no init() order to puzzle over. The dependency is an argument, so the test just passes a different one.

Consistency. Cross-cutting changes have exactly one place to happen. When the global --debug flag flips the log level, it does so on the Logger inside Props, and because every command reads its logger from the same Props, every command gets the new level. No command can drift, because none of them owns its own copy.

Extensibility. Adding a new framework-wide service is just adding a field to one struct. Every command can immediately reach it; none of them needed touching to make it reachable.

To sum up

Props is the dependency-injection container at the heart of go-tool-base: one struct, passed to every command, holding the logger, config, filesystem, assets, error handler and tool metadata. It’s a concrete struct rather than a context.Context payload entirely on purpose, because dependencies belong somewhere the compiler can check them, not behind a string key and a hopeful runtime type assertion. That single choice buys you testability, consistency and easy extension.

The name says it best, really. Props doesn’t score the tries. It’s the broad-shouldered thing in the scrum that stops the whole framework folding, so the rest of your code is free to go and play.

Design your whole CLI in one file

Fri, 20 Mar 2026 00:00:00 +0000

Here’s a question that sounds trivial and really isn’t: where, exactly, does a CLI tool’s structure live? Not the logic of each command… the structure. Which commands exist, what they’re called, which flags they take, what’s nested under what.

I’d never properly thought to ask it until go-tool-base forced me to, and the answer turned out to be a little bit embarrassing.

Where does a CLI’s structure actually live?

Picture a CLI tool with twenty commands, some nested under others. In a typical project, where does its structure live? The answer is “smeared across the codebase”. It’s in twenty cmd.go files. It’s in the AddCommand calls that stitch them together. It’s in the flag registrations. To understand the shape of the tool you have to read all of it and assemble the picture in your head, because the picture exists nowhere as a single thing you can point at.

That’s a strange state of affairs for the single most important design fact about a CLI. The command tree is the tool’s interface, it’s the thing users actually touch, and yet it hasn’t got a home.

The manifest gives it one

go-tool-base’s generator gives that structure a home: .gtb/manifest.yaml. The manifest is a single readable file describing the command tree. Every command, its name, its short description, its flags, its place in the hierarchy, whether it carries assets or an initialiser. The shape of the whole tool, in one place you can open and read top to bottom.

And the manifest isn’t documentation about the project. It’s the thing the project’s wiring is generated from. When you run regenerate project, the generator reads the manifest and rebuilds the boilerplate to match it: the command registration, the AddCommand wiring, the flag definitions. The manifest is the source of truth, and the Go wiring is its output.

Design-first, when you want it

This unlocks a way of working that the smeared-across-the-codebase approach simply can’t offer. You can design the interface first, in the manifest, and let the code follow.

Want to rename a command? Edit one line in the manifest, run regenerate, and the rename propagates through every wiring file that ever mentioned it. Want to move a subcommand under a different parent? Change its place in the manifest hierarchy and regenerate. Want to add a flag to three related commands? Add it in the manifest, in three obvious places, and regenerate, instead of going on a little hunting expedition for three flag-registration blocks scattered across the tree.

You’re editing the tool’s interface as a design, in the file whose entire job is to hold that design, and the generator does the mechanical donkey-work of making the code reflect it. The thing you change is the thing that describes the structure. The code is downstream.

If that shape sounds familiar, it should. It’s the same instinct behind spec-driven and test-driven development: write down what the thing should be before you assemble how it works, and keep that statement of intent as a first-class, living artefact rather than a comment that quietly rots in a corner. The manifest is a spec for your command tree, and regenerate is what keeps the implementation honest to it.

It doesn’t trap you

There’s an obvious worry about any generated-from-a-manifest system: am I now locked into editing the manifest? What if I just want to open a Go file and write some Go like a normal person?

You can. The generator is careful not to own everything. It owns the wiring (the registration and the structural boilerplate) and it leaves your command logic well alone. The RunE function where your command actually does its work is yours; the manifest hasn’t got an opinion about it. And the generator tracks the files it produces by content hash, so if you do hand-edit something it generated, regeneration notices and asks before overwriting rather than steamrolling you. That mechanism turned out interesting enough to get its own post.

So the manifest is an option, not a cage. Design-first via the manifest when that suits the change. Drop into Go directly when that suits it better. The two stay in sync because regeneration reconciles them, not because one of them has been forbidden.

Pulling it together

A CLI’s command tree is its most important design surface, and in most projects it has no single home… it gets reconstructed in your head from twenty scattered files every time you need to reason about it. go-tool-base gives it one: .gtb/manifest.yaml, a readable description of the whole tree that the generator rebuilds the wiring code from. Edit the manifest, run regenerate, and the boilerplate follows.

It makes CLI structure something you design in one place, in the spirit of spec-driven development, while still leaving you free to write Go directly when that’s the better tool for the job. The manifest is the spec for your interface. The generator just keeps the code faithful to it.

Scaffolding that respects your edits

Fri, 20 Mar 2026 00:00:00 +0000

When I introduced go-tool-base I made a passing promise to come back to “the generator that won’t clobber your edits”. This is me keeping it, partly because it’s the feature I’m quietly most proud of, and partly because it took the most head-scratching of anything to get right.

The problem it solves is one that every code generator runs into eventually, usually the hard way and usually at the worst possible moment.

The generator’s awkward second act

A project generator has an easy first act. gtb generate project, and you’ve got a complete, wired, idiomatic Go CLI project. Everyone’s happy, me included.

The second act is the hard one. The framework moves on. A convention changes, a new built-in capability appears, the recommended CI shape shifts. Your project, scaffolded three months ago, is now subtly out of date, and you’d quite like the generator to drag it back up to spec.

Except by now it isn’t a fresh scaffold. It’s your project. You tuned the CI workflow. You rewrote the justfile. You added a stanza to the Dockerfile that took an afternoon and a fair bit of swearing to get right. The generated files and your edited files are one and the same files.

A naive generator handles this with breathtaking confidence: it regenerates everything from the template and overwrites the lot. Run it once, lose your afternoon. You learn that lesson exactly once and then never run regeneration again, which means the upkeep feature you were sold is dead on arrival. A scaffold you can’t safely re-run is just a one-shot cp with extra steps.

What the generator needs to know

The thing standing between “safe to overwrite” and “absolutely do not” is a single fact: has this file changed since the generator last wrote it?

If it hasn’t, the file is still pristine boilerplate and the generator owns it. Overwrite away. If it has, a human has been in there, and the generator must not touch it without asking first.

The generator can’t just eyeball that, of course. It needs a record. So every time gtb generate writes a file, it computes a SHA-256 of the content and stores it in the project’s manifest, .gtb/manifest.yaml, as a Hashes map of relative path to hash. The manifest is the generator’s memory of the exact bytes it last produced.

Regeneration becomes a three-way decision

With that record in hand, regeneration stops being “overwrite everything” and becomes a per-file decision with three branches.

The file doesn’t exist. Easy. Write it, store its hash.

The file exists and its current hash matches the manifest. It’s byte-for-byte what the generator last wrote, so nobody has touched it. The generator owns it outright, regenerates from the template and updates the stored hash. No prompt, no fuss. This is the common case, and it’s silent precisely because it’s safe.

The file exists and its hash does not match. Someone has been in there since generation. The generator stops and asks. It will not silently overwrite your hard-won afternoon. You decide: take the new version, or keep yours.

The detail I’m genuinely fond of is what happens when you decline. Declining is non-fatal. Generation carries on with the rest of the files, and the manifest keeps the file’s stored hash rather than dropping it. That matters more than it looks, because it means the file stays tracked. Next time you regenerate, the generator can still tell that file has been modified, and still asks. Skipping a file once doesn’t quietly evict it from the generator’s awareness forever. It stays a known, watched, customised file across every future run.

When you want it to stop asking

Per-file prompting is the right default, but for files you’ve permanently taken ownership of, being asked on every single regeneration is just noise. If you’ve rewritten the CI workflows wholesale and you are never, ever going back to the generated version, you don’t want a prompt. You want the generator to leave them well alone and not bring it up again.

That’s what .gtb/ignore is for. It sits next to the manifest and takes gitignore-style patterns:

# I own the CI workflows now
.github/workflows/**

# ...except the release workflow, keep that managed
!.github/workflows/release.yml

# and my build config
justfile
Dockerfile

Anything matching is skipped during regeneration with no prompt at all. Patterns evaluate top to bottom and later ones win, so the negation (!) behaves the way you’d expect from .gitignore: exclude a whole directory, then claw one file back.

It’s a deliberate escalation ladder. Unmodified files are handled silently. Modified files get a prompt. Files you’ve formally claimed get total silence. Each rung asks for less of your attention than the last, and you choose how far up to climb, file by file.

Stepping back

A generator earns its keep twice: once when it scaffolds your project, and then continuously, every time it drags that project back up to the framework’s current shape. The second job is worth nothing if regeneration flattens your customisations, because you’ll simply stop running it, and who could blame you.

go-tool-base’s generator gets around that by remembering. It hashes every file it writes into .gtb/manifest.yaml, and on regeneration it re-hashes before overwriting: unchanged files it owns and updates silently, changed files it stops and asks about, and .gtb/ignore lets you mark files as permanently yours. Skipped files stay tracked, so the generator never loses sight of what you’ve made your own.

The point of a scaffold isn’t the first five minutes. It’s that you can still run it in month three without holding your breath.

Your CLI is already an AI tool

Thu, 19 Mar 2026 00:00:00 +0000

“Make it work with AI” has become one of those requests that lands on a developer’s desk with a thud and not much further detail attached. My instinct, the first time, was to brace for a big lump of integration work… a bespoke adapter for this assistant, another for that one, a treadmill of little wrappers stretching off into the distance.

Turns out I’d already done most of the work. So have you, if your CLI tool is any good. Let me explain what I mean.

You already described your capabilities

Stop and think for a second about what a well-built CLI tool actually is. It’s a set of named operations, each with a human-readable description, each taking a set of typed, named, documented parameters. You wrote all of that already, because a CLI without it is unusable by people.

Now look at what an AI assistant needs in order to call a tool. A set of named operations. A description of each, so it knows when to reach for them. A typed parameter schema for each, so it knows how to call them.

It’s the same list! A good CLI is already, structurally, a description of a set of capabilities. The information an AI agent needs isn’t extra work you have to go and do. It’s work you finished the moment your --help output was any good.

The only thing missing is a translator. Something that takes “this is a CLI” and presents it as “this is a set of tools an AI can call”.

MCP is that translator, and it’s a standard

The temptation, when you want your tool to be AI-usable, is to sit down and write an integration. A little adapter for Claude Desktop. Another for Cursor. Another for whatever turns up next month. Each one a bespoke wrapper, each one a thing to maintain, and the list never stops growing because new assistants keep appearing. That’s the treadmill I was bracing for.

The Model Context Protocol exists to kill that list. MCP is an open standard for how an AI model discovers and calls local tools. Implement it once and your tool works with every assistant that speaks it. Write once, not once-per-client.

So go-tool-base implements it once, in the framework, for everyone. (That’s rather the theme of this whole series, if you hadn’t spotted it yet… do the annoying thing once, properly, in a place where every tool inherits it.)

The `mcp` command, and the mapping it does for free

Every tool built on go-tool-base inherits a built-in mcp command. Run it:

mytool mcp

and the tool starts a JSON-RPC server over standard I/O, speaking MCP. That’s the whole user-facing surface. One command.

Behind it, the framework walks your Cobra command tree and maps it straight onto MCP tool definitions:

Each command becomes a tool.
Each command’s short description becomes the tool’s description, the text the AI reads to decide whether this is the tool it wants.
Each command’s flags and arguments become the tool’s JSON Schema parameters.

There’s no second schema to write and then keep in sync (and we all know how well “keep these two things aligned by hand” tends to go). The command tree is the schema. Add a new command to your CLI and it’s a new tool for the agent, automatically, with the description and flags you already gave it. Nobody has to remember to update an MCP manifest, because there’s no separate MCP manifest to forget about.

Configuring an assistant to use it

On the assistant’s side it’s just as undramatic. You tell your AI client (Claude Desktop, Cursor, anything MCP-aware) to launch mytool mcp. From then on the assistant:

Starts your tool in MCP mode when it boots.
Discovers every command as a callable tool.
Calls the right one, with the right parameters, when a user’s request needs it.

Your CLI tool has quietly become something the AI can pick up and use, mid-conversation, on its own initiative.

The safety property worth noticing

Now, “let an AI run things on my machine” is rightly a sentence that makes people nervous. It makes me nervous, and I built the thing. So it’s worth noticing the constraint sitting quietly in this design.

The AI can only call what you defined. The tools it sees are exactly the commands in your tree, and the parameters it can pass are exactly the flags and arguments you declared, validated against the JSON Schema generated from them.

It can’t invent a command. It can’t pass a parameter you never defined. The boundary of what the agent can do is the boundary of what your CLI does, and you drew that boundary already, back when you built the tool. Exposing the CLI over MCP doesn’t widen the surface one inch. It just makes the existing surface reachable. The AI isn’t running things. It’s running your commands, the ones you wrote, tested and shipped, and nothing else.

The gist

A CLI tool, built properly, is already a structured description of a set of capabilities: named operations, descriptions, typed parameters. Which is also exactly what an AI agent needs in order to call a tool. The gap between the two is only a translator, and writing a bespoke one per assistant is a treadmill you don’t need to step onto.

go-tool-base puts the translator in the framework. Every tool gets an mcp command that serves the command tree over the Model Context Protocol… commands become tools, descriptions become descriptions, flags become JSON Schema parameters, with no second schema to maintain. Point any MCP-aware assistant at it and your CLI is an agent-callable tool, bounded to exactly the commands you shipped.

You did the hard part when you built a good CLI. MCP just opens the door you’d already framed.

go-tool-base: I got tired of reinventing the wheel

Wed, 18 Mar 2026 00:00:00 +0000

If you’ve written more than two or three command-line tools in Go, you’ll recognise the shape of the first afternoon. I certainly do! You reach for Cobra for the command tree, Viper for config, and then you start the part nobody ever puts in the README… the plumbing.

Where does config live? A file, an env var, an embedded default? In what order do they override each other? How does the tool tell the user there’s a newer version, and how does it actually update itself? What does logging look like, and is it the same logging the next tool will use? And how do you wire all of that into each command without every command reaching into a pile of globals?

None of it is hard. That’s the problem! It’s not hard, it’s just there, every single time, and every single time I’d find myself reinventing it slightly differently to the last time. Different override precedence here. A subtly different update flow there. Logging that didn’t quite match the tool I’d written three months earlier. Each new tool was a fresh re-litigation of decisions I’d already made and then promptly forgotten.

Now, I’ve banged on about the Boy Scout rule for years (leave the codebase better than you found it), but it has an uncomfortable corollary. If you keep turning up to the same campsite and finding it in the same mess, at some point the honest thing to do is to stop tidying it and go and build a better campsite.

First, just packages

So I started pulling the recurring pieces out into their own packages. Nothing grand. A config package that did the hierarchical merge the way I always ended up doing it anyway. A version package that knew how to compare semver and spot a development build. A setup package that handled first-run bootstrap and self-updating from a release. They lived as separate repos, and if you go digging through my GitHub history you can still find the scruffy ancestors of them scattered about.

Separate packages was the right first move. It forced each piece to stand on its own and earn its keep on a real project before I trusted it on the next one. A package that’s only ever been used in the repo it was born in hasn’t really been tested… it’s just been agreed with.

But separate packages come with a tax. Each one has its own release cadence, its own changelog, its own CI. Worse, they have to agree with each other at the seams, and when they’re versioned independently those seams drift. I’d bump the config package, and the setup package that depended on it would quietly need a matching bump, and the tool that used both would need telling about both. I’d traded “reinvent the wheel” for “keep a dozen wheels in sync”, and I’m really not convinced that’s a better deal.

Then, one library

Once the packages had been used enough (used in anger, on real tools, by people who weren’t me) the shape of them stopped moving. The interfaces settled. The arguments about precedence and defaults were over, because the answers had survived contact with reality.

That’s the point where separate packages stop being a virtue and start being friction. So I forged them into one and called it go-tool-base. One module, one version number, one changelog, and one set of seams that are now internal and can’t drift, because they ship together.

The heart of it is a dependency-injection container, a Props struct, that holds the things every command needs: the logger, the config, the embedded assets, the filesystem handle, the error handler, the tool’s own metadata. Commands are handed Props explicitly rather than reaching for globals, which means a command is just a function of its inputs and is therefore trivially testable. That one decision has quietly paid for itself on every tool I’ve built since.

Around that container sits all the stuff I was so tired of rewriting: hierarchical config, structured logging, version checking, self-update from GitHub or GitLab releases, an interactive TUI documentation browser, AI integration, service lifecycle management. A new tool inherits the lot and gets to spend its first afternoon on the thing that’s actually novel… its own logic.

Finally, a generator

A library still leaves you staring at a blank main.go. You still have to know the conventions, wire the container, lay out the directories, register the commands. All knowable, but all boilerplate. And boilerplate is exactly the enemy I set out to kill in the first place.

So go-tool-base ships a generator. gtb generate project scaffolds a complete, working, idiomatic project: directory layout, the wired Props container, the command tree, CI, the whole lot. gtb generate command adds a new command and registers it for you. The generator also handles upkeep: when the framework’s conventions move, it can regenerate the scaffolding of an existing project without trampling all over the code you’ve written on top. (That last bit turned out to be a properly interesting problem in its own right, and a future post.)

The goal is blunt. Creating a CLI tool should be about the tool, not the scaffolding. The first afternoon should be spent on the part that’s actually worth writing.

One thing I was careful about

There’s a nasty failure mode with “batteries-included” frameworks: the day you outgrow them, they hold you hostage. You either stay inside the framework’s worldview forever, or you face a rewrite. I’ve been burned by that before and I had no intention of inflicting it on anyone else.

So go-tool-base generates idiomatic, standard-library-compliant Go. There’s no magic runtime you can’t see, no clever code you couldn’t have written by hand. If you ever outgrow the framework the generated code stands on its own and you walk away with a perfectly normal Go project. A framework should be a starting point you’re glad you took, not a room you can’t get out of.

Where this leaves me

go-tool-base exists because I was spending the first afternoon of every Go CLI tool rebuilding the same plumbing, and rebuilding it slightly wrong relative to last time. It started life as separate packages so each piece could earn its place on real projects; once they’d stopped moving I forged them into a single library so the seams couldn’t drift; and then I wrapped a generator around it so a new tool starts as a working project rather than a blank file.

It’s a framework for the unglamorous 80% (config, versioning, updates, logging, lifecycle) so you can spend your time on the 20% that’s actually yours.

Over the coming posts I’ll dig into the individual pieces… the generator that won’t clobber your edits, the credential handling, the self-update integrity checks, and a few Go techniques I’m rather pleased with along the way. Stay tuned!

Migrating away from Mediawiki and how to export its data

Wed, 19 Aug 2020 00:00:00 +0000

I like Mediawiki, it is a simple tool capable of doing a lot and can be very flexible and easy to customise. However its not always the right solution! I had a situation where we needed to migrate away from using it for a combination of security and usability reasons. So I thought it would be good to document it.

After reviewing a few things it was decided to move things over to the companies already existing O365 SharePoint as a new site. This sounded laborious as first, but actually turned out to be pretty straight forward.

We start with getting data out of Mediawiki, thankfully we only wanted the most recent revision and not the full history of a page. We use PostgreSQL as a back-end so it was reasonably straight forward to figure out how to extract the data in a sensible query.

SELECT 
 page_id as id, 
 page.page_title as title, 
 pagecontent.old_text as content, 
 page_touched as edited
FROM mediawiki.page
LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
ORDER BY page_touched DESC;

It tool a little sleuthing to realize that the slots table was the pivotal in extracting the latest page version. With the right join and a little mangling of the content_address field from the contents table to remove the “tt:” from the value and convert to an integer we now have a result set of all the page names and the latest revision of that page. I also added in the date the page was last updated to allow me to see when it was last edited as it was a live system migration and helped me to ensure content remained sync while both were still in play.

Once I had the query it was a simple job of writing an “Exporter” using Go Lang to extract the data and write it to files, I’ll chuck a snippet of code at the bottom of the post for you.

Mediawiki uses wikitext as a format so I needed to convert it to something more widely understood. Having used Pandoc in the past successfully I plumped for this as I knew it would handle a lot of options and was simple to use to convert to the markdown_mmd format

I Installed it via the ubuntu apt package available on my system and wired this in as a hacky exec command into my script… and voila! I had hardcopies of all the Mediawiki pages on my system in both wikitext and markdown_mmd format.

Why markdown_mmd I hear you ask… mainly because it gave me the cleanest conversion for use with the new markdown web page widget for Sharepoint’s modern interface.

Now we have the files we could do a little munging and parsing to convert URLs into the format needed for the new location in Sharepoint, easily done with a bit of regex pattern matching, which I wont go into as yours will be very different from mine… suffice to say looking for "wikilink" in my regex helped massively in finding all the occurrences I needed to update. I used sed but you could use whatever tool you like or add it into your version of the exporter

'SysAdmin/(.+) "wikilink"'

and with a little back referencing to substitute the values we need to keep and its all good.

Next came the import of the data into Sharepoint, but that is a post for another day.

package data

import (
	"bytes"
	"fmt"
	"github.com/jmoiron/sqlx"
	"github.com/rs/zerolog/log"
	"io/ioutil"
	"os"
	"os/exec"
	"path/filepath"
	"time"
	"wiki-export/src/util"
)

type Page struct {
	Id int
	Title string
	Content string
	Edited time.Time
}

type Exporter struct {
	Config util.ExporterConfig
	DB *sqlx.DB
}

func (l *Exporter) Export() {
	stmt := `
		SELECT page_id as id, page.page_title as title, pagecontent.old_text as content, page_touched as edited
		FROM mediawiki.page
		LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
		LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
		LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
		ORDER BY page_touched DESC
		;`

	page := Page{}
	rows, err := l.DB.Queryx(stmt)
	util.CheckErr(err)

	for rows.Next() {
		util.CheckErr(rows.StructScan(&page))
		wikiFilename := fmt.Sprintf("%s.mediawiki",filepath.Base(page.Title))
		mdFilename := fmt.Sprintf("%s.md",filepath.Base(page.Title))
		path := filepath.Dir(page.Title)
		wikiDir := fmt.Sprintf("%s/mediawiki",l.Config.TargetDir)
		mdDir := fmt.Sprintf("%s/%s",l.Config.TargetDir, l.Config.TargetFormat)

		if path != "." {
			wikiDir = fmt.Sprintf("%s/mediawiki/%s",l.Config.TargetDir , path)
			mdDir = fmt.Sprintf("%s/md/%s",l.Config.TargetDir , path)
		}
		util.CheckErr(os.MkdirAll(wikiDir, 0777))
		util.CheckErr(os.MkdirAll(mdDir, 0777))

		wikiTarget := fmt.Sprintf("%s/%s", wikiDir, wikiFilename)
		mdTarget := fmt.Sprintf("%s/%s", mdDir, mdFilename)

		log.Debug().Msgf("%s => %s -> %s", page.Title, wikiTarget, mdTarget)

		c := []byte(page.Content)
		util.CheckErr(ioutil.WriteFile(wikiTarget, c, 0777))

		cmd := exec.Command("pandoc", "-f","mediawiki", "-t", l.Config.TargetFormat, wikiTarget)
		
		var errorBuffer bytes.Buffer
		var outputBuffer bytes.Buffer
		cmd.Stdout = &outputBuffer
		cmd.Stderr = &errorBuffer
		err := cmd.Run()
		if err != nil {
			log.Err(err).Msgf("ERROR: %s", errorBuffer.String())
			util.CheckErr(err)
		}

		util.CheckErr(ioutil.WriteFile(mdTarget, outputBuffer.Bytes(), 0777))
	}
}

Pre-populating Neo4J using Kubernetes Init Containers and neo4j-admin import

Wed, 15 Jul 2020 00:00:00 +0000

Recently there has been an uptake in the use of Neo4j by the Data Scientists. This is a good thing! they are wanting to use the right tool for the job. However we need to run it inside our k8s cluster as a portable readable data source that has been dynamically populated from a pile of data in a combination of PostgreSQL and MongoDB.

This isn’t a problem for them working locally, they install and spin up a local copy of Neo4j and can interact with it quite happily. They even realised that they can generate CSV’s from PostgreSQL and MongoDB and then import them, blindingly fast, into Neo4j using the neo4j-admin tool that comes bundled. Fantastic!

At least until they come to want to run their Neo instance inside our k8s cluster. That’s where I step in and turn them aside from creating their own custom neo4j image with a bespoke entry point that loads all the data for them in some crazy threaded bash scripting!

“No, No, No!” I tell them. “It’s far easier to just add an init container to your pod, that will preload the data before Neo starts up”.

Init containers, if you haven’t come across before, them are a special type of container that lives inside a k8s pod and are set to run BEFORE your main container runs. In this case it means we can easily sequence a bash script to run the neo4j-admin import before Neo4j is even started. And here is how we did it!

The script

The data scientists had been using Neo4j 3.5.x locally because they had a need for the graph algorithms plugin (https://github.com/neo4j-contrib/neo4j-graph-algorithms) which at the time they were looking didn’t support Neo4j 4.x. The plugin is now deprecated and its replacement (https://github.com/neo4j/graph-data-science) thankfully supports 3.5.x and 4.x.

As Neo4j 4.x introduces a lot of new features and improves performance so I recommended we switch to using that. This meant a refactor of their bash script for neo4j-admin there some very subtle differences and a few caveats to work with. This is what they came up with

#!/bin/bash
DBNAME="neo4j"
if [ "$#" -eq 1 ]; then
 DBNAME=$1
fi

# extract data from SQL
python3 extract_data.py

# remove old db for rebuild
rm -rf "/data/databases/$DNBAME"

neo4j-admin import \
 --database=$DBNAME \
 --delimiter="|" \
 --nodes=Protein=${NODE_DIR}/nodes_protein_header.csv,${DATA_DIR}/nodes_proteins.csv \
 --nodes=UniProtKB=${NODE_DIR}/nodes_uniprot_header.csv,${DATA_DIR}/nodes_uniprot.csv \
 --relationships=HAS_AMINO_ACID_SEQUENCE=${EDGE_DIR}/edges_protein_sequence_header.csv,${DATA_DIR}/edges_protein_sequence.csv \  
 --relationships=HAS_AMINO_ACID_SEQUENCE=${EDGE_DIR}/edges_chembl_protein_biotherapeutic_molregno_header.csv,${DATA_DIR}/edges_chembl_protein_biotherapeutic_molregno.csv \
 --skip-bad-relationships=true \
 --skip-duplicate-nodes=true

The import command here is significantly shorter for example purposes, as the original is about 120 lines long. As you can see it’s pretty straight forward, they had another script in extract_data.py, that I wont bore you with suffice to say that it pulled out all the data they wanted from PostgreSQL and MongoDB, which got saved to disk as CSV files in the relevant directories.

Great, it worked on their local version!

The Dockerfile

ROM neo4j:latest
ENV NEO4JLABS_PLUGINS ["graph-data-science"]
RUN apt update && apt install -y python3
WORKDIR /srv
COPY src /srv/src
COPY headers /srv/headers

The plan is always to keep it simple. We have one image that we can run for both the init container and the main container. This docker file gives a vanilla neo4j instance with python and our scripts for extracting the data loaded into it

The k8s Manifest

apiVersion: v1
kind: Pod
metadata:
 name: neo4j
spec:
 containers:
 - name: neo4j
 env:
 - name: NEO4J_AUTH
 value: neo4j/password
 image: registry.example.com/phpboyscout/rnd_graph:latest
 imagePullPolicy: Always
 volumeMounts:
 - mountPath: /data
 name: neo4j
 subPath: data
 initContainers:
 - name: importer
 args:
 - neo4j_import.sh
 command:
 - /bin/bash
 env:
 - name: DATA_DIR
 value: /import/data
 - name: HEADER_DIR
 value: /srv/headers
 image: registry.example.com/phpboyscout/rnd_graph:latest
 imagePullPolicy: Always
 stdin: true
 workingDir: /srv/src
 volumeMounts:
 - mountPath: /data
 name: neo4j
 subPath: data
 - mountPath: /import
 name: neo4j
 subPath: import
 - name: neo4j
 persistentVolumeClaim:
 claimName: neo4j

Now we can pull it all together with our k8s manifest. From here you can see that we have our default neo4j container that we pass in our default authentication details to and an init container that runs our import.sh script. Both containers have access to a shared volume for the /import and /data folders.

And now we get to…

Troubleshooting

So right off the bat it didn’t work! No surprises there but here are a few things that caused us some issues and how we resolved them.

Database offline

At first glance everything seemed to work. Until we tried to connect to the neo4j database with the default UI, at which point we were presented with the error message

Database "neo4j" is unavailable, its status is "offline."

This took a little sleuthing and shelling into the neo4j container to take a look at the /var/debug.log file which gives significantly more useful information about whats going on with the server. First we were getting stack traces that contained messages like

Component 'org.neo4j.kernel.impl.transaction.log.files.TransactionLogFiles@59d6a4d1'
was successfully initialized, but failed to start. Please see the attached cause 
exception "/data/transactions/neo4j/neostore.transaction.db.0"

From experience this sounded like a permissions issue and lo and behold, checking the files on the filesystem showed that because the import script was run as root the database files were owned by root. We resolved this by adding:-

chown -R neo4j:neo4j /data/

to the bottom of the import script. Next we were then presented with an error that looked like

2020-07-14 16:56:33.919+0000 WARN [o.n.k.d.Database] [neo4j] Exception occurred while
starting the database. Trying to stop already started components. Mismatching store id.

This one seems like it would be an obvious one to google and I did come up with few pages that seemed to describe what was happening to me but gave some varied solutions, from starting and stopping the sever and running neo4j-admin unbind in between to deleting various files. It seemed very strange because we did test this with the 3.5.17 version of Neo and it worked fine.

The solution we needed was to wipe the slate clean properly. The line in our script to remove the previous build of the db

# remove old db for rebuild
rm -rf "/data/databases/$DNBAME"

just didn’t cut it. It turns out that because the 4.x version of Neo4j supports multiple databases the import command writes additional information to the system database and transactions database in the form of some identifiers for each database, BUT if you don’t do something to clear that value for the database your are building it wont match up when the server starts and you get a declaration of Mismatching store id

I’m not sure if the developers are aware of this flaw, so in the mean time we have to expand our cleanup to:

# clean up for fresh import
rm -rf /data/databases/*
rm -rf /data/transactions/*

removing the neoj4, system and store_lock databases and transaction logs from the data store. This solved the problem and the server was able to start and we could connect to neo4j database successful.

Its not an ideal solution, I can foresee definite situations we will have to work around when we get to a point where multiple databases may be needed and are built separately and independently from each other. but it will suffice for now.

Malloc(): Error message goes here

Once it was up and running we noticed that we were getting lots of restarts on the main neo4j container a quick look at the stdout log and we could see each restart ending with something that looked like

malloc(): corrupted top size

instantly this looks like an issue with memory sizing inside the container for the JVM. Thankfully the team at Neo4j have accounted for this and give you a nice tool in the form of

neo4j-admin memrec

which interrogates the databases and gives some sensible values you can set in the output which in our case looked like


# Memory settings recommendation from neo4j-admin memrec:
#
# Assuming the system is dedicated to running Neo4j and has 376.6GiB of memory,
# we recommend a heap size of around 31g, and a page cache of around 331500m,
# and that about 22400m is left for the operating system, and the native memory
# needed by Lucene and Netty.
#
# Tip: If the indexing storage use is high, e.g. there are many indexes or most
# data indexed, then it might advantageous to leave more memory for the
# operating system.
#
# Tip: Depending on the workload type you may want to increase the amount
# of off-heap memory available for storing transaction state.
# For instance, in case of large write-intensive transactions
# increasing it can lower GC overhead and thus improve performance.
# On the other hand, if vast majority of transactions are small or read-only
# then you can decrease it and increase page cache instead.
#
# Tip: The more concurrent transactions your workload has and the more updates
# they do, the more heap memory you will need. However, don't allocate more
# than 31g of heap, since this will disable pointer compression, also known as
# "compressed oops", in the JVM and make less effective use of the heap.
#
# Tip: Setting the initial and the max heap size to the same value means the
# JVM will never need to change the heap size. Changing the heap size otherwise
# involves a full GC, which is desirable to avoid.
#
# Based on the above, the following memory settings are recommended:
dbms.memory.heap.initial_size=31g
dbms.memory.heap.max_size=31g
dbms.memory.pagecache.size=331500m
#
# It is also recommended turning out-of-memory errors into full crashes,
# instead of allowing a partially crashed database to continue running:
#dbms.jvm.additional=-XX:+ExitOnOutOfMemoryError
#
# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.
# They can be used as an input into more detailed memory analysis.
# Total size of lucene indexes in all databases: 0k
# Total size of data and native indexes in all databases: 17300m

So how to get these values into the container… Thankfully this is handled for you in the form of Environment Variables you can pass into the docker image. A bit of a google and i found this little snippet which is a goldmine for telling us how to translate settings into environment variables.

# Env variable naming convention:
# - prefix NEO4J_
# - double underscore char '__' instead of single underscore '_' char in the setting name
# - underscore char '_' instead of dot '.' char in the setting name
# Example:
# NEO4J_dbms_tx__log_rotation_retention__policy env variable to set
# dbms.tx_log.rotation.retention_policy setting

As for getting the variables into the container, you could do this from the pod and inject it in. I this case because the data we are going to be using is reasonably stable and tested we decided to stick them into the Docker file with the ENV directive.

ENV NEO4J_dbms_memory_heap_initial__size 31g
ENV NEO4J_dbms_memory_heap_max__size 31g
ENV NEO4J_dbms_memory_pagecache_size 331500m

And so far we haven’t had a restart yet!

Adding Ambient Sounds to your Discord Server On LInux

Tue, 30 Jun 2020 00:00:00 +0000

I’m a Dungeon Master! I don’t mean that in the S&M sense! As in the game Dungeons & Dragons (https://dnd.wizards.com), where I run a weekly game as well as take part in a couple of campaigns as a player. It’s a lot of fun and something I would definitely recommend you have a go at if you are so inclined

There is a vast amount of tooling & tech out there that allows you to play remotely such as Virtual Table Tops, Character builders, online resources, etc. One such tool that gets used quite often is a chat service called Discord (https://discord.com) It’s really useful and allows you to easily be part of and manage communities of people…. Think IRC & Slack, but more up to date than IRC and less “workish” than Slack.

As part of my online games I like being able to have ambient music to match the surroundings the players are traveling through, as well as some active elements thrown in for good measure. This is possible in a few different ways using discord but the way I want to set it up can be somewhat frustrating to set up. Let me explain:

I have taken a shine to two tools in particular… Syrinscape (https://syrinscape.com) and Table Top Audio (https://tabletopaudio.com). The former being a windows app with an nice interactive mixing UI that allows you to combine and generate unique sounds, the latter being a lovely web service that has some fantastic loop-able ambient background tracks all 100% free.

I am wanting to be able to pipe the audio from these two services into my Discord server so that I can make use of the fantastic audio they offer. This is the journey of how I managed to get this working, partly as a reminder for me if I ever need to do this again and also to help others that may be looking to do the same.

My Setup

I’ve been a big fan of Ubuntu for a number of years, but since 20.04 I’ve found that the shine I’ve had for it has waned significantly. I wont go into the why and wherefore of it but I’m now running the excellent Pop_OS! from System76 (https://pop.system76.com) its an Ubuntu variant but with the bits I dislike removed. So assume that anything I’m doing is compatible with Ubuntu 20.04.

The Requirements

The ideal solution should see me being able to have a single instance of discord running that allows me to still use my mic to be able to talk, and to have my selected background playing with the ability to control the volumes of both the mic and the background independently.

Finding A Solution

A lot of googling led me to realise that there isn’t a perfect solution to fit my brief. The hardest part being not actually knowing what to google and a lot of the terminology being somewhat foreign to me as I’m not much of an audio engineer. However I finally stumbled upon a blog post by Emma Anderson dated June 2016 and thankfully it gives me a lot of the heavy lifting that I needed along with some explanation of what I’m trying to achieve, though I’m hopefully going to be more verbose here in what this all means and how it works.

PulseAudio

The first thing we need to do is make sure the packages for pulseaudio and pavucontrol are installed. These will allow us to manipulate the way we capture sound and redirect it to the appropriate inputs and outputs.

apt install pulseaudio pavucontrol

Virtual Input & Virtual Mic

What we are going to try to achieve, is to create two new elements inside of Pulseaudio;

a Virtual input that we can channel the applications creating our background sounds which will allow us to control the volume independently.
a Virtual Microphone that we can channel our both our normal microphone and the new Virtual input into.

By creating these elements we can then use the pavucontrol tool to select what needs to be redirected where. so lets get started.

pactl load-module module-null-sink sink_name=VirtualInput
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

Here we have two commands, the first will create our new Virtual Input as what is referred to as a “null sink”. This on its own is not really very useful for us as we also need what is referred to as a “source”, thankfully when we run this command it also created a new “source” for us.

On it’s own that should be more than enough, but running the 2nd & 3rd command makes our live a lot easier because it will apply some very useful labels to both of the newly created sink and source. In this case VirtualInput for the sink and VirtualInput.monitor for the source. Having these in place makes it a lot simpler to configure things with pavucontrol.

Next we need to create our Virtual Mic using some very familiar looking commands.

VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

again we have now created a new new pair of sink and source with some nice easy to recognise labels that we will use when we start working with pavucontrol.

The next piece to our puzzle is creating the components that will let us define a connection from the VirtualInput and our physical microphone to the newly created VirtualMic. We do this with two identical commands;

pactl load-module module-loopback sink=VirtualMic
pactl load-module module-loopback sink=VirtualMic

we now have most of the elements that we need to configure everything to work.

Listening to my own Ambience

Before we can start wiring it all together we need to ensure we can also listen back to our own ambience. This involves us creating one more “loopback” module that points to the speakers we are wanting to listen to. Lets find out what our options are by running;

pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device.description/ {print $0}

This lists all of the available “sinks” that we can use. on my daily driver laptop I get;

 * index: 1
	name: <alsa_output.pci-0000_00_1f.3.analog-stereo>
		device.description = "Built-in Audio Analogue Stereo"

This tells us the “index” for the device, its name and also some kind of description. The important bit for us here is the name as we will need that to create our new “loopback” with the command;

pactl load-module module-loopback sink=alsa_output.pci-0000_00_1f.3.analog-stereo

This creates the last piece for our puzzle!

Connecting it all together

I’m now going to assume you have logged yourself into the Discord client and fired up your copy of Syrinscape… but you should just as easily swap out these for something else of your choice.

Now we can start pavucontrol either from the command line or you can look for it in your applications menu. Once it loads you will hopefully be presented with something that looks like;

For this next step I am specifically starting on the “Recording” tab of pavucontrol this is to allow us to set up what is going to be captured. I have updated the drop-down at the bottom left to show “All Streams” as this will make it quicker to configure… Starting at the top we have two entries for;

Loopback to VirtualMic from: These are the result of the first two “loopback” modules we created with the pactl command we ran previously. They are going to allow us to capture the audio streams from our physical microphone, mine here is the TONOR TC-777 and our newly created VirtualMic.

followed by a single entry for;

Loopback to Built-in-Audio Analougue Stereo from: which is the last “loopback” module that we create to let us hear our own Ambience, Having this set to our VirtualInput means that anything that pipe into our VirtualInput will also come out of our speakers.

and finally;

WEBRTC VoiceEngine: Once you connect to a voice channel in discord this will appear and it allows us to specify which of our devices it should be reading the audio feed from. For our purposes we have this set to our VirtualMic so that we can have our mixed audio feeds

Now that recording is configured we can sort out our playback.

Here we can see the “Playback” tab of pavucontrol, again set to show “All Streams”. This time I’m going to run through the elements here starting from the bottom of the list and working my way up…

WEBRTC VoiceEngine: This again is our connection to a Discord voice channel, as you can see I have this set to play back all of its output via Built-in Audio Analogue Stereo which is how my Operating system has labelled my physical speakers.

Syrinscape.exe: This is the Syrinscape application, that I run through PlayOnLinux (https://www.playonlinux.com), and I will use to generate all of my lovely ambient sounds. This is set to play all of its “audio stream” on our VirtualInput.

The next two items in our list, providing you have configured the Recording tab first should look as in the image. Changing the “Loopback to VirtualMic” entries on the Recording tab will change the labels of these two entries.

Loopback of VirtualInput.monitor on: it seems we have two of these entries and where we can tell the to pip all of the audio we are now capturing on our VirtualInput In this case we want it to go to two places, our VirtualMic so that it can be sent to both our Discord audio channel and also to our Built-in Audio Analogue Stereo speakers.

Loopback of Built-in Audio Analogue Stereo on: is where we now direct the input from our physical microphone and feed that straight into our VirtualMic.

The other entries in the list here are for firefox and the system itself and are not relevant to what we are trying to achive.

Winner Winner Chicken Dinner

That’s is effectively all we need to do… From here on in anything you play via the Syrinscape app will be merged with your microphone input and passed to Discord. You can then use the volume sliders in pavucontrol to adjust the levels of all the inputs to suit your own personal preference.

Though I will make a few small suggestions about how to configure your discord settings. You shouldn’t need to make any adjustments to the input and output devices which should now be set to Default as your “Input Device” if you change this it will override the changes we have made and you will need to go back to the Recording tab of pavucontrol and switch WEBRTC VoiceEngine back to InputMic , but…

I would recommend disabling automatic input sensitivity and lowering the sensitivity slider all the way down to -100dB… this is to allow for the potential low and subtle tones and ambient elements you may want to play… be warned though it makes it very very easy for an low quality microphone (such as the Built-in Audio Analogue Stereo microphone found on my laptop) to pick up other noises such as your systems fans, mouse clicks and typing. A simple way to combat this is to get a reasonable quality external cardioid condenser microphone which eliminates a lot of this unwanted background.

One last thing

That should be it for now… I’ll leave you with one final thing. This is a simple little bash script I threw together that I run in can a terminal to create all the components and if I want will then clean them all up and remove them. If you really want you could set it up as a permanent implementation, but I’ll let you google for that solution!

#!/bin/bash

LB1=""

listenback() {
 echo ""
 echo "Listing all possible output devices"
 pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device\.description/ {print $0}'
 echo ""
 echo "Please enter the name of the output device to create a loopback for (leave blank to skip): "
 read S1
 if [ "$S1" != "" ]; then
 	echo " * Creating Loopback for '$S1'"
 	LB1=$(pactl load-module module-loopback sink="$S1")
 fi
}

cleanup() {
 while true; do
 read -p "Finished? do you want to clean up and remove modules [Yn]: " yn
 case $yn in
 [Yy]* ) return 0;;
 [Nn]* ) return 1;;
 * ) echo "Please answer yes or no.";;
 esac
 done
}

listenback

echo " * Creating VirtualInput"
VI=$(pactl load-module module-null-sink sink_name=VirtualInput)
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

echo " * Creating VirtualMic"
VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

echo " * Creating loopbacks for VirtualMic"
VML1=$(pactl load-module module-loopback sink=VirtualMic)
VML2=$(pactl load-module module-loopback sink=VirtualMic)


echo "All modules have been loaded have been configured! Run pavucontrol to configure your devices."

if cleanup; then
 pactl unload-module "$VML2"
 pactl unload-module "$VML1"
 pactl unload-module "$VM"
 pactl unload-module "$VI"
 if [ $LB1 != "" ]; then
 pactl unload-module "$LB1"
 fi
 echo "All modules have been unloaded"
else 
 if [ $LB1 != "" ]
 then
 echo "Modules $LB1, $VI, $VM, $VML1 & $VML2 remain loaded"
 else
 	echo "Modules $VI, $VM, $VML1, & $VML2 remain loaded"
 fi
fi

Encrypting additional drives with LUKS on Linux

Mon, 29 Jun 2020 00:00:00 +0000

Encryption is king nowadays with everyone having mobile devices. We have a significant number of people on laptops that travel around and also workstations that live in open plan offices. This means we encrypt all of our disks… just in case. 99% of the time is super simple to do as most OS installers give you the option to do it, some now ven enforce it as a default option. This post however is about adding an additional disk to the system and making it automatically mount on system startup.

So let me set the scene, we have a data-scientist that’s running out of disk space for a task they are running on their Ubuntu 18.04 Workstation. At some point the workstation had an upgrade to the HDD in the past to a shiny new SSD, and the old 4Tb spinning disk was left in the chassis that they want to use for this very specific task.

Now this workstation has been through a couple of data-scientists over the last 12 months and unfortunately the LUKS password that had been set up for the old spinning disk has gone walkabouts… so the plan is as follows

flatten the old disk and set up a new partition using the whole disk
generate a new secure encryption key
set up LUKS encryption on the new partition
use Ext4 as a filesystem
enable auto decryption of the disk
add the new partition to the fstab to mount on system startup

N.B Assume that we are running everything as the root user

Flatten the disk

As we cant recover anything we are going to flatten the disk using parted (apt install parted to install) to allow is to create a partition greater than 2Tb, but first we are going to identify the disk we are working with… I tend to favour using either fdisk -l or as a more concise option lsblk -p which gives us a an easy to interpret overview something like:

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk

I can tell from this that we are looking at using the disk that is currently at /dev/sdb and its showing as being 3.7Tb in size.

Great… now to set up our new partition using the command parted /dev/sdb which gives us an interactive shell to work with (you can see the prompts in the output below are prefixed with (parted)

GNU Parted 3.2 
Using /dev/sdb 
Welcome to GNU Parted! Type 'help' to view a list of commands. 
(parted) mklabel gpt

The command mklabel gpt will wipe the partition table for /dev/sdb and give us a clean slate to work from

(parted) unit TB

We now set parted to think in Terabytes as the default reference size using the command above.

(parted) mkpart primary 0.00TB 3.70TB

Now we get to create the actual partition. You can see from the command above that we are using the command mkpart and telling it to create a primary partition type.

(parted) print 
Model: ATA WDC WD4005FZBX-0 (scsi) 
Disk /dev/sdb: 4.00TB 
Sector size (logical/physical): 512B/4096B 
Partition Table: gpt 
Disk Flags: 
 
Number Start End Size File system Name Flags 
 1 0.00TB 4.00TB 4.00TB primary

We can chek everything went smoothly using the print command which gives us confirmation that a new primary partition is present. We can now leave parted with a simple.

(parted) quit

And we can now use fdisk -l or lsblk -p to see that we now have a partition waiting for us at /dev/sdb1.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part

Generating an encryption key

Our disk is now ready for use, but not yet encrypted, so our next step is to create a key that can be used when we encrypt the disk. As we are going to be mounting it automatically we want to use a keyfile to store the key. You can of course create a key by mashing the keys on the keyboard, but I tend to prefer letting something else do the hard part for me.

First we create somewhere to store the key… I opted for,

mkdir -p /etc/crypt/keys

But feel free to put it wherever you want just as long as its only accessible by the root user. Next we generate the keyfile using the command:

dd bs=512 count=4 if=/dev/urandom of=/etc/crypt/keys/sdb1 iflag=fullblock

Here I am using /dev/urandom as my randomness generator, but you could use any valid generator of your choice. With this set of parameted dd will read the stream of “randomeness” and write 2048 bytes to our keyfile at /etc/crypt/keys/sdb1. If you want to be a little more complex about teh size and shape of your key then have a look at https://man7.org/linux/man-pages/man1/dd.1.html

Encrypting the Disk

Hopefully it will already be installed because you encrypted your root disk at installation, but if not you can run apt install cryptsetup to get going.

The command to do the encryption is actually very simple.

cryptsetup luksFormat /dev/sdb1 /etc/crypt/keys/sdb1

You can see that using the cryptsetup tool we are asking it to execute teh command luksFormat but while it says format in the command this is a little misleading as it doesn’t actually format the disk but just rewrites a portion of bytes at the beginning of the partition to enable encryption. we then tell it the partition we want encrypting, here its /dev/sdb1 and finally we pass in the keyfile we just generated and saved at /etc/crypt/keys/sdb1. If you omit the keyfile it will still encrypt teh disk but will prompt you to enter the key manually.

As soon as you press enter you will be warned of teh danager of what you are doing… so double check you are encrypting the right partition and follow the instructions that should look something like :

WARNING! 
======== 
This will overwrite data on /dev/sdb1 irrevocably. 
 
Are you sure? (Type uppercase yes): YES 
Command successful.

And that’s it… the disk is encrypted and ready to use. There are a few ways you can now work with the disk. the quickest and easiest is to just decrypt the disk manually using cryptsetup to open the disk.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

Here we open /dev/sdb1 and give is a new name of sdb1_crypt and we unlock it using the -d argument to tell it the keyfile we generated before.

That is the dis decrypted and ready to roll… you can now use fdisk -l or lsblk -p to confirm that it is now available at /dev/mapper/sdb1_crypt.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part 
 └─/dev/mapper/sdb1_crypt 253:3 0 3.7T 0 crypt /mnt/4tb-1

This tells us that the newly decrypted disk is now available at /dev/mapper/sdb1_crypt and is a volume of 3.7Tb… Exactly what we were hoping for!

All finished with your encrypted disk… you can just as easily close it again using:

cryptsetup close sdb1_crypt

Setting up the Filesystem

Ok, we have an encrypted partition, we can decrypt it but we cant mount it yet as we don’t have a file system to work with. Let’s take care of that real quick by opening up the partition again.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

And now that its available we are going to set up an ext4 filesystem using the command mkfs.ext4 /dev/mapper/sdb1_crypt which, all going according to plan, should look something like:

mke2fs 1.44.1 (24-Mar-2018) 
Creating filesystem with 976753664 4k blocks and 244195328 inodes 
Filesystem UUID: d797be67-c53e-49d3-897e-c624b21a22d3 
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
 102400000, 214990848, 512000000, 550731776, 644972544 
 
Allocating group tables: done 
Writing inode tables: done 
Creating journal (262144 blocks): done 
Writing superblocks and filesystem accounting information: done

we are now good to go… lets try mounting the filesystem with

mount -t ext4 /dev/mapper/sdb1_crypt /mnt

I’m just mounting straight to /mnt but obviously this can be any folder you want. If the command worked we can easily confirm it with a quick df -h:

Filesystem 1K-blocks Used Available Use% Mounted on
udev 32846708 0 32846708 0% /dev
tmpfs 6578140 2308 6575832 1% /run
/dev/mapper/ubuntu--vg-root 1919562064 993193376 828790500 55% /
tmpfs 32890688 200 32890488 1% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 32890688 0 32890688 0% /sys/fs/cgroup
/dev/sda2 721392 276068 392860 42% /boot
/dev/sda1 523248 6232 517016 2% /boot/efi
/dev/mapper/sdb1_crypt 3844637680 0 3844637680 1% /mnt

Excellent… you can now start working with your new partition… however lets un-mount and close the drive quickly with a

umount /mnt && cryptsetup close sdb1_crypt

And then we can move onto…

Automatic Decryption

This is a lot simpler that you may realise… all we need to do is add a new line to the file /etc/crypttab! But first we need one last piece of information we don’t yet have, but we can easily get with the command

sudo cryptsetup luksDump /dev/sdb1 | grep "UUID"

This will use luksDump to get information about the encrypted partition and then uses grep to specifically target the property UUID which we will need to identify the partition in the next step.

Now in your favourite editor of choice add the following line, replacing the spoof UUID here with the one we just found.

sdb1_crypt UUID=1111111111-2222-3333-4444-555555555555 /etc/crypt/keys/sdb1 luks

Here we are giving the decrypted volume a unique label for teh decrypted label to be made available at the appropriate /dev/mapper/* location. We also specify the UUID to identify the partition to decrypt… we could use the path /dev/sdb1 but using the UUID is more explicit and prevents any confusion if another partition happens to present itself as /dev/sdb1 at some point in the future. Third we have the path to our newly generated keyfile and finally we have the encryption mode that we are using for encryption which here is luks. For more info on crypttab have a look at https://www.freedesktop.org/software/systemd/man/crypttab.html

We can now test that auto decryption is working using:

cryptdisks_start sdb1_crypt

which if successful should have an output like:

 * Starting crypto disk... 
 * sdb1_crypt: INSECURE MODE FOR /etc/crypt/keys/sdb1, see /usr/share/doc/cryptsetup/README.Debian. 
 * sdb1_crypt (starting).. 
 * sdb1_crypt (started)...

All that’s left to do now is set up

Auto-mount the filesystem

Hopefully we now are on really familiar ground… we can now treat /dev/mapper/sdb1_crypt as a bog standard ext4 partition that can be mounted via the /etc/fstab by adding the line:

/dev/mapper/sdb1_crypt /mnt ext4 defaults 0 2

As you can see its pretty ordinary, exactly as you would expect, obviously swapping out /mnt with the location of your choice to mount the filesystem. If you are not wholly familiar with fstab then its definitely worth having a look at https://help.ubuntu.com/community/Fstab as it gives a good overview for those who are new to it…

Finally we can check that it all works with:

mount -a

And at this point I am pretty sure I can hear the fat lady singing…

Connecting to Ubuntu 18.04+ using RDP

Mon, 20 May 2019 00:00:00 +0000

We have a mix of different setups that the Software Engineer and Data Scientists use to get their work done. There are some using just Linux on laptops, Some on MacBooks and some on the various versions of Windows.

For those not using Linux as their primary OS we have a bunch of Desktops that run Ubuntu 18.04+ for them to connect to. SSH can do quite a lot but a few of the team work remotely and in house we prefer RDP for that kind of thing rather than VNC.

We have had some issues with connections in the past so this post exists to remind me how next time I need to set it up. First we need to install the xRDP server package.

sudo apt install xrdp

Next we need to ensure that we have the right ports open on the workstation. If like me you also use UFW to manage your firewall rules then open port 3389 using…

sudo ufw allow 3389

The issue left, is that you will get an annoying pop up when you log in about a colour management profile needing to be set up and asking you to provide your password. Even then you may still get some annoying crash pop-ups.

I found a really good solution to this at http://c-nergy.be/blog/?p=12043 which I’ve cribbed and paraphrased below

Create the file /etc/polkit-1/localauthority/50-local.d/45-allow-colord.pkla (using your editor of choice and sudo)and add the following contents

[Allow Colord all Users]
Identity=unix-user:*
Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile
ResultAny=no
ResultInactive=no
ResultActive=yes

We now need to clear any crash dumps from the workstation

sudo rm /var/crash/*

You should then be good to connect to using whatever RDP client you prefer… I like Remmina myself but each to their own.

Technical CV writing is hard

Wed, 15 May 2019 00:00:00 +0000

Recruiting people is effing hard!

That’s all… I’ll get back to reading through CVs now and let you get on with your day!

Medicines Discovery Catapult is, at the time of writing, recruiting Software Engineers, and as “Head of” it falls to me to start filtering through the CVs that land in my inbox. But what a mess! I get all sorts, from 1-page masterpieces that look amazing and all glossy, but tell me nothing about the individual to 10 pages of war and peace that have so much in it spanning 20 years of experience that 70% is actually completely irrelevant to the job they have applied for.

So now I get to say that this is what the perfect portrait of a CV should look like… I’m sorry to disappoint but I don’t think there is such a thing as a “perfect” CV. It’s far too subjective and open to interpretation. Instead, I’m gonna rip apart my own CV and explain why I chose to write it like I have and justify reasons why I would expect to see similar things on the CVs I am forced to read.

If you want the “TL;DR” version you can find a copy of my CV at https://docs.google.com/document/d/1MM_6nXIVU_wbrvkhdGL5xaJEwc46L7JblUqUot_g1ec but if you have gotten this far you may as well stick around and read the rest. I promise it won’t take long.

The Basics

OK! let’s start with some general observations I follow with regard to my own CV. Then we can get into the nitty-gritty of how bad mine is!

Recruiters

Love them or loathe them they are a part of the recruitment ecosystem and they will help you get hired. When you send them a copy of your CV bear in mind that they will most likely try to squeeze it into a format that will include their own branding and a cover sheet with some pertinent details on it. They may also then start redacting information off your CV in an effort to anonymise you. It is also possible that they will use the same CV for multiple roles meaning you will need to insist that they send your tailored CV if you are really bothered about winning a specific position.

Make sure to discuss this with your recruiter before having them put you forward for a role you are keen on.

Keeping Up to Date

Design your CV so that it is easy to tweak and update! Keeping your CV up to date is essential, and I don’t mean just adding your last role when you finally get fed up with your current employer abusing your good nature and rage quit!

Keeping your CV in top form means that you need to review the entire thing whenever you make an update. As your career progresses you will find that your perspective will change on what previous roles consisted of and how they would impact/reflect on the position you now desire.

You will also find that an up-to-date CV is easier to tailor to any specific job you might be applying for

Prettiness

I know his name is Riccardo, but I’m struggling to focus on anything else

Having a pretty CV is great! and if creativity is relevant to the vacancy you are hoping to fill then go for it… make it gorgeous. It is however very subjective so make sure that you understand what your potential employer is looking for. Sometimes less is more, especially considering they may be scanning through hundreds of CVs. If they have to spend half an hour looking for something specific, then it’s gotten lost in all that creativity. If you want to see some gorgeous-looking CVs take a look at https://weare.guru/creative-cvs/ all of them are creative and beautiful…

But… they could take 20 minutes, or more, for someone to read and extract the right information. Which will waste a lot of time for your potential new boss. Plus we are talking about technical CVs and I am not the most creative of individuals, so keeping it clean and well formatted with consistent fonts usage and sizing

Length

I’ve heard a lot of people say different things about how long your CV should be. 1 side of A4… 2 sides of A4 but only if it’s printed double-sided, even “length doesn’t matter” because the more information you put in the better.

I would recommend taking a middle-of-the-road approach. Keeping things concise is paramount… but if you need 3 or 4 pages then that’s OK… as long as you make the content captivating and interesting for the reader, that is what matters

PDF

This one feels like it should be a “no brainier”… Make sure to submit your CV as a PDF!

There are two very good reasons behind this. First, it ensures that the reader will view it exactly as you intended. If you send it over as a Word, Google, OpenOffice or other such documents, then you are not guaranteed the reader will be using the exact same tools. I for one don’t use Word and an awful lot of presentation is lost because someone used a fancy word feature or have a font that I can’t get hold of.

Second, While it is not impossible it discourages recruiters from tampering with it and spoiling all your hard work. I have known only a few recruiters in my time that are willing to learn (or pay for) a good PDF editing suite. It is possible for them to alter things, but they tend to have to jump through hoops.

Beginnings

And so we have the start of My CV. Looks pretty boring, doesn’t it? Black text on a white background. Nothing fancy!

We have some basic contact details that can be used to contact me and a very short profile statement. That is all I feel an employer needs to see before we get into the next section of my CV. On the surface, it doesn’t actually say much but lets scratch a little deeper.

Font

I have specifically chosen a font that I think looks clean and professional, With a nice easy typeface, it becomes easy for an employer to scan the CV. I keep my CV in Google Docs so I went with Raleway, which I think is nice, clean, professional and easy to read.

I’ve also chosen consistent font sizes and spacing;

16 for the main title
14 for headings
12 for subheadings
11 for everything else

Limiting Liability

Middle-aged, liberal, heterosexual white male programmer Swipe right to Hire Me!!! Please!

It may sound odd but I put as little personal information in my CV as possible. Hence why just my name and contact details exist. I don’t mention my age, gender, race, driver status or political preferences. People come with inherent biases, that is just a fact of life. so putting as little as possible negates triggering these biases.

I’ve seen people put all sorts of information into CVs… a personal bugbear is photos. As they can introduce a massive amount of opinion in the eyes of the beholder. Think Tinder but for recruitment!

So unless specifically asked for I would strongly recommend keeping it to a minimum and using the space it would consume in selling the things that matter.

Profile statement

1 paragraph! that’s all I needed to say regarding what is effectively a personal statement. This is not a UCAS application and I don’t need to detail a million things about myself. What is to come next will be the real sales pitch

My profile statement focuses specifically on what my future employer is going to get from me if they hire me. Passion! It is a statement of intent, specifically written to demonstrate that I will strive to bring my “A game” to anything I do in the future and also that I intend to encourage those around me to do the same. In effect, it is the opening line to what is a sales pitch.

I could expand on this to state other goals and ambitions but that would just detract from the main objective of the CV. Plus we will get an opportunity to elaborate later on in the recruitment process.

Skills

Now we come to the meat and bones of the CV. This is the headline! the bit that you will tailor the most in order to impress whoever is scheduling those interviews and guarantee you a chance to shine. If you can make this captivating enough for the person you need to impress they will then be more than happy to read the rest

My CV has a LOT of skills listed. This is mainly because I’m a show off more than anything else and in reality, I will tailor this heavily to suit whatever role I’m applying for. For example, if it doesn’t have a requirement for managerial skills I would drop that section entirely. Depending on the required attributes and skills asked for I would happily add/remove bits to any section of my skills.

With different types of skills, I will define them differently. As you can see I am quite generic with Managerial skills. A lot of people understand these and are looking for confidence in your ability to do certain types of tasks. Technical skills define different aspects of what I do that are not specifically tied to languages. It can be very difficult to be curt and to the point here. I favour labelling a technology area and then listing a few of the most prominent or recent items from my repertoire. A bad example on my CV is databases… I can use quite a few as you can see, but in reality, I’ve been greedy by listing MySQL, MariaDB & Percona… they are all effectively connected to each other, but my need to show off is far too much for me to resist.

Experience

When I first started out as a Dev I had a hiring manager who once told me that most of the time the main thing he wanted to see was;

did the applicant have the skills he wanted
what was the candidates’ personal assessment of their ability
how much genuine commercial experience do they have

Three really simple things that stuck with me. So the next time I came to write my CV I added a table for Skills which had only a few things on it back then but has grown massively over the years. I’ve tweaked and tailored it over the years, adding and removing things that were relevant as needed.

By quite clearly labelling my own perception of my skills, I give the person reading it an indication of my potential value. It allows them to tailor the interviews and technical tests to fit me as an individual. It also acts as a double-edged sword as the higher I grade myself the more chance of falling flat on my face when I get asked a question I can’t answer… So it’s always better to try and be accurate, but also not to be timid, there are always other competitors in the recruitment race.

I favour using really simple words to identify my skill level; beginner, intermediate, expert and expert+. It makes it simple for the reader to gauge and also allows me to mix the terms around

Stating how many years of commercial experience you have lets the reader understand your commercial experience. The number of times I’ve had recruiters do a typical keyword match and see that I mention c# once on my CV and then try to wave a job advert under my nose. I did .NET for 6 months 10 odd years ago. I legitimately can claim that experience, but it should not be the core on which I base my next adventure.

As a useful side effect, by stating a duration in years prompts me to review and update my CV regularly to keep it up to date.

Previous Employment

By this point, I’m hoping that I’ve captured the readers’ imagination. That they now envision a Development God has graced them with a CV worthy of filling any role they have…

And then come back down to earth with a bump! Next up in the firing line is Work Experience. Here I define some of the previous positions I have held in my illustrious career in somewhat chronological order. I say some because I have a general rule of limiting what I put on the CV to either 10 years or 10 positions, whichever comes first. This hasn’t been a hard and fast rule over the years, and it’s fine to flex in order to suit the roles I’ve gone for. There is no reason whatsoever though to go so far back in time as to describe my time as a pot-washer when I was 17.

In some cases, I’ve also omitted some things from my CV. Somewhere in and around 2012 - 2015; I happened to found and run a co-working space in Manchester City Centre. It was an interesting venture that I’m really proud of and am glad to say is still running even though I’m no longer a part of it. But in truth, it does not add anything of value to my CV for the positions I plan to go for in the future.

To the point

I don’t like big sprawling paragraphs of text if you haven’t already got the gist. So for me, bullet points are the way forward. I try to keep it fairly obvious and detail achievements and document things I have done, and not the tools I have used.

I try to explain why I joined the company and what my purpose was. I highlight important successes and demonstrate improvements I made to the company. Short succinct sentences should be chosen to illustrate aspects of your accomplishments that actually relate to your future goals in your new position.

By doing it this way you also make it easier to tweak and change things without having to restructure a whole piece of prose in the future.

That said my own CV has a glaring exception! When I was doing freelance work I was not able to accurately describe everything I was doing due to some pesky Non Disclosure Agreements. So instead I have a simple paragraph providing a positive high-level explanation of what benefits I brought to my clients.

Formatting

There is a lot of information condensed into this section which makes could make it hard to read if left unformatted. I stuck to the font sizes I had chosen previously and decided to use a more subtle combination of indentation, italics, underline and bold to make it more pleasing to the eye and easier to scan.

Each company name acts as a subheading with a font size of 12. I take a little liberty and include on the same line the dates that I was with them. This gives a clear timeline of events that a manager can then refer to quickly when they need it, such as in an interview.

Indenting everything under a subheading makes it easier for the reader to separate out the content easily. I include an address for the company. I am not actually sure why if I’m honest, it’s just something I’ve always done. I put this in Italics mainly because it’s an aside to the core of the information.

The job title will come next in bold of course to help it stand out. Followed immediately by the relevant bullet points. My last role obviously plays the most prominent part as it’s my headliner. In the case of Medicines Discovery Catapult. I’ve held two roles, “DevOps Engineer” and “Head of Software Engineering” so I break these into their own sections within this piece of experience. Prior to that, I was with Wakelet and here I merge the two roles I held there in order to conserve space.

Education

The bulk of the hard work is now done! We have put in the sales pitch and hopefully, we are close to being invited to an interview. Time to put in some supporting information

This may come as a surprise to some people but I didn’t do the whole University thing… I mean I lived in a University city and frequented the student union bars, but was never actually enrolled in a course. To this effect, I bolstered my CV when I started out by going and obtaining some (now significantly outdated) professional certifications.

Regardless of my lack of educational achievements I would always recommend keeping it simple unless it is your first position and you have no work experience (writing a CV in that situation probably needs to be another blog post entirely). List them in chronological order with some dates and a summary of what you obtained… no one needs to know that I completely fluked getting a GCSE in Art.

Were I to have a Degree I would obviously have the institution and dates in there along with my final marks. I would also look at listing the modules I completed that are relevant to my career, providing I did a Computer Sciences degree and not something like Biology. Again, it’s all about keeping to the point and providing specific information to bolster everything else you may have done.

Wrapping up

Time to finish off with a little bit of something personal. I don’t want to spend too much time here but I want to show that there is more to who I am than just work. I decided to keep it simple, a simple list of things that I enjoy doing in my spare time (When I have any, being a father of 3). The idea here is that these can become conversation pieces with the people that may be interviewing you. I’ve ended up a number of times having interviews where I talk about Scouts.

The final flourish here should be something simple but gives someone a helping hand at learning more about you if they feel so inclined. I include a link to my blog and my GitHub account, but you could include anything that may be relevant.

Last but not least… References are always available on request. My referees are varied and have changed over time. Adding them to your CV doesn’t actually impart any other information that could get you hired. Name-dropping is not the right way to get a job.

Conclusion

OK! so it’s not a pretty CV by any stretch of the imagination. It’s a bit on the long side in its full un-tailored, raw form, though not as long as it could be if I wasn’t being diligent in how I want to present myself. But this is the format I’ve used as my CV for at least 15 years now and I would say that I’ve been really successful in getting interviews out of it. I would say I get at least an 80% success rate of conversions from seeing my CV to a first-stage interview (watch me now jinx myself for the future).

A CV will never get you the job! It’s all down to you excelling in an interview situation and proving how awesome you are and that you can do everything you say you can on your CV. All it is meant to do is get your foot in the door. Hopefully, this breakdown of my CV will help you to take a look at your own CV and work on ways to improve your chances.

Dell DisplayLink D6000 & Ubuntu 18.04+ Issues

Tue, 14 May 2019 00:00:00 +0000

I love Ubuntu… I’m pretty fond of dell kit too!

So I was rather chuffed when I started working at Medicines Discovery Catapult because they let me have both. When you look at my desk it looks like it could be an advert for Dell. Laptop, monitors, dock, keyboard and mouse…. its great when you have a corporate account with a Dell reseller

However while I’ve had a lot of success with the D3000 DisplayLink dock on Ubuntu I found that I’m now having to deal with the upgraded D6000… which doesn’t play very nicely with the more recent versions of Ubuntu (we are talking 18.04 and later)

I kept finding that after a random amount of time the D6000 would randomly seem to power down… I would lose the screens, audio, networking and USB. and the only way I could fix it is to unplug it from teh laptop and plug it back in. Not ideal, especially if I’m in the middle of a video call or debugging something on the net

Being the kind of techie I am my first port of call checking my logs… but I couldn’t see anything that would cause this random disconnect. So off to google I went… eventually I found a lot of information telling me it was part of power management causing things to start powering down… In this case it implied that it was something trying to suspend USB… which sounded really plausible!

So a little more research suggested that I should be using laptop mode tools to disable the ability for USB to be suspended. I gave it a go, though I was dubious as in my mind I shouldn’t have needed to install an additional package (albeit a great one for tweaking your power management on a laptop running Linux)

Alas no joy! And I had too much to do to start debugging in depth and ripping apart other peoples code to figure it out.

What did I do? you ask. Well, I just put up with it for a few weeks, but gradually it began to grate on my nerves. However there was that one day where it didn’t turn off… and that left me perplexed… I checked if any updates had been applied in my last apt update && apt upgrade … nothing…. it then dawned on me that I had plugged in the headset I used for conference calling into the audio in/out on the dock instead of directly into the laptop.

Now I had a little more information I was able to deduce (with googles help) that the laptop was actually suspending USB, but that the trigger was actually pulseaudio. At this point it becomes really easy to solve the problem.

Solution

Edit /etc/pulse/default.pa using your preferred editor (and sudo)

Find the line

### Automatically suspend sinks/sources that become idle for too long
load-module module-suspend-on-idle

And comment it out and save!

Lastly, because its run as a user service you need to restart the Pulse Audio daemon using the command

systemctl --user restart pulseaudio.service

or you could just logout and back in again

A reboot and a legacy moniker

Tue, 26 Feb 2019 00:00:00 +0000

So… my last post was a good 2 years ago now…. Hi how have you been?

It’s been a very busy couple of years with a lot of stuff shifting in my personal life meaning things inevitably take a back seat. However its been long enough that I needed to give myself a new start and see about blogging again and getting back into speaking again.

Still PHPBoyScout?

Since 2016 I’ve jumped around in my career a lot! This has exposed me to a lot of new languages and tech, all of which has been awesome but it now means that the moniker of PHPBoyScout probably isn’t accurate all that much any more. That said I still have a love for PHP and my first thought when presented with a new challenge is still “How would I approach this using PHP?”, so I think that means I can keep my twitter handle for a little longer rather than re-brand myself :-D

I’ve always had a penchant for picking up languages pretty quickly and have been an advocate of the idea that as long as you can think in the right way then languages can be learnt easily enough.

Employed and Employable

October 2017 saw me take on a new role with a company called Medicines Discovery Catapult. They are a grant funded not for profit organisation that is specifically focused on shaking up the medicine discovery pipeline by helping SMEs & CROs innovate and collaborate.

My role is very broad as I appear to have fallen in to the role of Head of Software Engineering very quickly. Though I started as a DevOps Engineer it has evolved very quickly. We work with a wide array of languages and tools and nothing is off the table if it helps to solve the problems we are working on. Currently we actively have code being written in Python, Node, Go & Scala (I live in hope that I’ll be ale to bring PHP into that mix eventually). Vast data sets and AI are the name of the game with my team of engineers helping to support the data scientists and Informaticians.

At the time of writing we are also hiring so if you are curious take a look at https://md.catapult.org.uk/about/careers/ and see if you like the sound of what we are doing and the challenge on offer.

Moving forward

The plan moving forward is to post about the tech we are working with, talk about the types of challenges we are facing and also maybe restart my public speaking career. Stay tuned for more and if you don’t hear from me soon then give me a nudge

Project Slayer: The Critical Path

Tue, 26 Feb 2019 00:00:00 +0000

The first of my new round of talk abstracts! In all honesty this isn’t a talk but something more that came out of a very drunken Saturday night at #phpbnl19. There were a bunch of us sat talking and somehow the topic of D&D came up which sent my mind racing with this idea… By the time 2am rolled around I had a fully formed idea along with some willing players to help with the idea. Now I just need to find a conference willing to take a chance on it! If you know a conference that would be interested let me know

Description/Abstract

Our heroes have just completed our latest quest! Having successfully delivered the latest iteration of their project they are approached by the aged and holy sage “PeeEm” with a new quest! How quickly can they implement the sacred and forgotten art of “Lo-ging” into the codebase

Do they accept? If they do will they be able to defeat the trials and tribulations that await them? Will they find treasure and glory, or suffer defeat at the hands of the vile and depraved Stakeholder?

What will people learn

The only way to find out if our heroes will complete this epic quest will be to join us and see if the dice of fate will be kind to them!

Along the way we will will learn some truths about feature implementation and how our heroes handle the challenges that lie ahead. And hopefully gain enough experience to level up!

Additional Information

This is an extremely unique talk! It takes the form of a live Dungeons & Dragons game. I will be on stage playing the part of Dungeon Master looking to guide our team of adventurers through the process of delivering a new feature for a project.

Our heroes currently consist of a heroic bard who will be inspiring our heroes, and audience, with ballads of past glories. A warlock with the demonic power to fork and merge code like no other in existence…. Our sorcerer has the innate and wild magic of Fire(base)! Finally our team is held together with the support of our cleric, worshipping the ancient god Rasmus.

The outcome of this adventure will genuinely be determined by the roll of the dice! It will be a game of 5th edition D&D that we can probably fit into an hour… but 2 or more would be better and far more fun.

Audience participation is expected! Cosplay is hoped for! I will, of course, be in full scout uniform!

Monty Python explains why your project failed!

Sun, 07 Feb 2016 00:00:00 +0000

As part of the attempt to develop my profile as a speaker, I’ve realised that I sometimes need to explain a few of my current talk abstracts a bit too much. This is mainly due to my lack of experience writing them and that the majority of my current talk ideas cover large topics that are not as technical as I would like.

My favourite so far is one title “Python explains why your project failed”. This is a tongue in cheek talk which aims to poke fun at the Developer, PM and of course the client!

The TL;DR

This talk has yet to be accepted by anyone… but will be eventually I hope. In the mean time I wanted to share some of the funny thoughts and comparisons I’ve had coming up with the content for the talk. I plan on doing this by writing a series of blog posts one for each topic or sketch that features in the talk.

The Abstract

Python is fantastic! If you haven’t seen it you really need to. Its simple, elegant, powerful and gives you an amazing perspective on what we do as Developers, it is also hilariously funny…. Yes Funny!!!!

Wait! You thought I was talking about Python the programming language didn’t you? I’m sorry to tell you but we are actually talking about the most awesome of British comedy acts… Monty Python.

Throughout this talk I will take you through the development life-cycle of a project and use the Comedy of Monty Python to illustrate both the Good and the Bad (mainly the bad) aspects of our industry. All the way from Client introduction, requirements gathers, spec writing, team selection, planning and scoping all the way through Development to Testing, Delivery and Support!

The Delivery

This is a little harder to explain as I’ve not given the talk (yet) and I don’t think I could ever match the delivery better than the Pythons themselves.

However I have been known to dress for the occasion, so it’s quite possible that you may find me standing on stage in a red cassock at some point.

I’ve also managed to convince @phpcodemonkey, who is as big a Python fan as myself, that this talk should really be performed as a 2 man show, rather than me monologuing at a room full of people.

The Sketches

Due to the prolific variety of skits and sketches that Pythons created I found it extremely hard to select the few needed to fill a single talk. I have however managed to select a few and will change them around from time to time to suit the audience. I’ve provided a short list of a few of my favourites Sketches and a couple of words describing what they explain:

Spanish inquisition - Client Indecision
Dead parrot sketch - Stubborn Project Managers
Ministry of silly (array) walks - Tool Selection
Brian’s Latin Lesson - Planning and Preparation
We demand a shrubbery - Demanding the Impossible
Black knight - Solution fixation / Code Blindness
Camelot Song - Stakeholder Morale
The People’s Front - Team Fragmentation
The Silly Job Interview - Stakeholder Communications
Four Yorkshireman - Rockstar Developers
Argument clinic - Product Delivery
Architects sketch - Taking Shortcuts and Cutting Corners

The Finale

These are just a few of the potential topics I will be looking to cover in the coming posts, but while your reading them I want you to remember to…

https://youtu.be/WlBiLNN1NhQ

Are you a Good Code Scout? - NomadPHP lightning talk video

Wed, 06 Jan 2016 00:00:00 +0000

https://www.youtube.com/watch?v=Tt0lnauF5lI

Just before the Christmas period I was lucky enough to be able to give my “Are you a good Code Scout?” talk as a lightning talk for NomadPHP. Here is the video that was recorded from it.

Using Gmail aliases with Evolution

Wed, 06 Jan 2016 00:00:00 +0000

If your anything like me you have a large number of email aliases that you use with Gmail which is great. However I use Evolution as a mail client more often than not when using Gnome3 as a desktop.

It’s very easy to set up Evolution to create separate outbound email accounts that you can use for handling all of your aliases. It doesn’t yet support OAuth2 as an authentication mechanism for any account that is not set up using the built-in Gnome Online Accounts integration.

This is a real pain as Google have disabled the more common ‘plain’ and ’login’ authentication mechanisms for use with an SMTP only account. Meaning that any time that you try to connect to smtp.gmail.com:587 with STARTTLS you will get some form of error message to the effect of “Bad Authentication”.

Hopefully I’ll find a workaround at some point in the near future or Evolution will add the facility to enable OAuth2 as an available authentication mechanism.

In the mean time there is a workaround if you visit https://www.google.com/settings/security/lesssecureapps you can enable these less secure authentication mechanisms allowing you to once again connect and send email via email addresses using SMTP

A metaphor about PSR-7 and Middleware for non-developers

Thu, 08 Oct 2015 00:00:00 +0000

Never one to shy away from coming up with a metaphor for explaining something technical I found myself having to come up with one on the spot for PSR-7 and Middleware while at the recent PHPNW15 Conference.

Normally my brain will come up with something completely inappropriate but this time round I found I quite liked the imagery that came to mind.

If you would like to find out more of the specifics about PSR-7 you can take a look at http://www.php-fig.org/psr/psr-7/ which will make a better job of explaining it than I could ever do.

Now on to the metaphor

Imagine a house on fire, a bizarre way to start I know but bear with me. The nearest well with water that can put out the fire is 500 meters away! We then have a human chain stretching between the well and the house with a bucket going back and forth between trying to put the fire out. So lets break this down, the house represents the internet, or more specifically you and your browser. The fact you are on fire means that you are desperately needing water to quench the flames. At this point you send an empty bucket which represents your “request”, along the human chain, which in itself represents the application, to the well.

At the start of the chain the bucket is pretty normal, it’s a bucket of course, its round, made of wood with a rope handle, lets say it has a small leak in it.

As it travels down the chain it’s passed from person to person, everyone in it has the opportunity to do something with the bucket, or not as the case may be and could just pass it to the next person in the chain. Others may attempt to fix the leak in the bucket, someone may choose to replace it with a metal bucket, change the handle or make it bigger. Regardless of what may be done to the bucket in essence it remains a bucket.

Inexorably the bucket will continue to move down the chain to the well. When it reaches the well it changes state because now it has been filled with water. All of the interaction with the bucket thus far, mean that what happens at the well could vary depending on the changes have been made . If its been made bigger, for example, it could be filled with significantly more water, if swapped for a metal one it could imply that the bucket descends the well to get the water quicker because its heavier. Either way it is filled with water and begins its journey back towards the house.

Again it passes through the hands of each person in the chain, but now that its state has changed it now has the opportunity to be modified again. Someone may empty some water out as there is too much in the bucket, others may say that there is not enough and send it back down the line towards the well to be refilled. Either way the bucket continues to change hands over and over until it reaches the house and the contents are thrown on the fire to complete the request for water.

During this whole time the human chain could have been in flux. Some people may have swapped places, left the chain, added to the chain, some extraordinary people may have played leapfrog in the chain and appeared to handle the bucket more than once. Regardless of these changes the chain remains and continues to pass the bucket from one person to the another as long as the requests for water keep coming.

This, in the simplest possible form, explains PSR-7 and the concept of Middleware.

The bucket remains a bucket because PSR-7 says that is what is needed to complete the request for water, it also defines how you should interact with it regardless of what modifications have been made. If the bucket cant be used according to how PSR-7 describes a bucket to be, then the middleware can’t complete the request.

Every person in the human chain can be classed as a piece of middleware all the way from the house to the well and back again. If at any point someone enters the chain that doesn’t agree that the bucket is a bucket or doesn’t know how to handle it, then the it is dropped on the ground and the request fails.

Badges & Stickers

Sun, 04 Oct 2015 00:00:00 +0000

One of the most prominent things I’ve been asked about regarding my promoting being a Good Code Scout, is where can we get the badges?

Following on from a number of questions and subsequent tweets about it

https://twitter.com/stuherbert/status/650591775732670466

Well… I’ve decided that (providing I can get permission from all the right people) I’ll start creating a range of Badges & Stickers for you to earn as a Good Code Scout.

So if your interested in having some stickers or badges let me know using the form below and if I get enough interest I will most definitely get some made up for you.

Update

Unfortunately I’m have no more stickers left, Though I’ll be working on diversifying some of the designs in the near future I wont be looking at ordering any more for a little while yet. Keep your eyes on twitter as I’ll most likely post there when they are available again.

Wow... What a Conference

Sun, 04 Oct 2015 00:00:00 +0000

So I attended the PHPNW15 conference this weekend and what a weekend. I’ve been an attendee of the conference for a number of years and have always enjoyed it immensely. However this year turned out to be something special.

Following on from my first ever appearance as the PHP Boy Scout I decided to submit to the Unconference at PHPNW15.

It was a good talk, an extension of the previous lightening talk I’d given and felt really good to give. Unbeknownst to me however there was mischief afoot. Normally the Unconference talks are rated by the organisers and the one that they selected as the best gets to have a guaranteed slot in next years PHPNW conference. All of which I had genuinely either no idea about or had forgotten had happened in previous conferences,

As you may have guessed from the fact this post exists, I ended up winning that slot.

However…. it appeared that a Speaker had taken ill at the last minute and couldn’t make it at which point I was asked a mere 3 minutes before it was announce that I was also going to be given the hangover slot on track three for the Sunday sessions!!!!!

Suffice to say I had an interesting evening to say the least, in preparing my talk for “the big time”

Amazingly I felt really calm about everything, and even had a good chuckle about managing to find some props to help break the ice!

Everything is ready, I’ve practiced, knowing my talk was going to be a bit short… but that was ok considering the short notice, and I had plenty of anecdotes I could use as filler. I’m sat there waiting for the moment I have to put my head above the parapet and all of a sudden……..

nothing

My mind goes blank!

The long and the short is that I survived, and the feedback I have had has been amazing and I’ll be taking all of it on board to make sure that next time its even better!

The recordings should be available in the near future so when they are I will share a link so you can judge how it went for yourselves. In the mean time I’ve published the revised slide deck for you on slideshare.net/phpboyscout/are-you-a-good-scout-phpnw15-track-3

I’m hoping that I can now find some opportunities to practice for my slot at #phpnw16

[slideshare id=53508432&doc=areyouagoodscout-151004073451-lva1-app6892]

What is a PHP Scout

Fri, 02 Oct 2015 00:00:00 +0000

Recently I’ve had a lot of people asking me what a PHP Scout is! I thought it would be a good opportunity to explain.

To understand what a PHP Scout is it helps to know a little of the background basics of Scouting in general. Knowing this helps to make it easier later on as well as we draw some direct parallels. If you would like to investigate more about the history of Scouting you can find a good starting point at http://scouts.org.uk/about-us/history/.

For now I’m going to give a tl;dr version;

Scouting started in 1908 as a movement for training young people to encourage them to develop physically, mentally and spiritually by Robert Baden-Powell. Over the next 100+ years it has evolved to encompass people of all ages, races, colours and creeds to get involved and try to be the best they can be.

The primary ethos of the movement today is to bring Everyday Adventure to young people and this is achieved through a comprehensive programme scheme that is designed to touch on all aspects of that young persons development. This is then rewarded in a variety of ways with the primary reward being the experience itself, the awarding of badges also strengthens then sense of achievement and desire to work towards the next goal.

All members of the Scouting movement are required to make and frequently renew a promise:

On my honour, I promise that I will do my best to do my duty to {insert deity/monarchy here}, to help other people and to keep the Scout Law.

The key part here is I will do my best. Scouts are continually encouraged to improve themselves in everything they do.

The Boy Scout Rule

Lets start with something easy! There is a pretty common piece of guidance that gets bandied about in a lot of different circles that is normally referred to as “The Boy Scout Rule” which promotes leaving things better than you found it. It came about as common practice for scouts to always try to leave a campsite cleaner and tidier than when they arrived so that its in a good state for the next group.

This is quite generic but can easily be made very specific to us as programmers:

Leave the codebase better than you found it.

So what do I mean by this? Ultimately I mean that regardless of the state of the code you are working on you should always try to find a way to improve it.

This can be something as simple as;

refactoring the code to make it more readable
adding some docblock to explain a file/class/method/function/variable
create a Readme file or add some documentation
remove obsolete code, old backup files, stray files, unused components
fix a failing test
write a new test even

Its not an exhaustive list at all but it gives you an idea of what kind of things you can be doing to improve your codebase. Any good Scout group leaving a campsite would also make sure to put out the fire and close the gate on your way out. Which is exactly what you should be doing by making sure all your Acceptance, Functional, Integration & Unit tests pass and writing a good commit message.

Right tool for the job

In every activity that a Scout takes part in they are always taught the correct way to work with their tools and equipment, such as how to use an penknife property. They are then encouraged to explore different ways of using those to achieve their goals. This is no different for a PHP Scout, by knowing how to use their languages and tools properly they can then use it to maximum effect.

Self Development

As a child you assimilate massive amounts of information every day that helps you to grow and develop. This is creatively harnessed by Scouts through a variety of different activities that are designed to help them learn new skills that can help them grow as people.

Now that we are older our brains don’t have the same capacity to soak up that volume of information. But that doesn’t mean we shouldn’t be trying! A good PHP Scout will continually strive to push the boundaries of what they know, to pick up new skills that can be used to make them more capable. This can be learning a new technique, or language or tool be it via formal training, conferences, social events or even just a good Google.

Granted the Scouts are rewarded with some cool badges, but I’m sure its only a matter if time before some entrepreneurial PHP Scout decides to start creating some achievement badges of their own (see http://phpboyscout.uk/php-scout-membership-badge)

Helping Others

We’ve all heard the adage of a Scout helping someone across the street. It’s a somewhat stereotypical example but extremely apt as it highlights that they are encouraged to take into consideration other peoples needs and to provide assistance wherever possible. Modern Scouting however goes far beyond aiding with avoiding getting run over on a road.

By encouraging Scouts to not only help individuals, communities and groups we make them more considerate of the needs of others as well as developing their sense of self. A fantastic example of this is the 2015/16 initiative A Million Hands which promotes finding ways to identify the needs of others and to take action to provide aid.

This is a fantastic trait to be teaching children and is something that any good PHP Scout would applaud, and would then go forward to do the same things but with the development community. This can be something as simple as;

helping a colleague at work (without being told to do so by your boss)
organising an event with a local user group
contributing to an open source project

All very simple stuff to do and all it takes up is a little of your time! Where is the reward? I here some of you ask! I would say that the act itself is its own reward, and in reality that is true as when working helping others your generate some very positive Karma which will eventually be paid back when the day comes that you yourself need some help. You should (hopefully) also have an opportunity to maybe learn something new and improve your ability to communicate, a soft skill yes, but essential to your growth if your are striving to be better than you are now.

Problem Solving

The Scout motto is a very simple two words… “Be Prepared”, but be prepared to do what? Its quite open ended really it could be anything at all! I like to think that its nearly impossible to be equipped with every possible skill and tool possible to meet any and every task you will encounter through life, though being a Scout does try to help arm you with as many as possible.

Yet as a PHP Scout we should always “Be Prepared” to solve problems. If we are doing our jobs right we should be looking to solve problems through the solutions we provide every day. Quite often I talk to developers and hear them make pigeon holing statements like

I am a WordPress developer

And then complaining that they are bored at work or that they cant get jobs working with anything other than their chosen platform. Now this infuriates me as a PHP Scout would never do this, when asked they profess loud and clear

I am a Problem Solver

and be prepared to prove it by making sure they are prepared by knowing more than one or two platforms or frameworks or even programming languages. This can be encouraged by actively seeking “problems” that you can solve with tools and techniques you are not familiar with.

This is echoed throughout the challenges that are presented to Scouts, where they are tasked with solving a practical problem such as putting up a tent without any instructions, the best way to light a fire with two sticks and a bit of kindling, how to cross a stream with only a few bits of wood and rope. In solving these types of problems the Scouts not only receive the obvious of shelter, food, heat etc but they also become more prepared for the next time a similar scenario presents itself.

Team work

No man is an island as the saying goes and the same goes for being a Scout. By being organised into lodges,packs & patrols they have a ready made team to work with and the only way they can progress is to work together. They may not like the people in their team (and as a Scout Leader I will quite readily admit to putting Scouts into groups with others they may clash with).

We may all have teams that we work with as part of our Jobs, and a PHP Scout will take the opportunity to work with as many different combinations of teams as possible both in and outside of the workplace. By diversifying the people you have to interact with you develop a broader understanding about the problems you may be trying to solve.

This can then be expanded upon as mentioned previously by then branching out into the community and working with user groups and opensource projects.

To Summarise

The ethos behind the Scouting movement is a solid foundation not only for children aged 7-18 but for everyone. By being a PHP Scout you strive to keep improving your ability to create great code, solve problems, work with others and in doing so become a better developer.

As with all Scouts they are Hard Working, Determined, Ingenious & Tenacious and so is a PHP Scout.

The PHP Scout Membership Badge

Tue, 11 Aug 2015 00:00:00 +0000

As the PHP Boy Scout I’m having some badges made and I wanted to introduce the all new PHP Scout Membership Badge.

This badge shows that you are more than just a PHP Developer but also a good PHP Scout. This means that you have all the qualities it takes to be a PHP Scout and will :

always leave the codebase better than you found it
help other PHP developers be good Scouts
get involved with your local User Group & PHP Community
Contribute to at least one open source project

If you would like to find out how to get hold of a PHP Scout Membership Badge then fill out the form below.

[contact-form to='matt@phpboyscout.uk’ subject=‘Someone wants a Membership Badge’][contact-field label=‘Name’ type=‘name’ required=‘1’/][contact-field label=‘Email’ type=‘email’ required=‘1’/][contact-field label=‘I will always leave the codebase better than I found it’ type=‘checkbox’ required=‘1’/][contact-field label=‘I will help other PHP developers’ type=‘checkbox’ required=‘1’/][contact-field label=‘I will get involved with my local User Group’ type=‘checkbox’ required=‘1’/][contact-field label=‘I will contribute to an open source project’ type=‘checkbox’ required=‘1’/][/contact-form]

My first ever public appearance as PHPBoyScout

Thu, 06 Aug 2015 00:00:00 +0000

So it’s finally happened!

I stood up in front of a group of developers and gave a lightning talk about how Scouting Principles should be applied to every day development.

[slideshare id=51344270&doc=areyouagoodscout-150806121954-lva1-app6892]

The amazing thing is that I didn’t get any rotten tomatoes thrown at me! quite the contrary in fact. Even with me doing the talk in full Scout uniform.

Now to see about finding some more places to speak and actually fleshing out the talk into something that can last a full hour and not just shy of 5 minutes.

Goodbye Dev in Charge

Wed, 05 Aug 2015 00:00:00 +0000

Over all the time that I’ve been a developer I’ve had people telling me that I should get in front of an audience and speak. However I’ve always suffered from a rather bad case of ‘Imposter Syndrome’ which meant my automatic response to those kind of statements has always been… I don’t really know enough about any one topic.

This is very true, I’ve spent a lot of my career learning a really broad swathe of technologies and techniques so I can turn my hand to any task that’s been presented to me so far. Even so people continue to try convince me that it would be a worthwhile pursuit.

Now that I work at Magma Digital I find that I’m often talking with @phpcodemonkey about all sorts of things and the topic of creating a talk came up while we were enjoying the most excellent PHP South Coast Conference. He knows I’ve been a Scout Leader for around 4 years now, and he suggested that I do a talk on the ‘Boy Scout Rule’.

I don’t know if he was serious or not at the time but it set my mind racing! This is a topic that I actually know quite a lot about!

So I’m now going to leave behind the Dev in Charge and have now rebranded as the PHP Boy Scout. I’ve already managed to pull together the basis of a talk on how Scouting principles can be used in conjunction with what we do as Developers and have a few other ideas that I’m going to work on over the next few weeks.

Fingers crossed I will be better at doing this kind of thing than I suspect I will be… but nothing ventured and nothing gained!

Free Open Source Website for Scouts

Mon, 12 Jan 2015 00:00:00 +0000

I’ve been a Scout Leader for a few years now and the District I work within have very little by way of internet presence. As a bit of a pet project I started building a simple Scout based website for them to use.

Its nothing too fancy, I created a simple module and theme for the Silverstripe CMS and have now put it into a GitHub Repository to share with the wider scouting community.

I chose Silverstripe because of the speed with which I could develop something usable as well as providing a super simple management interface that can be handled by users of all skill levels.

The module itself extends some very common extensions available for the CMS and makes them scout focused. Features include.

Customisable theme
Multi tiered Event Calendars
Customisable Group/Section Pages
Dynamic Forms
A reliable News/Blog system

These are just a few of the most obvious features and hopefully I will continue to add more.

I’m also offering to help any Scout Groups/Districts/Counties if they are wanting to use these modules and get their sites built and up and running for them free of charge.

If you want to take a look at the code and have a play yourselves you can find it online at

If you want to get in touch or would like more information about having a website built for you please fill in the form below.

[contact-form to='matt@phpboyscout.uk’ subject=‘Request for Scouts Website’][contact-field label=‘Name’ type=‘name’ required=‘1’/][contact-field label=‘Email’ type=‘email’ required=‘1’/][contact-field label=‘Scout Group/District/County’ type=‘url’ required=‘1’/][/contact-form]

Its time for a change

Tue, 02 Dec 2014 00:00:00 +0000

So… Its been a long time since I posted anything of any relevance. This is due to having been super busy with my previous company Zucchi.

However that has all changed now! After three and a half years of running my own company I have decided that its not for me. I gave it my all, but in the end I was becoming too much of a Salesman and I missed getting stuck in with code.

I’ve now moved on and have joined the fantastic team at Magma Digital who have been leaders in PHP software development for somewhere in the region of 14 years as well as heavily involved in the PHP community having been a essential part of the PHPNW user group and conference.

This means I should be able to pick up where I left off all those years ago and start being more active again.

See you soon

Flexbox cross browser

Fri, 16 Aug 2013 00:00:00 +0000

Despite having been around for a while and having been through a couple of revisions, its support across browsers can vary greatly. From “Candidate Recommendation” on Chrome/Opera, “legacy flexbox” on Firefox and no support at all on IE9 and earlier.

Making flexbox work consistently across browsers was a challenge for us on a recent project, but I have found a solution that seems to work quite well.

Below is an SCSS @mixin that will attempt to handle compatibility between CR and legacy cross browsers flexbox.

@mixin flex($content: flex-start, $items: stretch, $direction: row, $wrap: wrap) {
 $packLegacy: $content;
 @if $packLegacy == flex-start {
 $packLegacy: start;
 } @else if $packLegacy == flex-end {
 $packLegacy: end;
 }

 $alignLegacy: $items;
 @if $alignLegacy ==flex-start {
 $alignLegacy: start;
 } @else if $alignLegacy == flex-end {
 $alignLegacy: end;
 }

 $oritentLegacy: $direction;
 $directionLegacy: normal;
 @if $oritentLegacy == row {
 $oritentLegacy: horizontal;
 } @else if $oritentLegacy == column {
 $oritentLegacy: vertical;
 }

/** SAFARI **/
 display: -webkit-box;
 -webkit-box-orient: $oritentLegacy;
 -webkit-box-pack: $packLegacy;
 -webkit-box-align: $alignLegacy;
/** FIREFOX LEGACY **/
 display: -moz-box;
 -moz-box-orient: $oritentLegacy;
 -moz-box-direction: $directionLegacy;
 -moz-box-pack: $packLegacy;
 -moz-box-align: $alignLegacy;
/** LEGACY **/
 display: box;
 box-orient: $oritentLegacy;
 box-direction: $directionLegacy;
 box-pack: $packLegacy;
 box-align: $alignLegacy;
/** IE 10+ **/
 display: -ms-flexbox;
 -ms-flex-wrap: $wrap;
 -ms-flex-direction: $direction;
 -ms-justify-content: $content;
 -ms-align-items: $items;
/** CHROME **/
 display: -webkit-flex;
 -webkit-flex-wrap: $wrap;
 -webkit-flex-direction: $direction;
 -webkit-justify-content: $content;
 -webkit-align-items: $items;
/** NATIVE **/
 display: flex;
 flex-wrap: $wrap;
 flex-direction: $direction;
 justify-content: $content;
 align-items: $items;
} //@mixin flex

@mixin flexItem($width) {
 -webkit-box-flex: $width;
 -moz-box-flex: $width;
 box-flex: $width;
 -ms-flex: $width;
 -webkit-flex: $width;
 flex: $width;

 min-height: 0;
}

Firefox however only half supports flexbox (all revisions) and to get around this I would recommend using Modernizr as this will add the class “no-flexbox” to the tag. This provides us with a simple work around that allows non flexbox supporting browsers render correctly by using specifically crafted and targeted CSS for non-flexbox browsers

I found that IE9 support could be implemented using the flexie javascript plugin. In IE8 Modernizr will add the class “no-flexboxlegacy” which can again allow you to create targeted CSS that wont affect your Flexbox layout.

For a great overview of the “CR” of flexbox, CSS Tricks has an amazingly comprehensive coverage of the functionality here http://css-tricks.com/snippets/css/a-guide-to-flexbox/

Creating Custom Routes in Silverstripe 3.1

Wed, 31 Jul 2013 00:00:00 +0000

We wanted to create a Route to our custom Products Controller in our products module for SilverStripe 3.1, such as: “http://www.examplesite.com/products/”

However looking at the Controller Documentation it was not clear how to create a route without an Action being supplied. In our example above the action is not specified, as we just want to use ‘view’.

Solution:

Create a /_config/routes.yml file containing the following:

---
Name: productsroutes
After: 'framework/routes#coreroutes'
---
Director:
 rules:
 'product': 'Product_Controller'
---

The above will redirect any Url that starts with “/product” to our Product_Controller. Note that everything after the rule, so after “/product”, is used in the next bit for matching.

Now we need to add private static $url_handers to Product_Controller to match our path, so in this example we need to match “$Slug!” which will match “”. Note the ! means the slug is required. Of course we want to direct this to a specific action, in this case “view”, this gives us:

private static $url_handlers = array(
 '$Slug!' => 'view',
);

Now just add “view” to the $allow_actions and add the “view” function. This gives the final Product_Controller as follows:

class Product_Controller extends Page_Controller
{
 private static $url_handlers = array(
 '$Slug!' => 'view',
 );

 private static $allowed_actions = array('view');

 public function view(SS_HTTPRequest $request)
 {
 // Your action code goes here
 return $this->render();
 }
}

Handy note:

You can put ?debug_request=1 on the end of your URL to see how it determines which Controller to use.

Disabling Cache in Silverstripe 3.1

Wed, 31 Jul 2013 00:00:00 +0000

While working with Silverstripe we found ourselves having to run “?flush=1” a lot to clear the Cache. To switch it off, while you work, add the following to your mysite/_config.php:

SS_Cache::set_cache_lifetime('default', -1, 100);

Set up SilverStripe 3.1 using only Git (No Composer)

Mon, 29 Jul 2013 00:00:00 +0000

We recently tried to use composer to set up SilverStripe 3.1, but ended up with a dependency nightmare. In order to work around this we decided to make use of Git submodules.

First set up your Git repository and run:

git init

Next set up a site directory for the code inside your Git repository. Then navigate to SilverStripe Installer in your browser and Download a copy. Extract files, and copy contents to site folder. Now we need to add the CMS and Framework. Navigate in a browser to the Git Hub repositories for CMS and Framework. Now copy the HTTPS clone URL for each project and run the following, to add these as Git sub modules.

git submodule add https://github.com/silverstripe/silverstripe-framework.git site/framework
git submodule add https://github.com/silverstripe/silverstripe-cms.git <path-to-site>site/cms

Now delete mysite/_config.php and load the site. Follow the normal install instructions displayed and you will have a running version of SilverStripe 3.1

Enabling MYSQL_CLIENT_INTERACTIVE with Doctrine 2 on Rackspace Cloud Database

Fri, 26 Jul 2013 00:00:00 +0000

We recently ran into problem using Doctrine 2 connecting to a Rackspace Cloud Database using the MySqli Driver.

Problem:

We have a long running PHP script that can sometimes run for hours at a time whilst processing information. This script requires a connection to a database, but has long periods of inactivity where there is no actual interaction with MySQL. By default MySQL uses the “wait_timeout” setting which states, how long an inactive connection can exist before it is killed. This is normally fine with web pages requests, as it is usually a short lived request. Unfortunately you do not have the ability to alter this setting when using Rackspaces Cloud Database.

Solution:

When using the MySQLi extension you can create a connection in “interactive mode” by passing the “MYSQLI_CLIENT_INTERACTIVE” flag, which will then use the “interactive_timeout” setting. On Rackspace this is set to 8 hours!

Annoyingly Doctrine does not allow you to pass any flags to the MySQLi Connection. So we overrode Doctrine\DBAL\Driver\Connection with our own Driver which then allows us to pass a “flags” parameter through.

Feel free to look at some of the other helpful features in we have added to Doctrine 2 here: ZucchiDoctrine

Installing PECL extensions for Zend Server 6

Mon, 13 May 2013 00:00:00 +0000

Recently we have revisited using Zend Server for some of our projects and decided to give the new version 6 a chance to prove itself.

Overall its a big improvement over version 5. There are still some things that are extremely annoying but we have decided that we can overlook them.

However there is one thing that we couldn’t do without. By default you will find that a number of PECL extensions will not install out of the box (at least this is what we experience using the Debian based install).

To fix this you will need to make sure you install the additional packages in ubuntu

php-5.4-source-zend-server or php-5.3-source-zend-server depending on the php version you are using
autoconf
build-essential

Once this is done you should now be able to install extensions from PECL without too much hassle.

Better Output for MySQL Select Command Using \G

Wed, 24 Apr 2013 00:00:00 +0000

If you ever find yourself using MySQL via command line and end up with something like this:

And thought there must be another way, well here it is: Use \G instead of ; at the end of your select command.

For example:

select * from CHARACTER_SETS\G

Below is an image of the output from this select:

Happy Querying!

Introducing ZuQ - A Simple ZeroMQ Queuing Daemon

Tue, 19 Mar 2013 00:00:00 +0000

We recently had the need to create a queuing system to replace an implementation of RabbitMQ that was being used on a previous project. The reasoning behind this is that the requirements of the project required a very custom implementation of a queuing system that would drastically alter in architecture as the project grew and RabbitMQ just wasn’t going to fit the bill. However to start with we required something super simple and efficient that could be expanded and developed as required. After a little investigation and a lot of recommendation from others we decided to use ZeroMQ as our transport layer for that very reason, as we could build something which could span across multiple servers and was fast.

The diagram above helps describe our basic queuing system. We have a queue daemon that is continuously listening for connections and two clients, one that populates the queue and the other that retrieves from the queue.

The Clients

Each client is written in PHP and uses a 0mq socket to communicate with a service, in this case our queue service. We used a SOCKET_REQ type of socket in order to have a request/response communication with our queue service.

function client_socket(\ZMQContext $context)
{
 // SOCKET_REQ used to create a client that sends requests to and receive from a service
 $client = new \ZMQSocket($context,\ZMQ::SOCKET_REQ);
 $client->connect("tcp://localhost:5555");
 // SOCKOPT_LINGER = 0 Configure socket to not wait at close time
 $client->setSockOpt(\ZMQ::SOCKOPT_LINGER, 0);
 return $client;
}

public function injectIntoQueue()
{
 $context = new \ZMQContext();
  $client = $this->client_socket($context);
  $msg = "This is a message";
  $retries_left = 3;
  $read = $write = array();
  while ($retries_left) {
    // We send a request, then we wait to get a reply
        $client->send($msg);
        $expect_reply = true;
        while ($expect_reply) {
        // Poll socket for a reply, with timeout
            $poll = new \ZMQPoll();
            $poll->add($client, \ZMQ::POLL_IN);
            $events = $poll->poll($read, $write, 2500);
            // If we got a reply, process it
            if ($events > 0) {
            // We got a reply from the server, must match sequence
                $reply = $client->recv();
                if (intval($reply) == $msg) {
                $retries_left = 0;
                    $expect_reply = false;
                }
             } elseif (--$retries_left == 0) {
            break;
             } else {
               // Old socket will be confused; close it and open a new one
                $client = $this->client_socket($context);
                // Send request again, on new socket
                $client->send($msg);
 }
 }
 }
}

You can see from the code above, have a 3 strike rule. The reasoning behind this is that if the client fails to connect to the queue service more than 3 times, we can stop trying to inject into the queue and move on to the next item. As we ultimately intend to adapt the lazy pirate pattern we have made it so that if the socket times out, we can then create a new socket and retry. Without this, as the architecture becomes more complicated we may then end up in a situation where we might have errors, thus the recommend solution is to create a new socket. Once the client has sent its message to the queue, we poll for a response (i.e. which is the message we sent returned back). Once we have a response that is valid, meaning that the queue has been populated, we can stop polling until the next message.

The Frontend Client

protected function getFromQueue()
{
 $context = new \ZMQContext();
 $worker = new \ZMQSocket($context, \ZMQ::SOCKET_REQ);
 $read = $write = array();
 // Set random identity to make tracing easier
 $worker->connect("tcp://localhost:5556");
 // Tell queue we're ready for work
 $worker->send("ready");
 $reply = $worker->recv();
 return $reply;
}

Our Frontend Client is much simpler as it is part of a process that is being continually updated, therefore it doesn’t need the same connection retries are the Backend Client. We simply send “ready” to the queue system and if the queue is populated it will return us the first item.

The Queuing Service

This is a continuously running executable created using c++.

zmq::context_t context(1);
zmq::socket_t frontend (context, ZMQ_ROUTER);
zmq::socket_t backend (context, ZMQ_ROUTER);
backend.bind("tcp://*:5555");
frontend.bind("tcp://*:5556");

We create two sockets of type ZMQ_ROUTER which is an advanced pattern used for extending request/reply sockets. This means when we improve our queuing system we will be able to route packets to specific recipients using an address in the message.

After creating our sockets, we initialise them

// Initialize poll set
zmq::pollitem_t items [] = {
 { frontend, 0, ZMQ_POLLIN, 0 },
 { backend, 0, ZMQ_POLLIN, 0 }
};

//poll the sockets - this seems to poll both sockets at the same time
zmq::poll (items, 2, -1);

Backend Handler

If we get a message from the backend, we check the contents to see if it contains purge at which point we empty the queue, otherwise we push the msg contents onto the queue. Finally we send the message back to the backend to show that the message has been received.

//receive msg from client
if (items [1].revents & ZMQ_POLLIN) {
 //get message from client
 zmq::message_t message(0);
 string client_addr = s_recv (backend);
 string empty = s_recv (backend);
 assert (empty.size() == 0);
 string msg = s_recv (backend);

 //allow the backend to purge the queue
 if(msg == "purge") {
 while (!queue.empty()) queue.pop();
 } else {
 queue.push(msg);
 }

 //send response back to the backend
 s_sendmore (backend, client_addr);
 s_sendmore (backend, "");
 s_send (backend, msg);
}

Frontend Handler

If we get the “ready” message from the frontend client, we pop a message off the queue and return it to the frontend client. If the queue is empty we send an “empty” message back instead.

// Handle activity on frontend
if (items [0].revents & ZMQ_POLLIN) {
 //get message from worker
 zmq::message_t message(0);
 string worker_addr = s_recv (frontend);
 string empty = s_recv (frontend);
 assert (empty.size() == 0);}
 string msg = s_recv (frontend); string queueMsg;
 if(msg == "ready") {
 if(queue.size() > 0) {
 queueMsg = queue.front();
 queue.pop();
 } else {
 queueMsg = "empty";
 }
 }
 //send reply to worker with contents of queue
 s_sendmore (frontend, worker_addr);
 s_sendmore (frontend, "");
 s_send (frontend, queueMsg);
}

That is our complete queuing system using ZeroMQ with PHP and C++.

Summary

Using the above has allowed us to create a very simple in memory queue daemon that we can use to quickly pass data from one system to a another. On the whole it works well and we are looking to expand on it in the near future to increase both its functionality and scalability.

You can find the queueing daemon (christened as “ZuQ”) on github @ https://github.com/zucchi/ZuQ

Glorious Gluster - How to setup GlusterFS on Rackspace Cloud and Ubuntu 12.10

Fri, 15 Mar 2013 00:00:00 +0000

A few of our projects recently called for a distributed file-system that provided high availability and redundancy. After a tip off from a fellow techie and a quick browse around the net it appeared that a solution called GlusterFS appeared to tick all the boxes for what we were wanting.

However setting it up turned out not to be as trivial as I had originally anticipated. I’m going to try and put down the process we have evolved for setting it up on Ubuntu in the cloud

A couple of things to clear up first.

We are using Rackspace for our cloud but beyond the setup of the servers it should still be relevant
There are a number of ways to interact with Rackspaces set up but for this we are going to use the cloud control panel
We use Ubuntu as our preferred server which means that our config tends to be all over the place compared to other guides
You will need to set up a minimum of 2 servers and a separate block storage device for each.
We have set up and broken a few different variations of gluster setup so far and make no guarantees that the setup in this blog is infallable but its the best wehave so far.

Setting up the hardware

First things first. We are going to need to set up are some servers.

Feel free to create any size server you want. Just make sure to select Ubuntu 12.10 (or whatever version you may have that is newer).

You will also need to define a new network to work with. We use this to isolate the traffic between the nodes of our new gluster.

You can create a new network when creating the first of your servers. On the creation page under the networks heading you can find a “Create Network” button.

Hopefully this should be quite self explanatory. Now when you create subsequent servers you will then have the option to attach your new network (“GlusterNet” in my example).

Once the two starting nodes have been created then you need to add some additional block storage to store your data on. Make sure that you create blocks that have sufficient capacity for your needs. Something else to consider is using High Performance SSD storage. Its a little on the pricy side but well worth the expense if you are trying to eak out every ounce of performance from the implementation.

You will then need to attach one to each of your servers.

Once attached you will be able to see the details of the block mount point from the block storage details page.

Make a note of the mount point (in this case “/dev/xvdb”) as we will need that in a minute.

Prepare the Server

Now that we have a the hardware ready we can shell into a server to set it up.

First you need to shell into your server and update its OS as the images provided by most cloud supplier tends not to have the latest patches and updates. In our case it’s as simple as:

apt-get update 
apt-get upgrade

Once that’s done we then need to prepare the Block Storage device ( henceforth refered to as a “brick”)

if you run fdisk -l you should see that an entry that looks something like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

This indicates that our brick needs a partition table and formatting. We can achieve this be doing the following

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe7da4288.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
 p primary (0 primary, 0 extended, 4 free)
 e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): 
Using default value 209715199

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

I’ve highlighted the prompts and my responses. All we are doing here is creating a default partition table that has a single partition which uses up the whole disk.

now running fdisk -l should give us something that looks like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
43 heads, 44 sectors/track, 110843 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe7da4288

 Device Boot Start End Blocks Id System
/dev/xvdb1 2048 209715199 104856576 83 Linu

As you can now see we have a valid device of /dev/xvdb1 that we can mount_._ However we need to create a valid filesystem on the new brick before we can mount it. I have been doing this with Ext4 rather than XFS (which is the recommened filesystem from gluster), this is mainly down to the fact that when i tried using XFS I kept getting some issues with performance and access. I’m sure that with further investigation I could resolve this but as of yet haven’t had chance to. So far though I have had zero issues using Ext4. To create the filesystem we run:

mkfs.ext4 -j /dev/xvdb1

Next, create a folder to mount to, easily done by executing:

mkdir -p /glusterfs/brick

Finally, the simplest way to mount the device is via your /etc/fstab by adding the line

/dev/xvdb1 /glusterfs/brick ext4 defaults 1 2

and running mount -a as root (this will also mean that it mounts on boot for you automatically as well.)

Next we need to install the latest gluster version. At the time of writing this was v3.3.1. You can find a version to suit your OS at http://www.gluster.org/download. If you are using Ubuntu you can do the following

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-server glusterfs-client

By this point you will now have a single working server to continue on your going to need to set up your second server ready to create your new volume.

Once you have your second (or third, fourth, etc) setup its a good idea to add a reference to each one of them to your /etc/hosts file. This is not really necessary and you can just use the IP addresses of each server but it saves you having to remember each IP and makes it easier to identify.

Remember that we are going to be working with the new network interface you created earlier (i.e “GlusterNet”). to get the IP of your GlusterNet interface a quick ifconfig will show you an interface with an IP that matched the CIDR from earlier. In my case I now have 2 IPs of 192.168.3.1 & 192.168.3.2.

So now I add the following lines to my /etc/hosts file

192.168.3.1 gluster1
192.168.3.2 gluster2

Creating our volume

Now that the servers are prepared we can now play with the the tool gluster.This tool is a life saver in getting everything configured quickly and you can easily get a list of what its capable of by running gluster help. Now Im not going to take you through every command and option and would recomend reading the gluster manual to learn more.

What this tool actually does is help generate and manipulate all the required config that is then stored at /var/lib/glusterd/.

Firstly we need to tell gluster is that we have a pool of servers that will communicate with each other. Gluster refers to these as peers. To do this you need to run gluster peer probe gluster2 on each server for each server that will be used, replacing “gluster2” with the name names you defined in your /etc/hosts file. This will then create the appropriate files at /var/lib/glusterd/peers/

Now that all our peers have been defined we can get to actually creating the new distributed volume. This however requires a little consideration as there are some decisions you need to make.

If we take a look at the help for creating a new volume we can see that we need to decide on what options to use

volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>

- what are we going to name our volume
[stripe ] [replica ] - are we going to crate a striped or replicated volume and how many “bricks” are we going to create this volume with
[transport {tcp|rdma|tcp,rdma>] - What transport protocol do you want the peers to communicate with
- which servers/bricks do you want to use.

for more information on how to create you volume and what all the options mean have a look at these links

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Replicated_Volumes

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Striped_Volumes

for our purposes we are going to run

gluster volume create myvolume replica 2 transport tcp gluster1:/glusterfs/brick gluster2:/glusterfs/brick

This now creates a new volume that spans both of our servers. you can confirm that this is the case by running gluster volume info and you should get something that looks like

Volume Name: myvolume
Type: Replicate
Volume ID: d3dd24fd-9482-44c3-9503-24291fad8193
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/glusterfs/brick
Brick2: gluster2:/glusterfs/brick

running this on both servers should give you the same results.

What you will now find is that the gluster command has created a plethora of files at /var/lib/glusterd/vols/myvolume/. As you work with gluster more and more you will find yourself drawn to these files as they control all the different aspects of how the volume works and performs. Most importantly we will need some information from these files when we come to configure a client to mount the volume.

All that is left to do now is start the volume which can be easily done with a quick gluster volume start myvolume

At this point we have now completed setting up our volume but we need to add some security. I would strongly recommend setting up a firewall using ufw to control access to the server. The easiest way to do this is to allow all traffic on your “GlusterNet” network interface as only the servers you attach to that network will have access. you can find a guide to using ufw at https://help.ubuntu.com/12.10/serverguide/firewall.html.

Mounting a Client

Now that we have a working volume we need to add some clients. To do this you will need to create a new server as above that is attached to the “GlusterNet” network but without the block storage (unless you really want it that is).

Make sure to add your gluster dfinitions to your /etc/hosts file

Once you have your new client server ready we can install the gluster client

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-client

I’ve seen a number of different guides that tell you to install glusterfs-server as well but I have as yet had no need to as it all works without it.

Now there are a lot of way that you can mount your new Gluster volume. I have tried a few and have had varying results. What I have found is that the best way is to create a volume file. To do this we create a new file at /etc/glusterfs.vol.

volume gluster1
 type protocol/client
 option transport-type tcp
 option remote-host gluster1
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume gluster2
 type protocol/client
 option transport-type tcp
 option remote-host gluster2
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume replicate
 type cluster/replicate
 subvolumes gluster1 gluster2
end-volume

volume writebehind
 type performance/write-behind
 option cache-size 1MB
 subvolumes replicate
end-volume

volume cache
 type performance/io-cache
 option cache-size 400MB
 subvolumes writebehind
end-volume

What you will notice is that there is a and required for this to work. You can find these details on one of your peer servers in the file /var/lib/glusterd/vols/myvolume/trusted-myvolume-fuse.vol.

This /etc/gluster.vol file is basically going to inform the gluster-client software about how to connect to the gluster volume and all the available nodes to connect to. This provides us with some level of fail-over so should one node become unavailable the gluster client will seamlessly switch to a different one. It also allows us to define additional “translators” such as the performance-io one that you can see here. I would strongly recommend reading through the available translators to see which may be useful to you.

Now one of the main issues you will find with Ubuntu is that it will fail on boot if you try to add this mount to your fstab. To get around this you can use Upstart. if you create the following file at /etc/init/glusterfs-mount.conf making sure to change to the interface for your GlusterNet network (i.e. eth0 or eth1 or eth2, you get the idea)

author "Matt Cockayne"
description "Mount GlusterFS after networking available"

start on net-device-up IFACE=<interface>
stop on stopping network
stop on starting shutdown

script
    mount -t glusterfs /etc/glusterfs.vol /glusterfs
end script

As you can see we are using a straight mount command. The magic is that this will not be executed until the start clause validates which in this case is not until the network interface for “GlusterNet” is up and running properly. You will also see that we are mounting the /etc/gluster.vol file to /gluster (remember to create this folder to mount to) rather than mounting a network path as you might when mounting an NFS share.

If you wanted you could also add more to your upstart script to handle clean un-mounting of gluster thus allowing you to then use the service gluster-mount (start|stop|restart) commands

A quick reboot of the client server should confirm that it boots successfully and you will now end up with your volume mounted at /gluster. You can now test this by creating a new file. I tend to create an empty file at /gluster/mounted just so I have a quick reference that the folder is mounted. Once that’s created if you now go and take a look at the /gluster/brick on your “peers” you should see that there is now a file called “mounted” sat there looking all smug that it worked.

Caveats

Some important things for you to be made aware of

Never write directly to a brick. Make sure to write to the volume only through a configured client
Beware of split-brain. http://community.gluster.org/q/what-is-split-brain-in-glusterfs-and-how-can-i-cause-it/ http://www.gluster.org/2012/06/healing-split-brain/
RTFM - Read The F***ing Manual. Gluster is big and complex and there is a lot for you to understand. You can download a copy of the manual from here

Docblock, Oh Docblock, wherefore art thou Docblock (hint: Zend Optimizer Plus lost them)

Fri, 01 Mar 2013 00:00:00 +0000

tl;dr> I make a terrible assumption about Zend Optimizer+ and am corrected by Dominic in the comments;

Terrible post title I know but its the best I could come up with.

I’ve just come up for air after spending the majority of the day debugging some issues on our current development sandbox.

Now our sandbox tends to be quite bleeding edge in some circumstances and as such we run a fair few bits of unstable code. On the sandbox in question we have been running PHP 5.4.11 and unfortunately we have struggled to get APC working with it just the way we need it to. The lack of APC tends to make this sandbox quite slow.

We recently saw that Zend have open-sourced their OptimizerPlus extension (https://github.com/zend-dev/ZendOptimizerPlus) and that it was compatible with 5.4…. Fantastic, or so we thought.

So I added the new OptimiserPlus to the sandbox and everything was going swimmingly. That was until we had to run one of the utility scripts that we use to rebuild some of our data structures. These scripts make use of different parts of both Zend Framework and Doctrine which tend to rely on some heavy DocBlock annotations.

Now having used both APC and Zend Server knowing that they done affect this kind of functionality I had expected that OptimizerPlus would be fine…. Wrongo. It took me a good few hours of head scratching trying to figure out what had happened.

It turns out that OptimizerPlus suffers from the same flaws that eAccellerator does and strips Docblocks when caching the bytecode. This results in Reflection returning false when you call methods such as `getDocComment()`.

All in all its not the end of the world I just disable OptimizerPlus and have to wait till I can get APC working. Not my ideal scenario but I can live with it.

Something that does concern me is that there is currently an RFC that has gone to vote (https://wiki.php.net/rfc/optimizerplus) about integrating OptimizerPlus into the PHP 5.5 distribution. While this is great I do worry how many other things may break and will they be picked up and fixed for the 5.5 release.

Update: Since writing this post the RFC has finished being voted upon and has been approved. You can expect to see Optimizer Plus appearing bundled with PHP soon.

Update (15th Mar 13): Thanks to Dominics’ comment I now know that you can tell Optimizer+ to retain your Docblocks by setting your config using

zend_optimizerplus.save_comments (default "1")
	If disabled, all PHPDoc comments are dropped from the code to reduce the
	size of the optimized code. Disabling "Doc Comments" may break some
	existing applications and frameworks (e.g. Doctrine, ZF2, PHPUnit)

zend_optimizerplus.load_comments (default "1")
	If disabled, PHPDoc comments are not loaded from SHM, so "Doc Comments"
	may be always stored (save_comments=1), but not loaded by applications
	that don't need them anyway.

That’ll teach me to write a blog post without investigating more first.

Our Redmine install died, We all cried!

Sat, 23 Feb 2013 00:00:00 +0000

We have been using redmine for quite a long time and a few months ago attempted to upgrade from 1.3 to 2.something. Unfortunately I (quite typically) borked the installation and since then its been hobbling along after my attempts to fix it left it crippled.

Yesterday it finally gave up the fight and my attempts to resurrect the installation were futile. After a quick funeral (the eulogy was very touching), and wake in a nearby emporium of alcoholic beverages to commiserate our loss, I set about trying to figure out what to do next.

Alternatives

Now while Redmine is a worthy tool and has always managed to do what I needed in the past, recently its just not cut the mustard. I’ve kept toying with the idea of creating our own project management system but as with all in-house projects that we dream up its just never going to happen.

A quick google around our options are to either go for a hosted solution (not possible as we have some very specific requirements regarding our SCM that mean we have to host our own repos for client work) or Redmine (or chilli project).

Yes we looked at a number of other management tools and of them all Redmine is still the closes to what we needed.

Installation

So I spin up a new server instance of ubuntu 12.10 on the cloud and get to work installing the latest version.

As root I then run through these steps (you should assume that ALL of these steps require you to be root and files should be owned by root)

# update/upgrade base installation of ubuntu packages
apt-get update && apt-get upgrade

# install the requisite scm tools that we use
apt-get install git-core subversion mercurial cvs

# set up ruby
apt-get install ruby rubygems libruby ruby-dev

# set up apache & mysql
apt-get install apache2 libapache2-mod-passenger mysql-server mysql-client libmysqlclient-dev

# install imagemagick and the magick wand
apt-get install imagemagick libmagickcore-dev libmagickwand5 libmagickwand-dev

# create our user and database in mysql 
# replace uniquePassword with your own password
mysql -u root -p -e "create user 'redmine'@'localhost' identified by 'uniquePassword'"
mysql -u root -p -e "create database redmine"
mysql -u root -p -e "grant all on redmine.* to 'redmine'@'localhost'"
mysql -u root -p -e "flush privileges"

# clone redmine code to target location
cd /usr/local/share
git clone git://github.com/redmine/redmine.git

# set apache as the owner of redmine
chown -R www-data:www-data redmine

# move into our new redmine folder
cd redmine

# set up your database configuration
cp config/database.yml.example config/database.yml
vim config/database.yml

production:
 adapter: mysql2
 database: redmine
 host: localhost
 username: redmine
 password: uniquePassword

# install bundler gem
gem install bundler

# use bundler to set up redmine installation and without specified dependencies
bundle install --without development test postgresql sqlite

# set up our secret token
rake generate_secret_token

# set up our database and load default configuration
RAILS_ENV=production rake db:migrate
RAILS_ENV=production rake redmine:load_default_data

# edit /etc/apache2/sites-available/default
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
 <Directory "/usr/lib/cgi-bin">
 AllowOverride None
 Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
 Order allow,deny
 Allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log

 # Possible values include: debug, info, notice, warn, error, crit,
 # alert, emerg.
 LogLevel warn

 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# restart apache 
service apache2 restart

That should be enough for you to have a working installation of redmine ready for you to use/customise

Additional Config

We typically have additional steps that we would configure for our own installation.

# add plugin assets folder
mkdir /usr/local/share/redmine/public/plugin_assets
chown www-data:www-data /usr/local/share/redmine/public/plugin_assets

# enable some additional apache modules
a2enmod rewrite

# disable mod ssl
a2dismod ssl

# install gnutls 
apt-get install libapache2-mod-gnutls

# install ssl certificate bundle and key (this assumes that you have already copied the key and bundle to ~/)
mv ~/my_certificate.bnd /etc/ssl/certs/my_certificate.bnd
chmod 0644 /etc/ssl/certs/my_certificate.bnd
mv ~/my_certificate.crt /etc/ssl/private/my_certificate.key
chmod 0600 /etc/ssl/private/my_certificate.key

# now configure your /etc/apache2/sites-available/default-tls
<IfModule mod_gnutls.c>
<VirtualHost _default_:443>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log
 # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined

 GnuTLSEnable On
 GnuTLSCertificateFile /etc/ssl/certs/my_certificate.bnd
 GnuTLSKeyFile /etc/ssl/private/my_certificate.key
 GnuTLSPriorities NORMAL:!DHE-RSA:!DHE-DSS:!AES-256-CBC:%COMPAT
</VirtualHost>
</IfModule>

# Add some Rails / Passenger specific config to /etc/apache2/sites-available/default-tls
RailsEnv production
PassengerDefaultUser www-data
PassengerSpawnMethod smart
PassengerPoolIdleTime 300
PassengerMaxRequests 5000
PassengerStatThrottleRate 5
PassengerHighPerformance On

# change your /etc/apache2/sites-available/default to redirect to ssl
<VirtualHost *:80>
 ServerAdmin sysadmin@zucchi.co.uk
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 RewriteEngine On
 RewriteCond %{HTTPS} off
 RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
 Options FollowSymLinks
 AllowOverride None

 ErrorLog ${APACHE_LOG_DIR}/error.log
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# enable your new default-tls vhost and restart apache
a2ensite default-tls
service apache2 restart

# setup &amp; configure email
# when prompted select "internet site" and enter the domain you are hosting redmine from i.e. mysite.co.uk)
apt-get install postfix

# create config file and uncomment the production settings for sendmail
cp /usr/local/share/redmine/config/configuration.yml.example /usr/local/share/redmine/config/configuration.yml
vim /usr/local/share/redmine/config/configuration.yml

production:
 email_delivery:
 delivery_method: :sendmail

service apache2 restart

#install pixel cookers theme cos we like it
git clone git://github.com/pixel-cookers/RedmineThemePixelCookers.git /usr/local/share/redmine/public/themes/pixel-cookers

Debug PHP CLI on Remote Server with Xdebug and PHPStorm

Wed, 06 Feb 2013 00:00:00 +0000

This was a head scratcher when I ran into this yesterday and I thought I would share my solution to the following scenario:

I need to debug PHP Command Line script, located on Remote LAMP Virtual WebServer running in Virtual Box with a Shared Folder, using local PHPStorm 5.0.

The solution:

You first must set PHPStorm to use remote file paths. To set these go to the following:

PHPStorm -> Peferences -> PHP -> Servers

This gives the following display:

Replace the Name, Host and Absolute path on the server, to match your own settings. Note keep the Name and Host the same for ease.

Next add some breakpoints in PHPStorm and set it to listen for any debug connections using the listener icon:

Now login to your Remote Server via SSH etc.

You now need to change settings for Xdebug in either xdebug.ini or php.ini depending on how you installed it. You also need to know the IP of the local machine. This can permanently set in the Network Setting of your VM in Virtual Box, so you will never have to change it. In my example the local machine running PHPStorm is:

192.168.56.1

Now edit the ini file that contains your Xdebug settings and set the following:

xdebug.remote_host = 192.168.56.1
xdebug.remote_connect_back = 0
xdebug.remote_port = 9000
xdebug.remote_handler = dbgp
xdebug.remote_mode = req
xdebug.remote_enable = 1
xdebug.idekey = phpstorm1

Be aware you might have to change the remote_host and the idekey based on your own environment. To better understand what each option does, see Xdebug Settings

Finally, when running the script you must set the following variables:

PHP_IDE_CONFIG="serverName=dev.example.com"

PHP_IDE_CONFIG will tell PHPStorm how to map the Remote File Paths to what it sees Locally. Again replace the URL with the Name/Host you set in PHPStorm. Note: You can export this, if your system is only running one site; mine is not.

You can run this inline with your script:

PHP_IDE_CONFIG="serverName=dev.example.com" ./testscript.sh

This should send you to PHPStorm where you earlier placed breakpoints.

Happy Debugging!

Quick and easy setup of and connection to NRPE on Ubuntu

Wed, 06 Feb 2013 00:00:00 +0000

About NRPE

NRPE (Nagios Remote Plugin Executor) is a useful tool that allows you to execute scripts on remote servers and return the output for ingestion by some form of monitoring software.

Setup

We currently have our own instance of Icinga running to monitor our servers and have recently started to offer access to it for our clients.

The majority of our servers (and our clients servers if we set them up) use one variant or another of Ubuntu. This means we can very quickly get our servers connected to a Nagios/Icinga instance.

First things first we need to install the nrpe server and all the associated plugins

apt-get install nagios-nrpe-server \
nagios-plugins-basic \
nagios-plugins \
nagios-plugins-extra

Next we need to edit the main nrpe config file to be found @ /etc/nagios/nrpe.cfg. What your looking for is the lines

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames 
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1

# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed. This option only works
# if the daemon was configured with the --enable-command-args configure script
# option. 
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! *** 
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0

You will want to change this to the IP of your Nagios/Icinga instance and set the dont_blame_nrpe value to 1. Feel free to take a look round the rest of the file. Its all quite interesting and generally will documented. Be careful what you change though in case something breaks.

You will also want to look for some lines that are refererd to as “COMMAND DEFINITIONS” and look something like this

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

You can go ahead and comment these out as we will be adding our own definitions shortly. The main reason for removing these is that we will be configuring some specific scripts for our own use later that allow you to configure your requirements and thereshold from within your Nagios/Icinga config.

Configuration of Monitoring Server

Once this is complete you can now configure a new “check command” for use with your nagios/icinga server.

define command {
 command_name check_nrpe
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

define command {
 command_name check_nrpe_command_args
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$
}

Here you can see that we have set up 2 different check commands. The first is a simple command requiring only one argument of $ARG1$ which would be the name of the command we want to run on the remote server. The second command is almost identical except for the fact it takes a second argument which allows you to input a series of “arguments” to be passed to the command on your remote server. each argument should be separated by a space.

Now that you have these you can then configure your hosts and services to make use of it. I would recommend having a trawl through the Nagios/Icinga sites & documentation to find out how to create a config that suits you.

Configuration of Remote Server

Now that we have our monitoring server ready its time to add the command we want to run to the remote server.

To do this your /etc/nagios/nrpe.cfg shoudl hopefully have a line in it that looks like

include=/etc/nagios/nrpe_local.cfg

if it doesn’t have a line like that then add it and edit the `/etc/nagios/nrpe_local.cfg` file to look a little like this

command[check_apt]=/usr/lib/nagios/plugins/check_apt
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p /dev/sda1
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

These are a few simple commands that I tend to use most often. These translate to your “check_nrpe” commands like so

$ARG1$ = everything inside the square brackets [ ]
$ARG2$ = each of the $ARG?$ keys as a single string separated by a space

Once that’s done you should be able restart your nrpe server with `/etc/init.d/nagios-nrpe-server restart`

It really is that simple. Do bear in mind that because you can pass arbitrary arguments into nrpe this was you could leave yourself vulnerable to a bit of maliciousness so its a good idea to make sure your firewall restricts port 5666 (the default port) to IPs you trust.

Compiling Apache 2.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

I’ve decided that I need to up my game when it comes to webservers. However I’m not yet ready to switch to Nginx or one of the other webservers out in the wild as I need something up and running rapidly.

Granted the numbers are definitely against Apache in a lot of benchmarks but historically I’ve always had a good experience and the entry level makes it much more appropriate for me to stick with it.

However Apache 2.2 is rather long in the tooth, thankfully 2.4 has been out for a while now. The problem I have is that I tend to favour Ubuntu as a platform and there is no sign of a 2.4 version appearing on the horizon anytime soon as they are waiting for it to be implemented upsteam in Debian before including it in Ubuntu.

Now there are PPAs available out there but im not overly happy using them (especially on production environments) So the only option is to compile.

First thing is to install all the dependencies we are going to need. Thankfuly ubuntu has a nice and simple way of handling this.

apt-get build-dep apache2

We can then download the source code and start the compilation.

So from the root of our new copy of the source we need to run our configure.

./configure --prefix=/usr/local/apache2 \
 --enable-mods-shared=all \
 --enable-http \
 --enable-deflate \
 --enable-expires \
 --enable-slotmem-shm \
 --enable-headers \
 --enable-rewrite \
 --enable-proxy \
 --enable-proxy-balancer \
 --enable-proxy-http \
 --enable-proxy-fcgi \
 --enable-mime-magic \
 --enable-log-debug \
 --with-mpm=event

You will notice that I’m installing it using the event mpm. Hopefully I’ll be covering more about the event mpm in the future.

Next we need to run make

make && make install

Once that’s complete you should be able to run

/usr/local/apache2/bin/apachectl start

and get the “it works” message through your webrowser when accessing the server IP.

Dont forget to configure apache to suit your specific requirements.

Something that will come up is how to start apache on boot. Seeing as Ubuntu uses Upstart it makes sense to utilise it for controlling apache.

So in the file `/etc/ini/apache.conf` we need to put

# apache2 - http server
#
# Apache is a web server that responds to HTTP and HTTPS requests.
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog

author "Matt Cockayne <matt@zucchi.co.uk"
description "Apache 2.4 HTTP Server"

start on runlevel [2345]
stop on runlevel [!2345]

console output

pre-start script
 mkdir -p /var/run/apache2 || true
 install -d -o www-data /var/lock/apache2 || true
 # ssl_scache shouldn't be here if we're just starting up.
 # (this is bad if there are several apache2 instances running)
 rm -f /var/run/apache2/*ssl_scache* || true
end script

# Give up if restart occurs 10 times in 30 seconds.
respawn limit 10 30
respawn

script
 if test -f /usr/local/apache2/bin/envvars; then
 . /usr/local/apache2/bin/envvars
 fi
 ULIMIT_MAX_FILES="ulimit -S -n `ulimit -H -n`"
 if [ "x$ULIMIT_MAX_FILES" != "x" ] ; then
 $ULIMIT_MAX_FILES
 fi

 /usr/local/apache2/bin/httpd -k start -D FOREGROUND
end script

This is a rather simple upstart script and I will be looking to update it at some point… but it works

Once that’s done you should find that on reboot Apache will start and take advantage of all the management features of upstart including attempting to respawn Apache should it end unexpectedly. You should also be able to then use the following commands to control Apache.

# how to start start apache
start apache
# or 
initctl start apache

# how to stop apache
stop apache
# or 
initctl stop apache

# how to restart apache
restart apache 
# or 
initctl restart apache

# check the status of apache 
status apache
# or
initctl status apache

I generally tend to avoid using the apachectl script found at /usr/local/apache/bin/apachectl once upstart takes control.

Compiling PHP 5.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

So recently I’ve been working with PHP 5.4 a LOT. Unfortunately Ubuntu (my main dev environment) is behind the times. So I’m resorting to compiling PHP manually.

Not a daunting as it may first appear. The really tricky part is working out your dependencies and `configure` script.

Hence the reason for this post as a reminder for myself and others that may want to do a quick compile. (I would recommend that if your compiling for a production/live environment that you make sure you understand what it is your compiling though before just using what’s here)

So where to start. Dependencies first I think

Ubuntu allows you to install dependencies for building source `apt-get build-deps`. We will use this and install any extras we may need.

apt-get install \
libxml2 \
libxml2-dev \
libssl-dev \
pkg-config \
curl \
libcurl4-nss-dev \
enchant \
libenchant-dev \
libjpeg8 \
libjpeg8-dev \
libpng12-0 \
libpng12-dev \
libvpx1 \
libvpx-dev \
libfreetype6 \
libfreetype6-dev \
libt1-5 \
libt1-dev \
libgmp10 \
libgmp-dev \
libicu48 \
libicu-dev \
mcrypt \
libmcrypt4 \
libmcrypt-dev \
libpspell-dev \
libedit2 \
libedit-dev \
libsnmp15 \
libsnmp-dev \
libxslt1.1 \
libxslt1-dev

And now the configure

./configure \
--prefix=/usr/local/php \
--with-apxs2=/usr/local/apache2/bin/apxs \
--enable-fpm \
--with-fpm-user=www-data \
--with-fpm-group=www-data \
--with-config-file-path=/usr/local/php/conf \
--with-config-file-scan-dir=/usr/local/php/conf.d \
--enable-debug \
--with-openssl \
--with-kerberos \
--with-zlib \
--enable-calendar \
--with-curl \
--with-curlwrappers \
--with-enchant \
--enable-exif \
--enable-ftp \
--with-gd \
--with-jpeg-dir=/usr \
--with-png-dir=/usr \
--with-vpx-dir=/usr \
--with-freetype-dir=/usr \
--with-t1lib \
--enable-exif \
--enable-gd-native-ttf \
--enable-gd-jis-conv \
--with-gettext \
--with-gmp \
--with-mhash \
--enable-intl \
--enable-mbstring \
--with-mcrypt \
--with-mysql \
--with-mysqli \
--enable-pcntl \
--with-pdo-mysql \
--with-pdo-pgsql \
--with-pgsql \
--with-pspell \
--with-libedit \
--with-readline \
--enable-shmop \
--with-snmp \
--enable-soap \
--enable-sockets \
--enable-sysvmsg \
--enable-sysvshm \
--with-xsl \
--enable-zip \
--with-pear \
--enable-zend-signals \
--enable-maintainer-zts

Once these are done then we follow the standard make process. Notice we are also running make test… very important as it givges more data for the developers to work with.

make && make test && make install

The next thing is configuring your php.ini file as the install doesn’t have one yet so we copy either the production or development default from the source code to the new conf dir and edit to suit your needs.

cp {php-source-dir}/php.ini-(development|production) /usr/local/php/conf

Thats it. All ready to roll… almost, this installation is the one I use for use with a webserver so you will want to add the appropriate directives to apache.

LoadModule php5_module modules/libphp5.so

AddHandler php5-script .php
AddType text/html .php

Rsync and custom SSH commands

Tue, 31 Jul 2012 00:00:00 +0000

Rsync is a great tool but can be a pain if you have to jump through hoops to connect via ssh such as connecting via a different port.

A simple solution is to use the -e flag (also knows as –rsh=COMMAND). This flag allows you manually define the ssh command to use when connecting

rsync -e 'ssh -p2020' -rav ./* user@server:

Will allow me to connect to a server with SSH listening on port 2020

Nice New Nexus7

Thu, 19 Jul 2012 00:00:00 +0000

This morning I woke up to an email telling me that my Nexus7 that I had ordered 3 weeks ago was… “out for delivery”.

I couldn’t contain my excitement. I sat patiently waiting by my door. Finally 11 o’clock rolls around and there is a knock. I’m handed a brown parcel and hand over the obligatory signature. I close the door behind me and carefully place the box on the desk. I contemplate teasing myself and seeing how long I can hold out before opening it.

That lasted about 20 seconds!!!!

In fact… this video says it all

http://www.youtube.com/watch?v=Xijcwbg8CGQ

I’m not gonna bore you with how awesome it is (and it is awesome). I will however point out a few obvious foibles with it though (not that they would have ever stopped me from buying it).

No way to expand storage
Lack of Flash (may not seem important, but until everyone else catches up there is loads of content I cant use i.e. BBC iPlayer)
A number of apps (games specifically) that I have run on my phone are not yet supported
Google Now feels a little clunky at times and struggles with some of the regional British accents
MTP doesn’t appear to work out of the box with my Linux OS (I’m sure this will be remedied soon)
No obvious way to directly access the front facing camera (easily remedied with an app from MoDaCo)

Registering custom view helpers in ZF2

Wed, 18 Jul 2012 00:00:00 +0000

If you want to register custom view helpers with a module you can do so by using the service location built into the Skeleton Application and creating a module config that looks something like.

return array(
 'view_helpers' => array(
 'invokables' => array(
 // generic view helpers
 'truncate' => 'Zucchi\View\Helper\Truncate',

 // form based view helpers
 'bootstrapForm' => 'Zucchi\Form\View\Helper\BootstrapForm',
 'bootstrapRow' => 'Zucchi\Form\View\Helper\BootstrapRow',
 'bootstrapCollection' => 'Zucchi\Form\View\Helper\BootstrapCollection',
 ),
 ),
);

Bootstrapping ZF2 Forms

Tue, 17 Jul 2012 00:00:00 +0000

So…

With the release of beta 5 for Zend Framework 2 I thought it time for me to tidy up and fix a few modules I created back at beta 3.

Now I’m a big fan of Twitter Bootstrap CSS framework as I’m sure a lot of other people are as well. Seeing that the Zend Skeleton Application comes with bootstrap already included it was easy enough to set up my forms using the old ZF Forms found in ZF1.

However a brand spanking new Forms component has been rolled out with ZF2. The long and the short of this new component meant that I had the opportunity to hand roll a new way of making my forms work with Twitter Bootstrap.

So, a little tinkering, a quick pull request to ZF2 to allow the definition of arbitrary options and I came up with some useful View Helpers that can be dropped into a project and used.

You can find them at https://github.com/zucchi/Zucchi.

So how to use them. Lets start by creating a new form (we’ll keep it simple for now)

class MyForm extends Form
{
 public function __construct()
 {
 parent::__construct('myform');
 $this->add(array(
 'name' => 'price',
 'attributes' => array(
 'type' => 'text',
 'required' => 'required',
 'placeholder' => '0.99',
 ),
 'options' => array(
 'label'
 'bootstrap' => array(
 'help' => array(
 'style' => 'block'
 'content' => 'The price you wish to use'
 ),
 'prepend' => array('$'),
 'append' => array('¢'),
 ),
 ),
 );

 $actions = new Collection('actions');
 $actions->setAttribute('class', 'form-actions');

 $actions->add(array(
 'name' => 'submit',
 'attributes' => array(
 'type' => 'submit',
 'value' => 'Save',
 'class' => 'btn btn-primary'
 ),
 'options' => array(
 'bootstrap' => array(
 'style' => 'inline',
 ),
 ),
 ));

 $actions->add(array(
 'name' => 'reset',
 'attributes' => array(
 'type' => 'reset',
 'value' => 'reset',
 'class' => 'btn'
 ),
 'options' => array(
 'bootstrap' => array(
 'style' => 'inline',
 ),
 ),
 ));
 $this->add($actions);
 }
}

You’ll notice that I have highlighted some lines. Thanks to the ability to set arbitrary options we can define a “bootstrap” option which we can then use to allow us to pass data into our new bootstrap view helpers. You can also see that I have added a save and reset button to a collection. I’ll explain that later.

So what next… Rather than go into the mechanics of how to work with forms I’ll refer you to the ZF documentation and this excellent blog post

We then pick up by looking at your view, and the helpers I have created.

BootstrapForm($form, $formStyle)

One of the few things I miss from the ZF1 implementation of Forms is the self rendering aspect! So what did I decide to do? That’s right I created a view helper to render everything in one command.

The $this->bootstrapForm() takes two parameters. The first is quite obviously the form. The second is the style of form. This is directly related to the form types that can be found http://twitter.github.com/bootstrap/base-css.html#forms. You can use any of ‘vertical’, ‘inline’, ‘search’ & ‘horizontal’. If you dont specify a formStyle then it will default to ‘vertical’

Caveat: This helper will then iterate through all of the associated elements and render them first. Only after the direct elements have been generated will it then move onto Collections or Fieldsets (as soon as I work out how I’ll fix this).

BootstrapRow($element, $formStyle)

This is a straightforward modification of the FormRow helper that come bundled with the new component.

We have a few differences now though. We have a second parameter as with the BootstrapForm view helper and the output is generated using sprintf and a set of templates that mimic the structures of the different form styles from bootstrap.

This helper can be used by itself to generate an element row and is used by the BootstrapForm helper

We can also now take advantage of the “bootstrap” options we set earlier.

Bootstrap Options

style

The style of form element to use regardless of what style may be passed into the view helper (you can see an example of this in the buttons from the MyForm example above)

help

This works in the same way as “description” did from ZF1 but allows you to define it either as a string or an array with the keys “style” for either ‘inline’ or ‘block’ and “Content” which should be self explainatory

prepend

Takes advantage of Bootstraps ability to prepend blocks to an input field. This can be defined as a single string, or an array of strings to allow you to add multiple blocks should you want to

prepend

Takes advantage of Bootstraps ability to append blocks to an input field. This can be defined as a single string, or an array of strings to allow you to add multiple blocks should you want to

These options get evaluated and spat out from the new renderBootstrapOptions() method as part of the “render”.

BootstrapCollection($element, $style, $wrap)

Again this is a direct rip off of the FormCollection helper found in the ZF2 Form component witha few modifications. The main difference is that is makes use of the BootstrapRow helper and has methods and properties to allow the setting of the form style to use.

You can see from the MyForm example above that we set a Collection called ‘actions’. This is a pretty standard way of grouping elements together. You can also see that we set a class for the Collection which may look familiar to those that have used Twitter Bootstrap for a while.

What our helper will then do is wrap the buttons in a div with the appropriate class attached. If you were to define a label for the Collection/Fieldset You would then also find that the fieldset and legend tags are also spat out with our <div class="form-actions"> sandwiched between them and the elements.

Result

So what we now get when we use MyForm with out helpers.

$this->bootstrapForm($form, 'horizontal');

Should now look something like this

How you can use it

As of right now you can get the library from its repo on github @ https://github.com/zucchi/Zucchi and can be found on packagist for use with composer

Edit: The bootstrap stuff has moved to a new location as a separate ZF2 module. you can find it @ https://github.com/zucchi/ZucchiBootstrap or @ packagist for use with composer

Loaded Testing

Sat, 30 Jun 2012 00:00:00 +0000

I recently had to do some load testing for a site recently that would allow me to test in excess of 100k requests in a 60 second period…

So I decided to do some testing using JMeter as it seemed like a suitable tool for doing what I needed and I had used it for some simpler testing in the past.

After a little fumbling around I managed to get a test plan designed that would simulate 10k users actually navigating the site and adding to a cart etc, with a number of various interactions. It wasnt perfect but it would correctly simulate over 100k requests.

So feeling quite pleased with myself I started the test from my laptop. Now I’m not a big gamer, I’m known to play a little World or Warcraft from time to time but that’s about it. So when it comes to computing power i tend to opt for battery life over sheer grunt.

Suffice to say, my laptop fell flat on its face, and if it hadn’t it turns out that the connection I was using just wasn’t up to the task of handling that much traffic adequately.

So plan B…

I quickly fired up the largest AWS instance available and got a copy of jmeter installed. A little tinkering with my test plan and some googling on how to run jmeter without a gui and a quick

./jmeter -n -t test-plan.jmx

and it appeared to be running.

(Please bear in mind that I’m being overly kind… it took a LOT of tinkering and twice as much Googling to work out how to get the test results out so i could actually get some idea of WTF was happening during the test)

So… client “happy”… I decided to go and find a better way to do my load testing in the future.

Sticking with JMeter I managed to find this gem of a page

http://jmeter.apache.org/usermanual/remote-test.html

tl;dr > use your local install of jmeter to trigger tests to run on one or more remote “nodes” and then have all the results sent to your local install.

So I set to work!

Building a Node

First I need to set up an AWS instance that we can use and duplicate so I can quickly build a cluster of nodes on demand. I’m a big fan of Ubuntu so I spin up a micro instance of 12.04 server. Next I shell into the instance and install the default Java runtime from apt

apt-get install openjdk-7-jre

Yes I know there are other more appropriate runtimes, but i dont really care… i just need it to work and it does.

next I grab a copy of the latest stable from http://jmeter.apache.org/download_jmeter.cgi and un-tar it to /usr/local/jmeter

(N.B. JMeter is available through the apt but I had issues with that version and you need to make sure that both your local version and all the nodes run the same version of jmeter)

We can now test that the install is working running /usr/local/jmeter/bin/jmeter-server and you should get some output that looks similar to

Created remote object: UnicastServerRef [liveRef: [endpoint:[10.???.???.???:38939](local),objID:[46522b57:138381f1023:-7fff, 2635011707874933136]]]

Which tells us that the server is running.

BUT unfortunately its not going to work just yet. Because we are using Amazons EC2 we are going to relying on their NAT for routing. Out of the box JMeter just wont work properly.

However there is something we can do to combat this. We can set the parameter RMI_HOST_DEF that the /usr/local/jmeter/bin/jmeter-server script will include in starting the server.

export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

I’ll explain what we are doing here. Amazon have been quite clever by providing a meta-data endpoint that you can poll from within your instance to get key pieces of data… Including the public dns record.

We can use this endpoint and using wget pipe that into the RMI_HOST_DEF param (ensuring that we prepend -D) and then export that so it becomes available to the /usr/local/jmeter/bin/jmeter-server script.

Now to get the server to start on boot.

a quick upstart script should solve this

# Upstart script to initialise jmeter-server

description "JMeter Server"
author "Dev in Charge "

start on started networking
stop on stopping networking
stop on stopping shutdown

console output

script
 # get the current public DNS record
 export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

 # start jmeter in server mde
 /usr/local/jmeter/bin/jmeter-server
end script

saving this to /etc/init/jmeter-server.conf will mean that it will auto-start jmeter-server on boot and allow you to manually control the process using start jmeter-server and stop jmeter-server

and thats it… instance configured

All you need to do now is save the instance as an AMI and you have an on-demand image for spinning up a cluster of remote JMeter servers for you to play with.

Configuring your local installation

Now that the server side is working we need to configure our local installation to allow it to connect.

First things first however, make sure you are using the same version of JMeter as you are running on the server.

We need to edit the jmeter.properties file that can be found in the bin folder of the installtion you downloaded. Look for the parameter remote_hosts This needs to be set with the public dns of the remote server(s) your connecting to. for example

remote_hosts=ec2-176-34-164-170.eu-west-1.compute.amazonaws.com,ec2-123-34-456-789.eu-west-1.compute.amazonaws.com

Thats your local version configured. You will now be able to tell your local version to run tests on any or all of your specified remotes.

However if your like me you work behind a router/firewall. If so this isnt the end of the story. When you send a test plan to a remote from your local install it will also send the IP address of your local machine for it to send the results back to. JMeter does this by looking up where your current hostname resolves to. In my circumstance it resolved to 127.0.1.1. The reason it did this is down to the fact my systems host file had the line

127.0.1.1 devincharge.local

To resolve this I had to change it to my external IP address

89.345.871.79 devincharge.local

And set up port forwarding from my router to my local machine for all ports from 1024 to 65535. Now, you can if you want use specific ports so you dont have to port forward everything from your router, but i’ll leave that for you to lookup as there are plenty resources on how to do this for you to google and I’ve waffled on for far too long already.

Happy testing

PHP Boy Scout

Why I still write code

I’m a builder, and that isn’t really a choice

What it actually buys

The bill, and who paid it

Does it still count when the robot types?

What I’d actually put my name to

A signature the platform can't forge

The thing a checksum can’t do

The crypto is the easy part

Pulling the two questions apart

Three answers to “where does the key live”

The separation is the whole point

Requiring it, without breaking everyone

What it actually buys

Anything under an 8

The carcass and the vultures

The number we never checked

Don’t blame the robots for this one

The version I’m betting on

The consent you can't ask for

Where I’m coming from

The trouble with a service

The part I’ll own

Three traps release-plz sets for a Rust workspace

First, what release-plz is doing

Trap 1: the default tag template is built for one crate, not a workspace

Trap 2: “one release for the whole workspace” isn’t a setting, it’s a category error

Trap 3: a release reads its config from the release commit, not HEAD

If you set one thing

Telemetry that asks, and telemetry that doesn't

Why bother putting it in the framework at all

The catch: two telemetries, opposite instincts

What they actually share

The refactor: making the old telemetry stand on the new core

Where the line is

Where this leaves the framework

Same config, two answers

Rust brings its own toolchain

The gauntlet before the gun

“Tag v0.5.1 already exists”

I assumed the default

Same config, two answers

Doing it the way I’d have done it a year ago

Stop being clever and RTFM

Building a web service with go-tool-base, part 6: seeing what your service is doing

The three things you actually want to see

Turn it on

Wire it onto the transports

Go deeper than the plumbing

Point it at a collector

One request, the whole way through

A word on the other telemetry

Where that leaves the service

The security service I had to switch off

What the baseline switches on

SubscriptionRequiredException

Switching two watchers back off, on purpose

Why this lives in the module, not in a hack

The honest version of “secure by default”

Building a web service with go-tool-base, part 5: docs that write themselves

The annotations, a third time

Generate the OpenAPI document

Serve it, viewer and all

Why it goes on the same server

One proto, three faces

From allow_failure to blocking

What allow_failure actually buys you

A check earns the right to block

trivy: blocked once there was nothing to block on

integration-tests: blocked once it could actually run

coverage: blocked once it could run at all, then once it cleared the bar

The three I left advisory, on purpose

The upshot

Building a web service with go-tool-base, part 4: REST for free, with the gateway

What the gateway actually is

Map HTTP onto the proto

Generate the gateway

Wire it on

Delete the duplication

A page from `html/template`

Why not just `net/http`?

What `update` does, step by step