At a previous employer I went looking for a shell script and found eleven of it.

Not eleven copies, but eleven variants, each one someone’s improvement on the version they’d copied from, spread over hundreds of laptops, each of them launching a pod into a Kubernetes cluster from whichever container image its author had last pulled. Some of those clusters sat inside a compliance boundary (FedRAMP, for the American ones). Some of those images were years old and carried CVEs by the hundred.

And the script was good. That was the problem.

Photocopies of photocopies

Hardly anyone sits down to write a bad script. Someone sits down to solve a problem, in bash because bash is right there, and it works, and a colleague asks for it, and it gets pasted into a second repo, where it picks up a flag that repo needed. A third repo takes that version. Someone fixes a bug in the fourth copy and the first three never hear about it. Give it two years and a couple of re-orgs and you’ve got a photocopy of a photocopy of a photocopy, each generation a little blurrier than the last, and no way of telling which one is the original because there isn’t one any more (I looked, for longer than was sensible).

We counted more than seventy scripts in that state across thirty-odd repositories. The Kubernetes one was just the most dangerous, because of where it pointed.

I want to be fair to bash here, because I’m about to spend the rest of this piece explaining why I replaced it and it would be easy to read that as contempt. It isn’t. A script is the right first move almost every time. You can read it, you can change it, there’s no build and no release and no repository to look after, and if it proves a bad idea you delete it and nobody mourns. Each reason that script got copied eleven times is a reason it was a good script. Bash didn’t fail. Bash did what bash does, which is get run wherever it’s been put.

The day it needs changing everywhere

One morning you need to change it, and not just the copy in front of you… all of them. The image the script launches is three years stale, a scanner has just told you in red how many known vulnerabilities are sat inside it, and some of the clusters it launches into are ones an auditor is going to ask about. So the fix is simple. New image tag. One line.

On hundreds of laptops you cannot reach.

That’s the moment the whole thing turns, and it’s not about how big the script had got or how clever it was. It isn’t size (I’ve seen thousand-line scripts that were perfectly fine as scripts), and it isn’t language or fashion either. It’s reach. A script is a thing you run where it lives. A tool is a thing you can change in places you can’t get to. The instant a change has to land on a machine you don’t control, and has to land whether or not its owner remembers to pull, you’ve stopped writing a script, whatever the file extension says.

We had no channel back to those laptops, none at all. The best we could do was a Slack message asking people nicely to update their copy, and everybody knows how that goes. (The people who read it had usually already updated. The people who hadn’t were the reason we were asking.)

So: a binary, and Go

What we needed was one thing every engineer ran, that knew how to replace itself, so the fix I made on a Wednesday was the fix on every laptop by Thursday without anybody being asked. Self-updating was the first requirement and everything else came after it.

That rules out a script almost by definition, and it also rules out the obvious next step, which is “a better script, in Python, in one repo”. A Python tool needs Python, and a particular Python, and its dependencies, on every machine it lands on, which is a second thing you can’t update, sat underneath the first. I’ve watched a “just pip install it” tool spend more of its life being installed than being used!

So it had to compile to a single file with no runtime under it, and that’s most of why it was Go: not because Go is the fashion in platform teams (it is, and I know how this reads), but because go build hands you one binary that runs on a laptop with nothing else installed, and cross-compiles for the Mac on the next desk from the same source. The other reason is that Go is about the easiest language there is to hand to a busy engineer who’s never written it. I wanted people fixing their own bugs in this thing rather than raising tickets against it, and a language you can pick up in a weekend was a real consideration, not an afterthought.

What it cost, because it did cost: a repository, a build, a release pipeline, an install script for three operating systems, and an owner. A script has none of that. If the script never needs to change on someone else’s machine, that cost buys you nothing, and rewriting it in Go anyway is the fashion-victim move platform teams get accused of. I’ve done that too, earlier in my career, and the result was a worse script that fewer people could read.

Two commands, then the rest

The other decision that mattered was not to rewrite seventy scripts.

We launched with two commands. Log in to the cloud, and launch the tools pod. That was the one script, replaced, with the image chosen by us rather than by whoever ran it, and the pod only permitted in the one cluster it was ever supposed to be in, and nothing else. Quiet launch, no announcement, and I sat with a couple of teams and walked them through it rather than sending round a link.

That was deliberate and it was the right call. A big-bang “the platform team has replaced all your scripts” is a thing engineers resent on principle, and rightly, because it takes away tools they understand and hands them one they don’t. Two commands that did one job noticeably better than the script they replaced earned the next command, and the one after that, and over a year and a bit the rest of the sprawl came in a script at a time, each one folded in when we could show it was better rather than because we’d said so.

By the time I left it was one binary, self-updating, with everything from pipeline analysis to Terraform state migration under it, and the eleven variants were gone. Not deprecated, actually gone.

What it didn’t fix

If I stopped there I’d be leaving something out.

That tool sat on top of an estate that had grown organically for years, layer on layer, and nothing about a well-distributed binary fixes an architecture. It doesn’t pretend to. What it does is get the sprawl under control enough that you can see the architecture, and it was meant to be the vehicle for fixing it: the plan we had for bootstrapping an environment in hours rather than months was going to be built on the same framework. Then the manager who sponsored that plan left the business, and the plan went with him, and the tool carried on being a very good way of interacting with an infrastructure nobody was allowed to change.

So if you’re looking at a compiled tool in your own estate and thinking “that’s covering up a design problem”, you might be right. It might also be the only thing that was allowed to ship. From the outside I’m not sure you can tell those two apart.

The one I kept

The framework underneath that tool wasn’t born there, and it didn’t stay there. I’d been carrying a bag of Go modules from job to job for years before that one: config management, a self-update mechanism, an initialisation protocol, logging wrappers, the bits every tool needs and nobody wants to write twice. That employer got the then-current iteration to build on, and iterating on it for a year and a bit is what I took away. The first commit of go-tool-base is that iteration as it stood the month I left, and the eight months since have changed it beyond recognition, but it will still turn a script into a command for you, because that’s the migration it grew up doing.

I still write scripts. Loads of them. Most of what’s in my bin directory is bash and will stay bash, because it lives on one machine and only I run it. If one of them ever needs to change on a laptop I can’t reach, that’s the day it becomes something else, and not a moment before.

Nobody’s had to photocopy one since.