PHP Boy Scout — From one framework to fifty modules

I graded every package for how badly it wants to leave

Tue, 29 Sep 2026 00:00:00 +0000

I sat down one weekend and gave twenty-seven packages a mark out of ten, four times each. A hundred and eight numbers, in one table, about my own code (which is either diligence or a cry for help, I haven’t decided).

Not the reason you’d assume

The second paragraph of that report is a disclaimer, and I put it there on purpose because I could see where this was heading:

The goal is not to break GTB apart. The goal is to identify components with clear reuse value outside the framework, then describe the broad decoupling work required to extract them without hollowing out GTB’s integrated experience.

go-tool-base is an all-in-one framework and I still want it to be one. Type gtb generate project and you should get a thing where the logging, the config, the transport and the CLI already know about each other. That integration is the product.

But I’d started wanting pieces of it elsewhere, and signing had already left the building. So the question was less “how do I dismantle this” and more “which of these has a life outside, and what would it cost to give it one”. You can answer that by instinct, of course. I’ve watched people do it, I’ve done it myself, and what you get is whatever happened to be most annoying that week.

Four columns, and just the one surprise

So it got scored. All twenty-seven, out of ten, on four axes:

Extraction. The overall recommendation. Should this be its own module at all?
Package value. What’s it worth to someone who wants it without the framework?
Code quality. Cohesion, testability, API shape, how grown-up it is today.
Ease of decoupling. How much framework-specific coupling has to come out first.

The first three are the ones you’d expect, and they mostly agree with each other. Good code tends to be worth having, and things worth having tend to have been looked after. The fourth column is the one that earns its keep, because it’s measuring something else: not should this leave but can this leave yet. Those are different questions and the table is only useful because it keeps them apart.

Four rows from the table

Package	Extraction	Value	Quality	Ease
`chat`	9	10	7	4
`redact`	9	9	8	10
`regexutil`	8	8	9	10
`telemetry`	6	7	6	4

chat is the highest-value package in the framework. A Go library that talks to several AI providers with one interface is exactly the thing people want without a CLI framework attached. Ten out of ten, no argument.

It also scored four for ease, the joint-worst in the table. Because at that point chat reached into props, the GTB HTTP helpers, the config and credentials abstractions, and the logger. Each of those was a small sensible line on its own, and together they had the package thoroughly bolted to the floor. So the most valuable thing I owned was the hardest to shift! Meanwhile redact and regexutil, both tens for ease, would have taken a morning apiece and delivered a fraction of the value.

So value and ease aren’t correlated, which is obvious once it’s written down, and invisible until you score them separately, because instinct blends the two into one vague feeling that we should probably do something about chat. That feeling can’t tell you whether to do it first or last.

What the numbers were actually for

Sequencing, as it turned out, more than judgement. Once the two questions are separated, the running order writes itself, and the report ends with six steps:

Low-coupling leaf utilities. redact, regexutil, browser, workspace.
User-facing CLI helpers. output, forms, logger, changelog.
Security and runtime foundations. credentials, authn, tls.
controls, then http and grpc onto it as optional transport adapters.
vcs/release and the provider adapters.
chat, last, once there were local interfaces to replace the GTB bits it was clinging to.

Easy things first, not because they matter most but because each one takes a bit of coupling out of the framework and makes the next one cheaper.

By the time you reach chat, half of what made it a four has already been dismantled by the work in front of it. The other thing the scores bought was permission to stop. Anything at five or below was excluded from consideration entirely, which sounds like a small administrative decision and isn’t. Without it, a scoring exercise turns into a to-do list with twenty-seven items on it and no way to argue against any of them.

What it didn’t tell me

The table is a good instrument, and it was wrong about several things in the direction I’d least expected: the easy ones. logger scored ten for ease. forms scored ten for ease. Both sat in step two, both looked like a morning’s work each, and neither of them is a module today. One was deleted outright and the other is still sat in the framework where it started.

Being wrong about chat would have been ordinary. Being wrong about the packages I’d already marked as trivial is the interesting failure, and a hundred and eight numbers later the column I’d been most confident about was the one that lied to me.

The packages that didn't want to leave

Fri, 02 Oct 2026 00:00:00 +0000

There are sixteen directories in go-tool-base’s pkg/ today, and that’s after the extraction. I counted them twice, on a Sunday afternoon, because the first count didn’t seem right.

The part that doesn’t get written up

Decomposition stories are always told through what came out. Here’s the module, here’s the import path, here’s the tidy little go.mod with three lines in it. Fine, but I’d scored twenty-seven packages before any of that started, and the scoring was the easy bit.

The hard calls were the ones where the answer came back no. There were four of them, and no two of them had the same reason, which is the thing I took away from it: “extract or don’t” turned out to be nowhere near a big enough vocabulary. Ask a binary question and most packages eventually become modules, because “no” tends to look like you couldn’t be bothered.

Delete it, don’t extract it

pkg/forms was the wizard layer, the prompts you get when you run gtb generate project and it asks what you’d like it to build. It scored ten out of ten for ease of decoupling: trivial to lift out, no framework coupling worth the name, straight into wave two alongside the other easy wins. And then, before touching it, the obvious question (the one I should perhaps have asked first)… does this still earn its keep at all?

It didn’t. huh had moved on to v2 and absorbed the entire purpose of the thing, so what forms was doing in 2024 was papering over gaps that no longer existed, and I’d have been extracting a wrapper around a library that had learned to do the job itself.

So it went. Commit 8b5732c0 rewrites the wizards on native huh v2 and deletes the package in one go, which is a very satisfying shape for a commit to have.

The most valuable extraction decision I made that week was an rm -rf.

Keep it, because the world has enough of these

logger also scored ten for ease, also went into wave two, and also never left. Completely different reason, and it’s the call I’m surest about of the four. I feel like there are too many loggers out there at the moment and adding another package to that pile becomes its own problem, this implementation is very specific to gtb, and all the packages I’d already extracted use slog as their intersection.

That second half is the bit that matters. If half a dozen extracted modules all talk to slog, then the standard library is already the shared interface, and there is nothing left for a go/logger module to do except sit in the middle of a relationship that was working fine without it.

Extracting it wouldn’t have shared anything! It would have manufactured a dependency, on me, in every module that adopted it, in exchange for an interface Go already ships. The framework keeps its own logger because the framework has opinions about how a CLI should look when it talks. That’s a product decision, and I’m not sure a product decision is something you should hand to other people as a library.

Keep it, because it really is framework-shaped

telemetry is the opposite case, and it got the opposite treatment. I suspected it was too tangled to be worth extracting, but a suspicion is not a verdict, so it went off to be examined properly: dig into the design limitations, work out whether it even warrants extraction, don’t assume.

Verdict: staying, and definitely staying.

How I got there matters rather more than the answer did, though, because if I’d just trusted the hunch I’d have got the same result and learned nothing, and the next hunch would have been every bit as unexamined. The rubric exists so these calls aren’t vibes, including the ones where the vibe was right all along.

Or widen it, which I didn’t see coming

The fourth one refused both options. go-tool-base carried a wide GitHub client, and a lot of that code was left over from an earlier incarnation of the framework, written while I was at a very GitHub-centric employer. Extracting it would have shipped one vendor’s API as though it were an abstraction; deleting it would have thrown away work that did something useful.

So it got widened instead: the whole thing became the go/forge contract with per-provider adapters underneath, and today there are four of them. That has its own story and I’ll write it up properly another time, because the interesting part isn’t the refactor, it’s carrying a previous job’s code around for years without noticing what shape it had left you in.

Five words, not two

So here’s the vocabulary the programme actually needed.

Extract. Real reuse value, and the coupling can come out.
Keep, too specific. It’s framework-shaped. Someone else would have to bend it to fit.
Keep, too generic. The ecosystem has this covered, or the standard library already is the interface.
Delete. Something else grew into the job while you weren’t looking.
Widen. It’s the right idea, dressed for one vendor.

Only the first is what people mean when they say “extract”. Two of the others leave the package where it was and neither is a shrug, one kills it outright, and one keeps the idea while throwing the implementation away. Five different answers, and a yes/no question can only ever reach two of them.

What’s left is what it is

After the waves, pkg/ holds sixteen things: the CLI wiring, props, setup, docs, the transports, the logger, the telemetry, version, chat. That’s not a leftovers pile so much as the answer to “what is go-tool-base, actually”, written out as a directory listing, and I couldn’t have told you it before I started. I suspect most frameworks have a list like this, and never find out what’s on it, because nothing ever forces the question.

I do think the whole exercise was worth it, and said so at the time in rather more exclamation marks than were strictly necessary, but the modules aren’t the thing I’d point at.

Sixteen directories, and now I know why each one is still there.

The extraction playbook only got real when chat left

Mon, 05 Oct 2026 00:00:00 +0000

I wrote the playbook before I’d done the hard one, which, said out loud, sounds a bit daft.

It didn’t feel daft at the time (they never do), and I’d do it again.

Why there was a playbook at all

Two things existed before it. signing had already left, months earlier, back when extracting a module was a one-off rather than a programme. That gave me the mechanics: a framework-free module implementing a small contract, consumed back by the framework through adapters, with its own CI, its own docs site, its own dependency-footprint guard. It worked. The spec later calls it the validation dry-run, which is generous… at the time it was just “the signing thing”.

Then came the report, which said which packages should go and in what order. Neither of those tells you how.

And I could see what was coming: a dozen extractions, each one re-deciding the same handful of questions. What do we call it? Where do the docs live? What does the repo need before it counts as real? When does the framework cut over? Get that wrong and you don’t get a toolkit, you get a dozen modules that each look like whichever week they were built in.

So on the 12th of July I wrote it down, and the spec is blunt about its own purpose: codify the repeatable process and conventions so every subsequent extraction is consistent and recognisable, rather than re-decided each time. A meta-spec, if you like, whose job is to stop other specs arguing.

Eight steps and a lot of opinions

The procedure came out at eight steps, and they’re deliberately boring:

Bootstrap the repo, CI green on an empty scaffold.
Move the code, and sever anything still reaching back into the framework.
Tests and guards carried over, coverage held above 90%.
Diátaxis docs and a Pages microsite.
Cut v0.1.0.
Framework cut-over in a single MR: add the dependency, delete the in-tree package.
Cross-reference the new microsite from the framework’s docs.
Downstream consumers repoint.

Alongside that, the naming: a subgroup rather than a prefix, so it’s go/chat and not gtb-chat. Bare component names, because recognition comes from the subgroup rather than from stuttering the framework’s name into every import. Docs subdomain mirroring the module path, so go/chat becomes chat.go.phpboyscout.uk and there’s nothing to remember. All very tidy. And all written by a man who had extracted one thing, and an easy one at that.

And then chat

The playbook names pkg/chat as the first greenfield extraction, in its own words. chat is also the package that scored 10 for downstream value and 4 for ease of decoupling, the widest gap in the whole table, and the reason the running order put it last rather than first.

So the first real test of the process was the worst case in the building, not quite by accident, but not by design either, just where the two documents happened to point at each other. And it went through all eight steps: go/chat exists, it’s on its own release line, and its go.mod mentions go-tool-base zero times! That was the whole point, and very much the hard part.

And then it broke step six.

The step that didn’t survive

Step six says: add the module dependency, delete the in-tree package. Open go-tool-base today and pkg/chat is still there.

Ten files of it: config_adapter.go, adapter_helpers.go, providers.go, a doc comment and some constants, each of them importing gitlab.com/phpboyscout/go/chat and doing something to it. I don’t think that’s a failed cut-over so much as a category the playbook didn’t have.

Because a framework that consumes an extracted module still needs somewhere to do the paperwork. go/chat takes typed settings and knows nothing about config containers, hot reload or the environment-prefix rules, and that’s why it’s reusable, but something has to turn the framework’s resolved configuration into those typed settings and that something has to live on the framework’s side of the line. It’s the border pattern again, arriving from a completely different direction and refusing to be optional.

So what step six actually means, now that something hard has been through it, is closer to delete the implementation, keep the border post. The name on the directory stays the same, and what’s inside it changes from a few thousand lines of chat client to a few hundred lines of translation.

I’d write it that early again

The playbook was written too early, and I’d do it again tomorrow. Steps one to five held perfectly: naming, repo scaffold, coverage bar, docs site, version cut. A dozen modules later those are the reason the toolkit looks like one thing rather than twelve, and not one of them has been re-argued since.

What the hard case changed was one step, and it changed it in a way I could never have reasoned my way to in advance. I’d have written “delete the package” every time, because that’s what happened on signing… and signing really did have nothing left to translate.

That isn’t an argument for waiting until you’ve done the hard one, because then you never write it at all and every extraction is a fresh argument. It’s more that a process document is a hypothesis. It looks like a rule, it’s formatted like a rule, people follow it like a rule, but until something difficult has run all the way through it you’re reading a description of the easy case with the confidence of a standard.

chat was the hard one, and it went first through the door marked first greenfield extraction, and it came out the other side having edited the instructions on its way past.

Config is a border, not a chain

Tue, 08 Sep 2026 00:00:00 +0000

go-tool-base was meant to be an all-in-one framework, and for a good while that was exactly right. Then I started wanting bits of it on their own. Signing went first, off into its own module, and that went well enough that I lined up the chat package to follow.

So I costed it. To let a package go off and find an API token by itself, I’d have to take Viper along. And afero. And fsnotify, pflag binding, and every last scrap of environment-prefix behaviour we’d accumulated over the years.

All that, so a package could learn a string.

The helpful thing had turned into the fence

I’d done this dance once already with logging. A package that reaches for the framework’s logger is a package that can’t leave, and the answer was to hand it an interface instead of a dependency. That one’s written up in a logging interface that doesn’t leak its backend.

Config was the same trap in a much better disguise. Worse, actually, because config is genuinely useful. Everything wants some. Our container was the one thing in the building that knew how a value had really been resolved: which file it came from, whether an env var had beaten it, what the default was if nobody said anything at all. That’s a real service. The packages were right to want it.

The trouble was what wanting it looked like once you opened the file. pkg/chat reached into that container in three separate places. Picking a provider. Working out a fallback. Resolving a credential. Three small, sensible, entirely defensible lines. And between them, a chain bolting the package to the framework’s floor.

The obvious answer, which was wrong

Extract config first. Do it before anything else, because it’s the blocker.

I said that out loud, quite confidently, and it has a lovely logic to it. If the thing in the way is config, move config.

It’s wrong, and it took me an embarrassingly long time to see why. Our config isn’t a data structure. It’s a set of promises about resolution order. File, then environment, then flags, with a prefix convention and hot reload sitting on top of the lot. Pull that out wholesale and you either drag every one of those promises into every consumer, or you break them somewhere nobody notices until it’s in production and somebody’s PHPBS_ prefix has quietly stopped winning.

And I had one line I wasn’t crossing. Whatever I did here, a user’s config file had to carry on working exactly the way it did yesterday. Not mostly. Exactly.

Paperwork, not a tow rope

So it went the other way round.

Instead of packages reaching into the framework, each package writes down what it needs as a plain typed struct of its own, and the framework fills it in at the boundary. In Go that’s just unmarshalling: take a lump of resolved config, pour it into a struct the package defined for itself.

Think customs paperwork rather than a tow rope. The package says “endpoint, timeout, token, please”. The framework, being the only thing that actually knows where any of those came from, fills in the form and hands it over. Nothing crosses the border except values. Constructors ended up in two flavours. A plain one that takes typed settings and knows nothing whatsoever about the framework, and an adapter next to it that takes the config container, does the paperwork, and calls the plain one.

Reusable as the default, framework compatibility as the special case. Which is the right way round, and I got there via one thoroughly wrong turn… I had it backwards at first, with the typed version wearing the awkward name, and had to go back and revert the lot.

The thing that actually made it stick wasn’t a code rule at all. It was a testing rule. Package tests use typed settings. Only the adapter’s own tests are allowed to build a config container. Sounds like bookkeeping, and it’s the most useful line in the whole design, because the moment an ordinary test reaches for that container you’ve found a leak.

Then hot reload nearly did for it

Here’s the wall I walked into.

A package is holding a struct of values that got copied at construction. Somebody edits their config file while the thing is running. That package is now sat there holding a polite lie. The framework knows the value moved; the struct hasn’t the faintest idea. So I built observers for it. The adapter notices a reload, rehydrates its typed settings, and the package behind the border ends up holding the new values instead of the old ones.

The bit that mattered wasn’t the mechanism though, it was making it compulsory. I’d written it up first as guidance, which is a polite way of saying optional, and optional is exactly the sort of thing that gets skipped on a Friday afternoon and bites you six months later when nobody remembers the rule existed. So it got tightened into an actual rule: long-lived adapters use the common helper, and the helper does the lot for you.

Which is ObserveSection. Initial unmarshal, register an observer, then publish a fresh snapshot whenever the typed section genuinely changes. Not when any old key moves, which would have every package in the building waking up because somebody fixed a typo in a log level. When that struct differs from the last one it published.

There’s a SectionChange carrying the before and after too, so a package can see what actually moved rather than just being told something did. The reload machinery itself is reloading config without a restart; this is the bit where the packages behind the border finally get told about it.

And then I moved it anyway

Now. Everything above argues that config stays put. Crown jewels, holds the promises, packages get paperwork and the container never leaves home. I believed that. I wrote it down. I based a fortnight of refactoring on it.

Within a fortnight I’d extracted it.

go/config went off and became its own module, and then I did something considerably ruder than merely moving it… I tore Viper out and rebuilt the thing around a Store with real provenance on every value. spf13/viper doesn’t appear in that go.mod at all now. pkg/config doesn’t exist in go-tool-base. The commit that finished the job is called, with no ceremony whatsoever, refactor(config): consume the extracted go/config module, and these days the framework depends on go/config v0.13.3 like any other library off the shelf.

So by the letter of it I was wrong. The premise of the whole exercise was that config wouldn’t move, and it moved.

Except nothing noticed! Not one package. chat, tls, http, grpc, the gateway, the telemetry core, every last release provider… none of them changed, because none of them was reaching through the border to begin with. They were sat there holding their own structs, filled in by an adapter, and swapping out the entire machine behind that adapter turned out to be somebody else’s problem entirely. Mine, obviously. But only mine, and only on one side of the line.

What the border actually bought

I built it so I wouldn’t have to move config. Then I moved config, and the border is the only reason that didn’t hurt.

Which isn’t the lesson I set out to learn, and it’s a better one. I thought I was buying extractability for the packages. What I actually bought was the freedom to rip up my own side of the line without asking anyone’s permission, because the only thing I’d promised them was a shape. Where the values came from, how they layered, what beat what… all of that stayed mine. And in the end I changed the lot of it. Viper’s gone, and chat still doesn’t know.

The writer I added, and the Viper I deleted by accident

Fri, 11 Sep 2026 00:00:00 +0000

keryx has a studio, and the studio saves as you work. Change a setting, it goes to disk, the app picks it up and carries on. Nothing exotic about that. Every desktop app written in the last thirty years does the same thing.

So keryx put in a request against go/config, which had only just been extracted into its own module and was feeling rather pleased with itself. The ask was boring to the point of being polite: when you write the file back out, keep the comments the human wrote, keep the key order, keep the formatting, and only touch the file that actually owns the key I changed. Nobody sensible rewrites a config library over comment preservation.

What “save your config” was really doing

Here’s the problem with a config library that merges layers for you. By the time it can answer a question, it has already thrown away the one thing you’d need to write an answer back.

Viper resolves your configuration down into a single view. Defaults at the bottom, then the file, then environment variables, then flags, each one covering the last. For reading that’s exactly right. You ask for server.port, you get the winner, and nobody has to care where it came from. Then you ask it to save. And it writes out the view. Not your file. The view.

Two things fall out of that, and the second one made me put my coffee down. Every comment the author wrote is gone, because comments were never in the merged view to begin with. And every value that came from somewhere else is now sat in your file, promoted to a permanent fact on disk. Defaults you never chose. Flags from one invocation last Tuesday. And environment variables.

So a user exports an API token, runs your app, your app helpfully saves its configuration… and you have written their token into a file in plain text. Nobody asked for that. Nothing warned them. The save button did it. I’ve banged on before about how everyone reads your config and how a configurable AI endpoint is an attack surface, and this is the same beast wandering in through a door I’d never thought to check.

And you can’t wrap your way out of it. The thing you need, which is which layer did this value come from, gets discarded during the merge, long before anybody contemplates a write. It isn’t hidden away somewhere awkward. It doesn’t exist.

The question I didn’t want to ask

I went into this fond of Viper and fully intending to keep it. It had done us proud for years and I had no appetite whatsoever for a rewrite in the middle of an extraction. Find the right place to hook in, bolt a writer alongside, done.

An hour in, I asked myself the question that ended that plan. Was Viper actually the problem here? Were we tying ourselves in knots honouring its contract, when what we fundamentally needed was a thing it was never going to give us?

Then, because I didn’t remotely trust myself to answer that fairly, I rigged the test in Viper’s favour. Greenfield exercise. Pretend there’s no existing code and no incumbent, write down everything this module needs from a config component, and only then go back and count how many of them Viper covers. I expected about 70%. More to the point, I wanted about 70%, because 70% is a lovely comfortable number that lets you keep what you’ve got and paper over the rest.

It came out at 20%.

That needs a caveat or it’s just unfair. Viper’s own docs reckon it does about 95% of what most projects need, and I’d say that’s honest. Both numbers are true at once. It’s excellent at the common case, and this module had walked itself somewhere distinctly uncommon by deciding that writing files properly was a first-class feature rather than a hack bolted on the end.

Record it on the way through

The fix falls straight out of the diagnosis. If the problem is that provenance gets binned during the merge, then record it while you merge.

So go/config grew a Store that owns configuration I/O end to end. It layers the sources itself, and as it goes it keeps track of which layer every value came from. Reading is View(), which hands back resolved answers exactly as before. Writing is Apply(), which takes a set of changes and, crucially, can work out where each one belongs.

Change a key that came from the project file, and the project file gets edited. A value that arrived from the environment was never yours to persist, so it doesn’t get written anywhere at all.

The editing goes through a document layer that rewrites bytes in place rather than re-encoding, which is what saves the comments and the ordering. Where a change spans several files they commit one at a time, each writing to a temp file and renaming it over the target, with a rollback if a later one falls over. Sequential and boring, deliberately. There was a fan-out design floating about at one point that would have made it concurrent and exciting, and I’m glad it never shipped.

Then, somewhere near the end of all that, I looked at what Viper was actually still doing.

Nothing. Not a thing. Everything left had already been taken over: the Store did layering, resolution and precedence, the codecs did formats, the document layer did writes. Whatever remained was being handled directly, and rather better, by the two libraries Viper wraps. It was a box on the diagram with no arrows coming out of it. I sat there for a few seconds going, hang on… Viper’s gone? Nobody decided to remove it. It just ran out of reasons to be in the building.

I never actually decided to

I opened that session having explicitly ruled out replacing Viper. First path considered, first one rejected, on the entirely sensible grounds that it was wildly disproportionate to a request about preserving comments. I closed it having replaced Viper, in commit 1f5129c, without ever actually deciding to. It wasn’t a decision. It was a consequence. Solve the provenance problem properly and the dependency stops earning its keep all by itself.

For the record, Viper went and its neighbours stayed. afero, cast, pflag and mapstructure are all still sat in that go.mod doing jobs they’re good at. This was not a purge, and anyone who tells you they removed a whole ecosystem in an afternoon is describing their go.mod, not their code.

The bit that still nags

The rewrite was the interesting work and the reversal is the better story, but neither is the thing I keep coming back to.

It’s that the real bug was never “the writer doesn’t preserve comments”. It was “saving your config can write your secrets to disk”, and it had been sat there the whole time, in a library I’d used happily for years, down a code path I’d simply never had cause to walk. It took a feature request from an unrelated project, about comment preservation of all things, to march me into it. I only found it because keryx wanted its settings file to stay tidy.

A format per module, not a fatter config core

Mon, 14 Sep 2026 00:00:00 +0000

Rewriting go/config around a Store fixed the thing I’d set out to fix, and handed me a much more obvious problem on the way out the door (it was about ten at night, and I’d been feeling rather pleased with myself). The new one read YAML.

Only YAML.

Viper, which I’d just deleted by accident, reads a dozen file extensions. So the objection writes itself before anyone has to be polite about it: a team with a config.json can’t adopt this at all, and telling them to convert their config file to suit my library isn’t a serious answer to anything.

I wasn’t putting five parsers in the core

The obvious fix is five more parsers in the core. Most config libraries in most languages have stood about here, and most of them took that road.

I didn’t fancy it, and the reason was already sat in the estate. Two other modules had solved the same shape of problem… chat has a sibling module per AI provider, forge has one per git host. So the question became how to support the other formats without bloating the core. Adapter projects, one per file type, the way chat and forge already do it. YAML stays the default, and if you want XML you import the XML one and say so.

And then the naming call, which matters a good deal more than it looks. One module per file type, named for it. config-json, config-xml, config-toml. Specifically, and only, to keep the dependency footprint lean.

Why a module and not a package

This bit is peculiar to Go and it catches people out, so it’s worth a plain-English paragraph. Go is good about code. Import a package you never call and the linker drops it; your binary doesn’t carry it. So on the face of it, one big config-formats package with all the parsers in costs a JSON user nothing at runtime.

Except dependencies aren’t worked out per package. They’re worked out per module. One go.mod, one go.sum, listing what the whole module needs. Put five parsers in one module and everybody who imports it inherits all five parsers’ requirements, whether they touch them or not. Their go.sum grows. Their vulnerability scanner has more to say for itself. Their audit has more to read. The binary’s fine and everything around the binary is a bit worse. One module per format is the shape where the graph tells the truth, or at least the only one I could find.

There’s a second benefit I didn’t plan and now rate higher than the first: a format either has a module or it hasn’t. There’s no list of supported formats sat somewhere, slowly drifting out of step with reality.

I found the next bit while I was measuring, and it made me laugh.

What Viper advertises, and what Viper ships

Viper v1.21.0 exports the list of extensions it supports. Twelve of them:

SupportedExts = []string{"json", "toml", "yaml", "yml", "properties", "props",
 "prop", "hcl", "tfvars", "dotenv", "env", "ini"}

Its internal/encoding directory, where the codecs actually live, contains four. dotenv, json, toml, yaml.

HCL is on that list. So are tfvars, INI and Java properties. Point Viper at a .hcl file and it’ll happily accept the extension, get as far as wanting a decoder, and not find one. Now, that’s a maintenance artefact rather than anything underhand, and those formats were almost certainly real once. But it is the failure mode that “a format either has a module or it hasn’t” makes structurally impossible, and I only spotted it because I’d gone looking for a number to stick in a spec.

Right, some actual numbers

Assertions about dependency weight are cheap, so here are some real ones. All the figures come from the same probe: an empty module that imports exactly one thing, then go list.

What you import	Modules	Non-stdlib packages
`spf13/viper` v1.21.0	26	53
`go/config` core (YAML)	28	23
core + `config-iofs`	29	24
core + `config-billy`	35	26
core + `config-json`	33	28
core + `config-gcp-gcs`	249	478

Two things in there worth saying. The core pulls fewer than half the packages Viper does, while doing considerably more. That’s the result I was after. But it pulls more modules than Viper, because the estate is deliberately built out of small ones. Measure the thing I optimised for and I win comfortably; measure module count and I lose a bit. Both numbers are real, so both go in the table.

The other is that bottom row. Reading config out of Google Cloud Storage costs 455 packages more than the core on its own, and there’s no clever trick that makes that cheaper, because most of it is Google’s client libraries and they are what they are. And that’s rather the point: the only people paying that are the people who asked for it. The spread from config-iofs adding a single package to config-gcp-gcs adding 455 is the whole argument, sat in one table.

One nuance, which I couldn’t put into words for a while. Early on I waved the dependency question away for myself, because keeping libraries current is a normal cost of maintaining anything and Renovate does most of the legwork. Still true. The footprint argument was never about the dependencies I maintain. It’s about the ones I impose on someone else, who may never open that parser as long as they live.

Two things I got wrong, left on the record

The specs carry their own reversals, deliberately unedited, and both are about writers. I’d assumed TOML would need a whole bespoke document module to edit files without mangling them, on measured evidence that no Go library round-trips TOML cleanly. That went down as a decision. Then it collapsed: go-toml/v2/unstable exposes each node’s source byte range, which turns a TOML edit into a targeted byte replacement of about two hundred lines, inline. The module I’d specced never got built.

HCL was the same mistake pointing the other way. I had it down as the hard writer, and it turned out to be the easy one, because HashiCorp already ship hclwrite and it does structure-preserving edits properly. config-hcl shipped reading and writing in one pass. The spec puts it plainly:

inverts the assumption in the first draft of D11, which treated writing HCL as the hard part. It is not; the hard part was always the semantics.

Both mistakes had the same cause. I’d been thinking about the editor as the difficult component, when the parser is the reusable primitive and the editor is thin work on top of it. Get the primitive right and the writer is a couple of hundred lines.

The hard part, as the spec says, was semantics. config-hcl refuses to evaluate anything at all: var.*, ${local.x}, functions, include, all rejected at load and by name rather than resolved. HCL-as-configuration, explicitly not Terraform. Silently resolving half a Terraform file and handing back a plausible-looking answer is far worse than refusing the thing outright.

The format I threw out

HOCON didn’t make it, and the reason is the best argument for the whole design. There’s essentially one usable HOCON parser in Go. Reading it, it hardcodes environment-variable fallback, so a probe with ${LEAKED_SECRET} in a config file resolved it straight out of the process environment. That’s the very behaviour the core has a rule against, for the reasons in the last post about save-your-config writing your tokens to disk.

So the choice was ship a format whose only implementation defeats a core guarantee, or don’t ship it. Viper never advertised HOCON either, so declining costs nothing against any parity claim I might want to make later. It’s out. Meanwhile XML, INI and Java properties ship deliberately read-only, and say so on the tin. A half-working structure-preserving writer that mangles someone’s file is worse than no writer at all, and “we support writing” isn’t a claim I want to make loosely.

Every last one of them

Same decision, made three times over, gave three families. A module per format, a module per filesystem, a module per backend system. Each one gets its own approved spec before anyone writes a line, because the expensive part of a remote adapter is understanding the other system’s semantics rather than writing the Go.

Here’s the lot of them as they stand. Each links to its page on the core docs, because adapters don’t get their own docs site in this estate… they carry a strong README and signpost home.

Adapter	Family	What it gives you
`config-json`	format	JSON, read and write
`config-toml`	format	TOML, read and write
`config-hcl`	format	HCL as configuration, never evaluated
`config-dotenv`	format	`.env` files
`config-ini`	format	INI, read-only
`config-properties`	format	Java properties, read-only
`config-xml`	format	XML, read-only
`config-iofs`	filesystem	any `io/fs.FS`, including `embed`
`config-afero`	filesystem	afero
`config-billy`	filesystem	go-billy, as used by go-git
`config-sftp`	filesystem	config over SSH
`config-aws-s3`	filesystem	an S3 bucket as a filesystem
`config-gcp-gcs`	filesystem	Google Cloud Storage
`config-azure-blob`	filesystem	Azure Blob Storage
`config-consul`	backend	Consul KV
`config-etcd`	backend	etcd v3, with real compare-and-swap
`config-vault`	backend	HashiCorp Vault
`config-filekv`	backend	mounted files, so Kubernetes secrets
`config-keychain`	backend	the OS keychain
`config-aws-ssm`	backend	SSM Parameter Store
`config-aws-secrets`	backend	Secrets Manager
`config-gcp-parameter`	backend	GCP Parameter Manager
`config-gcp-secret`	backend	GCP Secret Manager
`config-azure-appconfig`	backend	Azure App Configuration
`config-azure-keyvault`	backend	Azure Key Vault

Twenty-five. That number was twenty-four when I started writing this, because config-etcd shipped while the draft was open! That’s either a decent argument for the architecture or an argument about my inability to finish anything. And the one I was convinced would break the taxonomy didn’t. keryx wanted to keep credentials in the OS keychain, and for a good while I couldn’t have told you which family that belonged to. Looked a bit like a filesystem. Behaved a bit like a backend.

Turns out it’s a backend, and a fairly unremarkable one at that. macOS Keychain, Windows Credential Manager, Secret Service on Linux, sat in the layer stack like anything else, so a refresh token finally has somewhere to live that isn’t a plaintext file sat next to the port number. Three tidy categories, twenty-five modules, and the awkward one filed itself. I’ll take that. The twenty-sixth will be along before I’ve finished the next post, no doubt…

YAML has comments for a reason

Mon, 14 Sep 2026 00:00:00 +0000

There’s a file on your machine that a person wrote by hand. A config file, probably. It has comments in it, because whoever wrote it knew the next person would need them: why the port is that port, which flag needs a firewall change, a note to keep a list alphabetical. The blank lines are where they are on purpose. So is the alignment of the inline comments, because a person lined them up.

Now change one setting in that file with a program, and look at the diff.

I’ve spent most of a weekend on that diff (it’s the Monday after as I write this, and I’m a tad cross-eyed), so bear with me while I explain why.

Two jobs, one word

“A YAML library” covers two different tools that happen to share a file format, and most of the fickleness people complain about (and YAML is fickle, in every language I’ve used it from) comes from using the one for the other’s job.

A value library turns YAML into Go values and back. It’s what yaml.Unmarshal into a struct is for, it’s fast, and it’s lossy by design: comments, key order, quoting, blank lines and layout have no representation in a map[string]any, so they can’t survive one. That’s fine for data a machine wrote and a machine will read.

An editing library changes one thing in a document and leaves the rest exactly as the author wrote it. That’s what a config set command needs, and a settings screen over a file people also edit by hand, and a bot that bumps a version across two hundred repositories, and anything at all where the change is going to be reviewed. Every other ecosystem I can think of keeps the two apart: Rust has toml and toml_edit, HashiCorp has hclsyntax and hclwrite, Python has tomli and tomlkit. Go had the value half many times over and the editing half not at all.

I needed a rock solid YAML editor, not a values library. So I built one. It’s called yamldoc, the docs site is the place to start, and the rest of this post is why it exists.

Built on someone else’s assumptions

The first version wasn’t a library of its own. It was the document layer of go/config’s Store, the bit that writes a value back to your file, spun out into its own module the same day it was written, and it was functional… but not perfect. It sat on goccy/go-yaml, which is a fantastic parser and, for a while, the best syntax tree of the bunch. It also had 149 open issues and 70 open pull requests at the time of writing, with issues going back to 2020 and pull requests waiting since 2022, so the odds of getting a contribution in were starting to look slim. There are more active forks, of course, and I’d have happily lived on one of them if the problem had been a bug.

The problem was an assumption. Comments in a file were effectively ignored… not discarded, goccy is better than that, just ignored. They get attached to the nearest token and carried along in the hope they’ll land somewhere sensible on the way out. Every editing feature I wanted was built on someone else’s idea of what an editor needed, and I needed more than that: higher fidelity, more opinionated in the places I cared about and less in the places I didn’t, and it absolutely had to honour the YAML specifications rather than one library’s reading of them.

Comments should be first-class citizens of a file, not an afterthought. YAML isn’t just a machine-readable format, it’s meant to be human-readable too… why else would a serialisation spec include comments at all? For pure machine readability, well, that’s what JSON is for!

So over a weekend (a long one, more on that in a minute) the wrapper became something I wasn’t prepared to keep, and yamldoc got an engine of its own. No parser underneath it. It’s spec 0008 if you want the hundred and seventy-odd decisions that took, and I suspect that’s a post in itself.

What it does to a file

Let’s take a file. A small one, with the things a person puts in a file: a comment explaining a decision, two inline notes lined up so they read as a column, and a blank line between the sections because they are different sections.

# Which port the public listener binds to.
# Changing it needs a firewall change too.
server:
 port: 8080 # not 80, we sit behind a proxy
 host: localhost  # loopback only in dev

# Feature flags. Keep alphabetical.
features:
 beta_ui: false

Now the job. A user runs myapp config set server.port 9090. One key, one new value, and the file should come back with one line different. I ran that edit through three libraries, four ways, each through its own API (the harness and the exact versions are linked after the outputs), and what follows is their output, byte for byte.

The official library, the map route

go.yaml.in/yaml/v3 is the YAML organisation’s own Go library (yaml/go-yaml), the continuation of the gopkg.in/yaml.v3 everybody learned on, which is archived now; the two produce identical bytes here, and so does the v4 release candidate. The usual route is yaml.Unmarshal into a map, change the value, yaml.Marshal back out. At its defaults you get this:

features:
 beta_ui: false
server:
 host: localhost
 port: 9090

The port changed, so in one sense it worked. Everything else is gone. Both comment blocks, the two inline notes, the blank line. The indentation has doubled, because four spaces is the marshaller’s default. And look at the order: features now comes before server, and host before port, because a map has no order and the marshaller sorts keys alphabetically on the way out. The author wrote the port first because it’s the one you change; the library has an opinion about that and it wins. This is what “lossy by design” looks like on a real file, and I’d stop short of calling it a bug, because it’s a value library doing what it’s for.

The official library, the node route

The same library has a second door, yaml.Node, which keeps comments. Walk the tree to server.port, change the value, encode. With the indent set to two, because at its default this route also re-indents every line to four, you get this:

# Which port the public listener binds to.
# Changing it needs a firewall change too.
server:
 port: 9090 # not 80, we sit behind a proxy
 host: localhost # loopback only in dev
# Feature flags. Keep alphabetical.
features:
 beta_ui: false

Better. The order held and the comments survived. But the column of inline notes has collapsed to a single space, because the emitter has one rule for where a comment goes and applies it to every line, touched or not. And the blank line between the sections has vanished, because a blank line isn’t a node and the tree never had it. A reviewer looking at this diff sees four lines changed for a one-line edit, three of them in places no one asked for.

goccy, through its path API

goccy/go-yaml is the library the first yamldoc sat on. Through its path API it does this:

# Which port the public listener binds to.
# Changing it needs a firewall change too.
server:
 port: 9090
 host: localhost # loopback only in dev

# Feature flags. Keep alphabetical.
features:
 beta_ui: false

The blank line survived, which is why goccy was the best of the bunch for a long while. But the inline note on the line we edited has gone altogether, # not 80, we sit behind a proxy, which is the very comment the next person needs when they come to look at that port. And the note on the line below lost its alignment. This is the “ignored, not discarded” problem from earlier: the comment is attached to a token, the token got replaced, and the comment went with it.

yamldoc

And yamldoc gives you this:

# Which port the public listener binds to.
# Changing it needs a firewall change too.
server:
 port: 9090 # not 80, we sit behind a proxy
 host: localhost  # loopback only in dev

# Feature flags. Keep alphabetical.
features:
 beta_ui: false

One line changed, and only the four characters of the value on it. The comment on that line is where it was, still aligned with the one below. The blank line is there. The order is the author’s. Run a diff and the diff is the bump.

Why it comes back like that

The reason isn’t a longer list of special cases. The others parse into a tree and emit the tree, so whatever the tree doesn’t model is gone and the emitter’s opinions apply to every line. yamldoc edits the source: the bytes inside the edit’s footprint get replaced (here, 8080), and every other byte is handed back untouched. There’s no list of preserved constructs because there’s nothing that isn’t preserved.

Then it checks its own work. Every transaction re-parses the bytes it just produced, composes them fresh, and compares the result against a statement of what the edit meant that was written down before the writer ran: the values, the correspondence between old and new, the untouched bytes, and which comment belongs to whom. Only if all of that agrees does the new revision get installed. A bug in the writer gives you an error and an unchanged file, not a damaged one. I’ve had the damaged one (more than once, if I’m honest) and I’d take the error every time.

That’s one file, hand-picked to make the point. The measured version runs every library that can round-trip a document over eight real project config files, first with no edit and then setting each of seventy reachable scalars through the library’s own API:

Files back byte-identical with no edit: yamldoc 8 of 8, the next best 5.
Single-key edits that changed one line and one line only: yamldoc 70 of 70, the next best 24.
The official YAML test suite: all 306 valid cases parsed and all 94 invalid ones rejected. 304 match the suite’s pinned events and the other two match libyaml instead, and I’d rather tell you that than round it up. I did ask whether we could close the gap. The answer was no, and it’s good enough now we can explain it.

The harness, the versions and the raw output are in the comparison report, measured on the 13th of September, and the construct-by-construct table (CRLF, emoji, a comment before ---, a missing final newline, and the rest) is on the Should you use yamldoc? page, which is the plain version of that question, including the bit where the answer is no.

The four outputs above came from this harness on Go 1.27.1: go.yaml.in/yaml/v3 v3.0.5 (the archived gopkg.in/yaml.v3 v3.0.1 and the go.yaml.in/yaml/v4 v4.0.0-rc.6 candidate return the same bytes on this file), goccy/go-yaml v1.19.2 and yamldoc v0.6.1. Run it yourself, that’s rather the point.

What it costs

In the same breath, because it isn’t free. An edit costs four to seven times what the value libraries charge, and a decode about three times, because yamldoc keeps everything (every token, the whitespace, the comments, the spelling of every number) and then pays again to check the result. A parse is level with goccy. And a parse followed by writing the file back with no change is free, level with the fastest of them, because it never re-emits: it hands you the bytes it started with. Why an edit costs more is the accounting, stage by stage, and the instruction I gave for that page was to justify the expense against the safety, not to hide it.

What it cost me

The weekend, mostly, and ninety per cent of an OpenAI Pro subscription overnight. One run got carried away, seven hours of tokens on scaffolding and tests and precious little engine, because I hadn’t given it enough direction to know when to prefer implementation over testing. That’s on me, and it’s a war story for another day. I don’t think it was wasted, though… it was a real push of a new model and the results speak for themselves: a rock solid spec and testing out the wazoo. The bill for what it costs me long-term hasn’t come due yet… I’m happy with it so far.

Who it’s for

keryx uses it directly, to save its workspace file without wrecking what a person typed into it. Perhaps more to the point, any project built on go-tool-base relies on yamldoc as a default part of its configuration management, so as I carry on building cool shit, more and more of the estate ends up standing on it whether it knows or not. That’s the reason it has bounds on everything (bytes, nodes, depth, alias expansion, work per transaction) and refuses a hostile file with an error rather than a large allocation.

It is not a struct decoder and it won’t become one. Use it with a value library, not instead of one. If you’re reading YAML into a struct, go.yaml.in/yaml/v4 or goccy is the right tool and yamldoc will only get in your way. If you’re changing a file that a person maintains, this is the one. The docs are at yamldoc.go.phpboyscout.uk, getting started says to allow fifteen minutes, and the install is the usual:

go get gitlab.com/phpboyscout/go/yamldoc

f, err := yamldoc.Parse(src, yamldoc.Options{})
if err != nil {
 return err
}

d, _ := f.Snapshot().Document(0)
if err := d.Set("server.port", 9090); err != nil {
 return err
}

out, err := f.Snapshot().Bytes()

Parse bytes, change one key, write bytes back. The file comes back as its author wrote it, comment and all, and I’d suggest that’s what the spec had in mind when it gave you comments in the first place.

Layering config without losing where a value came from

Thu, 10 Sep 2026 00:00:00 +0000

Someone edits a setting, restarts the service, and nothing changes. The file is right. You can cat it back and it says exactly what they typed. The process carries on doing the old thing regardless, and now there’s a call open and four different places that value could be coming from.

I’ve lost afternoons to that, and the annoying part is that it’s never really a mystery. It’s just that nothing in the process can tell you which layer won.

So this walkthrough builds the whole chain a real service uses, defaults under a file under the environment under flags, and then spends most of its time on the more useful half: asking the thing which layer supplied a value, which layers lost, and what would happen if you wrote to it. Twenty minutes, one module, no services and no network past the first go get.

If you want the argument for why config is its own module rather than something the framework owns, that’s a separate post. This one is just using it.

This is the canonical walkthrough from the go/config docs, reproduced here so the two don’t drift. The docs are the source of truth: https://config.go.phpboyscout.uk/tutorials/layering/. I built the module and ran every step against v0.17.2 on Go 1.27.0 before this went out, and every output below is what came back, with one divergence flagged in step 6.

1. Create a module

mkdir cfgdemo && cd cfgdemo
go mod init cfgdemo
go get gitlab.com/phpboyscout/go/config github.com/spf13/pflag

pflag is only for the flags layer in step 5. The other three need nothing beyond the module itself, which is rather the point of it having been extracted.

2. Compile the defaults into the binary

A default isn’t a config file. It ships inside the binary, so the program starts correctly on a machine with no configuration at all, and it can never be written to, because there’s nowhere to persist it. That last bit matters later than you’d think.

main.go:

var builtinDefaults = []byte(`
server:
 host: 0.0.0.0
 port: 8080
 timeout: 30
log:
 level: info
`)

func main() {
	store, err := config.NewStore(context.Background(),
		config.WithReaders(config.NamedSource{
			Name: "embedded:defaults.yaml",
			Content: builtinDefaults,
		}),
	)
	if err != nil {
		log.Fatal(err)
	}

	v := store.View()
	for _, key := range []string{"server.host", "server.port", "server.timeout", "log.level"} {
		fmt.Printf("%-16s = %-10v %s\n", key, v.Get(key), v.Explain(key))
	}
}

$ go run .
server.host = 0.0.0.0 server.host = 0.0.0.0 (from default:embedded:defaults.yaml)
server.port = 8080 server.port = 8080 (from default:embedded:defaults.yaml)
server.timeout = 30 server.timeout = 30 (from default:embedded:defaults.yaml)
log.level = info log.level = info (from default:embedded:defaults.yaml)

Give the source a name you’d recognise at the wrong end of an incident. embedded:defaults.yaml tells you where to go and look; reader1 tells you nothing, and provenance is most of what this module is for.

3. Put a config file over the top

config.yaml, next to main.go. Keep the comments, step 7 comes back to them:

# Deployment settings for the demo service.
server:
 # The port the operators agreed on.
 port: 9090
log:
 level: debug

It sets two of the four keys deliberately. Add it to the store after the defaults:

	fsys, err := config.Dir(".")
	if err != nil {
		log.Fatal(err)
	}

	store, err := config.NewStore(context.Background(),
		config.WithReaders(config.NamedSource{
			Name: "embedded:defaults.yaml",
			Content: builtinDefaults,
		}),
		config.WithFiles(fsys, "config.yaml"),
	)

$ go run .
server.host = 0.0.0.0 server.host = 0.0.0.0 (from default:embedded:defaults.yaml)
server.port = 9090 server.port = 9090 (from config.yaml); also defined in default:embedded:defaults.yaml
server.timeout = 30 server.timeout = 30 (from default:embedded:defaults.yaml)
log.level = debug log.level = debug (from config.yaml); also defined in default:embedded:defaults.yaml

Precedence is the order you added the sources, and later wins. No ranking table to learn, no priority number to set. The file outranks the defaults because WithFiles came after WithReaders, and if you want it the other way round you move the line.

Two keys came from the file, two from the defaults, and the merge happened per key rather than per file. A file that sets one key doesn’t blank out the rest of the tree, which sounds obvious right up until you meet a library that does it the other way.

4. Add the environment

		config.WithEnv("CFGDEMO"),

$ CFGDEMO_SERVER_PORT=7000 go run .
server.port = 7000 server.port = 7000 (from env:CFGDEMO_SERVER_PORT); also defined in default:embedded:defaults.yaml, config.yaml

CFGDEMO_SERVER_PORT became server.port: prefix stripped, the rest lowercased, underscores into dots.

That prefix isn’t a nicety, it’s a security control. Without one, every variable in the process environment is a candidate configuration key: PATH, HOME, a token injected for something else entirely. The full rules, including what happens when an underscore is ambiguous, are in Environment variables.

5. Add flags on top

Flags go last, because the person typing one has the most immediate intent of anybody involved.

	flags := pflag.NewFlagSet("demo", pflag.ExitOnError)
	flags.Int("server-port", 0, "port to listen on")
	flags.String("log-level", "", "log verbosity")
	if err := flags.Parse(os.Args[1:]); err != nil {
		log.Fatal(err)
	}

and as the last store option:

		config.WithFlags(flags),

$ CFGDEMO_SERVER_PORT=7000 go run . --server-port 6000
server.port = 6000 server.port = 6000 (from flag:--server-port); also defined in default:embedded:defaults.yaml, config.yaml, env:CFGDEMO_SERVER_PORT

One value, four layers that could have supplied it, and the top one won. Dashes become dots the same way underscores do. When a flag’s name and its key genuinely differ, BindFlag is the way.

The bit worth stopping on: a flag you didn’t type contributes nothing

server-port has a declared default of 0. Run without it:

$ go run . --log-level warn
server.port = 9090 server.port = 9090 (from config.yaml); also defined in default:embedded:defaults.yaml
log.level = warn log.level = warn (from flag:--log-level); also defined in default:embedded:defaults.yaml, config.yaml

server.port is 9090, not 0. Only flags actually set on the command line join the layer, because the store asks pflag which flags were visited rather than reading what every flag’s value happens to be. Skip that and every unset flag’s zero value sits at the top of the stack burying every layer underneath, which makes the whole flags layer worse than useless.

It’s also why declared flag defaults are the wrong home for configuration defaults. Put those in the defaults layer, as in step 2, where anything can override them.

6. Ask which layer won, and which lost

Explain is the readable form you’ve been printing all along. Two lower-level calls give you the same facts as data, which is what you want when you’re rendering them somewhere:

	if src, ok := store.View().Origin("server.port"); ok {
		fmt.Printf("winner: %s (writable: %v)\n", src, src.Writable)
	}
	defined := store.View().DefinedIn("server.port")
	if n := len(defined); n > 1 {
		for _, s := range defined[:n-1] {
			fmt.Printf("lost: %s\n", s)
		}
	}

Origin names the layer whose value you actually get. DefinedIn lists every layer that defines the key, lowest precedence first, so its last entry is that same winner and everything before it lost. Slicing the last one off gives you the losers, which is how Explain builds its “also defined in” list, and it’s why the loop checks for more than one: a key only one layer sets comes back as a one-entry list. Writable on a Source is the field the next step turns on.

One that caught me out. That method used to be called Shadowed, and the tutorial said it listed the sources that defined the key and lost. At v0.17.2 it handed me the winner as well, so the old loop printed env:CFGDEMO_SERVER_PORT as having been shadowed by… itself. I raised it as go/config#10, and the answer was that the code was right and the docs were wrong: returning the whole chain is the specified behaviour, and the tutorial now says so. Fair cop. It’s since been renamed DefinedIn, which says what it does on the tin, and Shadowed hangs around as a deprecated alias, so on an older go/config the same loop works with the old name. With the loop above, the same run prints:
winner: env:CFGDEMO_SERVER_PORT (writable: false)
lost: default:embedded:defaults.yaml
lost: config.yaml

7. Write a value back, and watch routing skip what it can’t write

Here’s where the defaults layer being unwritable stops being trivia. server.timeout exists only in the compiled-in defaults, a layer with nowhere to persist to. Ask where a change would go before making it:

	plan, err := store.Plan(config.Set("server.timeout", 60))
	if err != nil {
		log.Fatal(err)
	}
	for _, op := range plan.Operations {
		fmt.Printf("plan: %s -> %s (effective: %v)\n", op.Change.Path, op.Target, op.Effective())
	}

plan: server.timeout -> config.yaml (effective: true)

Routing walked past the defaults because they can’t be written, and picked the highest layer that can. Apply it, and config.yaml becomes:

# Deployment settings for the demo service.
server:
 # The port the operators agreed on.
 port: 9090
 timeout: 60
log:
 level: debug

Both comments survived, port kept its place, and timeout landed in the section it belongs to. The file was edited, not regenerated from a parsed tree, which is the difference between a tool an operator will let near their config and one they’ll ban after the first time it eats their comments.

8. Write a value that a higher layer still shadows

Which brings us back to the call at the top. Write server.port while the environment is also setting it:

$ CFGDEMO_SERVER_PORT=7000 go run .
plan: server.port -> config.yaml (effective: false)
 shadowed by [env:CFGDEMO_SERVER_PORT]

The write isn’t refused. config.yaml really does get port: 5555, and I checked. But the plan told you, before you committed to it, that reading the key back will still give you 7000:

	if !op.Effective() {
		fmt.Printf(" shadowed by %s\n", op.ShadowedBy)
	}

That’s the whole afternoon I mentioned, handed to you as a boolean. The setting saved, the file is correct, the running process ignores it, and instead of three people staring at a file that is definitely right, something can say out loud which layer is winning and why.

What you built

Layer	Added with	Writable	Typical source
Defaults	`WithReaders`	✗	compiled into the binary
File	`WithFiles`	✓	an operator edits it
Environment	`WithEnv`	✗	the platform injects it
Flags	`WithFlags`	✗	someone typed it

Exactly one of the four can be written to, which is the only reason routing has a decision to make.

From here, Configure a service from Consul adds a remote layer that behaves precisely like the four above, and Precedence and merge is the reasoning underneath all of it, including what happens to lists and maps.

Four layers is not the hard part. Being able to answer why is.

When a project's config inherits yours

Sat, 19 Sep 2026 00:00:00 +0000

keryx has a set of house styles for the cover art on this blog, and they live in a library in my home directory. Any project I point it at gets the same eight. That’s what I want, right up until a project has to build on a runner that has never seen my home directory, and then it needs its own copy pinned in the repo.

So that tool now has two configurations, mine and this project’s, and nothing anywhere writes down which one wins.

You can carry both around and remember the rule. I did that for a while. It works until the morning someone asks why a setting they definitely changed is having no effect, and the answer is four levels down a call stack in an if nobody has read since it was written.

The better move is to stop treating them as two things. Make one config a layer of the other, so there is one store, one precedence order, and one place that can tell you where a value came from.

This is the canonical walkthrough from the go/config docs, reproduced here so the two don’t drift: https://config.go.phpboyscout.uk/tutorials/composing-stores/. Every command and output below was validated by running it against v0.17.3, including the bit where a file gets rewritten and keeps its comments.

It builds directly on precedence and routed writes, so do the layering walkthrough first if you haven’t. Twenty minutes, Go 1.26.5 or newer, and nothing else.

1. Create a module and two config files

mkdir cfgcompose && cd cfgcompose
go mod init cfgcompose
go get gitlab.com/phpboyscout/go/config

global.yaml, standing in for ~/.config/demo/config.yaml:

# The user's settings, shared by every project.
editor: vim
theme: dark
telemetry:
 enabled: false

project.yaml, standing in for a .demo.yaml in the repo:

# This project's settings, checked into the repo.
theme: light
build:
 target: wasm

They overlap on exactly one key, theme, which is the interesting one.

2. Nest the global store inside the project store

main.go:

package main

import (
	"context"
	"fmt"
	"log"

	"gitlab.com/phpboyscout/go/config"
)

func main() {
	ctx := context.Background()
	fsys, err := config.Dir(".")
	if err != nil {
		log.Fatal(err)
	}

	global, err := config.NewStore(ctx, config.WithFiles(fsys, "global.yaml"))
	if err != nil {
		log.Fatal(err)
	}

	project, err := config.NewStore(ctx,
		config.WithBackend(config.Nested(global, "global", config.NestedPromotable)),
		config.WithFiles(fsys, "project.yaml"),
	)
	if err != nil {
		log.Fatal(err)
	}

	v := project.View()
	for _, k := range []string{"editor", "theme", "telemetry.enabled", "build.target"} {
		fmt.Printf("%-18s = %-8v %s\n", k, v.Get(k), v.Explain(k))
	}
}

go run .

editor = vim editor = vim (from global.yaml)
theme = light theme = light (from project.yaml); also defined in global.yaml
telemetry.enabled = false telemetry.enabled = false (from global.yaml)
build.target = wasm build.target = wasm (from project.yaml)

Provenance survives the join. editor is reported as coming from global.yaml, the actual file, rather than from “the aggregate” or “the nested store”. The inner store’s layers pass through as a contiguous block at the position the backend was declared, each keeping its own source, so nothing is interleaved and nothing is anonymised.

That is worth pausing on. The whole reason to compose rather than juggle is to keep the straight answer at the end of it, and a join that flattens everything into one anonymous blob has thrown away the thing you were trying to protect.

The "global" id names the aggregate in error messages. It is not a layer name, because an aggregate contributes no layer of its own.

3. Watch an ordinary write stay local

editor is defined only in the global file. Routing’s usual rule is to edit a key where it already lives, so does a write to it reach into the global config?

	p, _ := project.Plan(config.Set("editor", "helix"))
	for _, op := range p.Operations {
		fmt.Printf("ordinary write: editor -> %s\n", op.Target)
	}

ordinary write: editor -> project.yaml

No. A nested store is never a routing candidate, even with NestedPromotable passed. An ordinary project-scoped edit lands in the project’s own file, and creates the key there.

That asymmetry is most of the design. If routing could reach inside, a mundane “set my editor for this project” would walk past the project’s file and rewrite the shared config every other project inherits. That is the destructive version of a helpful default, and you only find out about it weeks later, when three unrelated repositories have quietly changed their minds about something.

4. Promote a setting deliberately

Promotion, moving a setting up into the shared config, has to be asked for by name:

	promote := config.Set("theme", "solarized", config.To("global.yaml"))

	p2, err := project.Plan(promote)
	if err != nil {
		log.Fatal(err)
	}
	for _, op := range p2.Operations {
		fmt.Printf("promotion: theme -> %s (effective: %v)\n", op.Target, op.Effective())
		if !op.Effective() {
			fmt.Printf(" shadowed by %s\n", op.ShadowedBy)
		}
	}

	if _, err := project.Apply(ctx, promote); err != nil {
		log.Fatal(err)
	}

promotion: theme -> global.yaml (effective: false)
 shadowed by [project.yaml]

global.yaml really is updated, comment and all:

# The user's settings, shared by every project.
editor: vim
theme: solarized
telemetry:
 enabled: false

But effective: false is the bit you want. The project file still sets theme: light, so reading theme back in this project still gives you light. The promotion will show up in every other project, and not in this one until the local override goes.

Tell the user that, because it is the difference between a tool that saved their setting and one that appears to have ignored them.

NestedPromotable is what made this possible at all. Without it a nested store is strictly read-only and even a named write cannot reach in, which is the safe default and the usual case: a shared organisational base, a team standard nobody should be editing by accident.

5. Bound what the inner store may contribute

A nested store contributes everything it can see. When that is a shared config carrying settings this tool has no business reading, wrap it:

		config.WithBackend(config.Filtered(
			config.Nested(global, "global", config.NestedPromotable),
			config.Deny("telemetry.*"),
		)),

editor = vim editor = vim (from global.yaml)
theme = light theme = light (from project.yaml); also defined in global.yaml
telemetry.enabled = <nil> telemetry.enabled is not set
build.target = wasm build.target = wasm (from project.yaml)

telemetry.enabled is not merely hidden from reads. It is not set, as far as this store is concerned, so Has is false and a write would not be accepted for it either. Hiding a value from reads while still letting something write to it is the worst of both, so the filter applies to the whole surface.

Allow is the other half. With no Allow, every key is permitted unless a Deny excludes it; with one, only matching keys get through. Deny beats Allow where they overlap, so an Allow can never re-expose something explicitly denied.

Filtered wraps any backend, not just a nested store. The same call bounds a Consul prefix that a broad token happens to be able to read.

6. Push a runtime override above everything

Some values get decided while the program is running: a --set flag, something fetched at startup, a test fixture. AddLayer puts one above every layer declared at construction.

	if err := project.AddLayer(ctx, "cli-override", strings.NewReader("theme: high-contrast\n")); err != nil {
		log.Fatal(err)
	}

theme = high-contrast theme = high-contrast (from override:cli-override); also defined in global.yaml, project.yaml

An override layer is read-only, because there is nowhere to persist it, and it survives reloads, because it is re-read each time rather than merged in once. If its content will not parse it is refused and withdrawn, leaving the last known good configuration live rather than half-applying something broken.

One restriction worth knowing before you hit it: calling AddLayer from inside an observer returns ErrWriteFromObserver. Observers see a snapshot, and letting one mutate the store mid-notification is how notification ordering stops being defined.

What you built

override:cli-override ← AddLayer, runtime, read-only, highest
project.yaml ← the repo's config, writable, routing's target
global.yaml ← via Nested + Filtered, promotable by name only

Call	What it does
`config.Nested(s, id)`	inner store’s layers become layers of the outer one, read-only
`config.NestedPromotable`	a named write may reach in; routing still may not
`config.Filtered(b, ...)`	bounds which keys a backend contributes, and accepts writes for
`config.To("name")`	pins one change to a named layer
`Store.AddLayer`	a read-only layer above everything, added at runtime

Where to go next

The docs carry the task-shaped versions of all of this: composing stores including cycles and reload semantics, filtering a backend with the full pattern syntax, and writing configuration with routing and targets in full. The Store is the why underneath: one component owning all configuration I/O is what makes composing them safe.

And if you want the argument for config being its own module rather than something the framework hands you, that one’s separate.

As for keryx, it now does just this. The eight styles come from my library, the project pins whatever it needs to build unattended, and when I can’t remember which one a cover actually used, I can ask.

The bill for fifty-one modules

Thu, 08 Oct 2026 00:00:00 +0000

Change one line in the tls module and you have signed yourself up for twelve releases (releases, not commits), in four sequential waves, each one waiting on the one before it, and every single one gated on a human pressing a button. I know, because I was the human.

What the review actually called it

I had an architectural review run across the whole toolkit last July, and it came back with sixty-odd findings, the usual mix of things I already knew and things I’d rather not have. The one that mattered wasn’t a bug at all. It was theme five, and it got a title I’ve not been able to un-see:

Propagation debt as the ecosystem’s principal systemic risk. The many-small-modules architecture is clean but expensive: a tls change fans out into 12 human-gated releases across 4 sequential waves; a config change into ~20–23.

Principal systemic risk. Not the credential-timeout bug or the redirect-token hardening… the way I’d built it! At that point there were fifty-one modules.

Why it costs what it costs

The maths isn’t complicated, it’s just relentless. tls sits near the bottom, above it are the transports, above those the framework, and above the framework the tools. Change tls and nothing above it moves until tls cuts a release, then its direct consumers bump and they cut releases, then their consumers bump. Four layers deep, twelve artefacts, and each hop is a merge request I have to look at.

I’m not sure any of it counts as waste, exactly. Each of those releases is a real version of a real module that did change, and it’s the correct behaviour of a correctly built dependency graph.

It’s also most of a night, for one line.

The finding underneath the finding

What I keep coming back to isn’t the twelve so much as this:

The evidence shows propagation stalls where fan-out is largest.

It isn’t the obvious thing. You’d assume the big fan-outs get done first, on the grounds that they’re the important ones, and the opposite happens: the change with twelve downstream releases is the one you look at late on a Friday and decide is a problem for next week, and then the week after, while the modules furthest down the chain sit three minors stale and the fix waits, merged and unreleased, at the bottom.

So the debt accrues fastest where it hurts most, simply because that’s where the activation energy is highest.

What was meant to happen

The review proposed two moves worth more than the rest put together. Re-pin the whole config family in one sweep, which clears two of the ecosystem findings at a stroke. And fleet-manage the CI component pin rather than nudging repos one at a time.

It also floated something I’ve been chewing on since: whether the config adapters ought to live in one multi-module repo with synchronised tagging. That would be a partial retreat from the thing I split up to begin with, and it might still be the right call.

Where it stands now

I checked, while writing this. There are sixty-seven modules, not fifty-one… sixteen more mouths to feed than the review was even looking at. And the config family, which the review singled out as the worst fan-out in the estate, is clean: twenty-six adapters, all pinned to config v0.15.0, which is the current core, with nothing stale and nothing sat three minors back hoping I don’t look. That’s the number I expected to be bad, and it’s the one that came back spotless (it didn’t get that way on its own).

Getting the invoice down

The cost was never the extraction, because you budget for the extraction. The cost is a recurring tax on every change afterwards, and the awkward part is that it scales with how well you did the splitting. Smaller units of work, cleaner layering, a graph that actually reflects how the thing is built. And the price of all that is twelve releases for one line. Do it badly and you get a mess; do it properly and you get a bill.

So the question was never whether the architecture was worth it, it was how cheap I could make the tax. Three things, none of them finished. Renovate does the bumping, fleet-wide, from a single group-level bot rather than a config per repo. That turns “I remember to update twenty-six adapters” into something that happens whether I remember or not. It’s the reason all twenty-six are sat on the same core without me having chased a single one of them.

The pipelines do the gating. A release is a merge request like any other, so the same guards that stop a bad commit stop a bad release, and nothing needs a human except the button.

And a release-train skill, currently sat in review, which is the piece I’ve been missing. Releasing one repo is solved and has been for ages. Releasing twenty-six that depend on each other is a different problem, and it has a right answer that isn’t the obvious one. Cutting the leaves first looks like it dodges a Renovate cycle:

Order	Releases
Leaves first: 26 adapters, then core, then Renovate bumps 26, then release again	53
Upstream first: core, then Renovate propagates, then adapters absorb it and release once	27

Cutting the leaves first doesn’t avoid the cycle. It defers it until after the releases, so every leaf gets cut a second time the moment the core moves. Upstream-first costs you a wait while Renovate propagates, and nothing else. I had that backwards while driving the config family’s release train (for about an hour, which was long enough to feel it) and corrected myself before I doubled my own workload. That’s why it’s going into a skill rather than into my head: the mechanics are easy and the reasoning is what fails.

What I’m not paying for

There is, of course, a shop-bought answer to a chunk of this. GitLab has merge trains and they’d help, but they also live behind a subscription tier I’ve opened, looked at, and closed the tab on more than once. So I’m building my own out of Renovate, CI components and a skill that knows which way round to cut a release… on the grounds that I’d rather spend the evenings than the money. Ask me again when I’ve made my millions.

Sixty-seven, and counting

Fifty-one when the review was written, sixty-seven as I type this. That number going up is the estate working: each of those is a thing you can use without taking the rest of it, which was the point of the exercise. There’ll be another one along next week, there always is, and the trick was never to stop adding them, it’s to get to the point where the sixty-eighth doesn’t cost me a Saturday.