Security on PHP Boy Scout

The secret that wasn't on my branch

Sat, 04 Jul 2026 00:00:00 +0000

My CI went red on a secret I’d never committed.

Not a close call, not a near-miss I’d half-forgotten about. gitleaks, the secret scanner, failed a merge request of mine on a private key that was not on my branch, was not in my change, and as far as I could tell had nothing to do with me at all. The job was adamant. I was baffled. Somewhere in between was a lesson about what a secret scanner actually scans.

Prove it isn’t yours

First rule of being accused: don’t get defensive, get evidence. The scanner pointed at a couple of commits carrying a test private key and a PEM block in a spec document. I genuinely didn’t recognise them, but “I don’t recognise it” is a feeling, not a fact, and feelings don’t reopen a pipeline.

Git will tell you the truth if you ask it precisely. The question is: are these flagged commits actually part of my branch?

git merge-base --is-ancestor <flagged-sha> HEAD

That asks “is this commit an ancestor of where I am?”. The answer came back no. The commits the scanner was choking on were not in my history. They weren’t mine.

So whose were they? A bit of digging turned them up on a completely separate, still-unmerged branch, where someone (me, a few days earlier, on a different feature) had committed a throwaway test key and a PEM example in a spec, on purpose, as fixtures. Deliberate, harmless, and nowhere near the branch under review. And yet here they were, failing a merge request that had never touched them.

What gitleaks actually scans

Here’s the bit I’d taken for granted. I assumed gitleaks detect scanned my change. It doesn’t. With no further instruction it scans the whole history reachable in the checkout it’s handed.

And the checkout it’s handed is where the second half of the surprise lives. GitLab runners default to GIT_STRATEGY: fetch, which reuses the runner’s working directory between jobs rather than cloning fresh every time. It’s faster, and most of the time you never notice. But it means a shared runner accumulates refs from every branch it has ever built. My MR’s job happened to run on a runner that had, at some point, built that other branch, so the fixtures were sitting right there in the local object store, fair game for a scanner walking the whole graph.

So gitleaks did exactly what I’d asked it to do, which was “scan everything”, and “everything” turned out to be a great deal more than my change. It walked the lot and dutifully reported fixtures from a branch I wasn’t even proposing to merge. The scanner wasn’t wrong. My idea of what it was looking at was.

Unblock now, fix properly after

Two problems on two timescales. I needed the MR to merge today, and I needed this to never happen again. Those want different fixes.

The immediate one: tell gitleaks those specific fixtures are known and intended. They’re test material, they’re meant to be there, so they go in the allowlist:

paths = [
 # ...
 '''internal/cmd/keys/keys_test\.go''',
 '''docs/development/specs/2026-06-08-keys-mint-command\.md''',
]

That unblocks the merge. It does nothing about the root cause, which is that the scan was looking at the wrong commits in the first place. Allowlisting individual false positives as they crop up is closing the stable door after the horse has bolted, one horse at a time, forever.

The real fix lives in the shared CI component, not in any one repo. Scope the scan to the commits the merge request actually introduces (cicd v0.10.3):

if [ -n "$CI_MERGE_REQUEST_DIFF_BASE_SHA" ]; then
 gitleaks detect --source . --verbose --redact \
 --log-opts="$CI_MERGE_REQUEST_DIFF_BASE_SHA..$CI_COMMIT_SHA"
else
 gitleaks detect --source . --verbose --redact
fi

--log-opts is handed straight through to git log, so that range, base-of-the-MR to tip, is the exact set of commits the merge request adds and nothing else. On a merge request the scan now sees only what you’re proposing to merge. Off a merge request (a plain branch or a tag pipeline) it falls back to the full scan, because there you genuinely do want the lot. The before-and-after in the job log tells the whole story: the entire accumulated history on one side, the handful of commits you actually wrote on the other.

Fixing the fix

There’s a tax on touching CI shell, and I paid it. The change went into the go, rust and tofu security templates. Go and tofu went green. Rust failed.

The rust template had built its optional --config flag the clever way, inline:

... $([ -n "$CONFIG" ] && echo "--config $CONFIG")

As a command argument, that substitution is harmless: its exit status is thrown away, so nobody cares that the test inside returns false when there’s no config. But when I rewrote the block and reached for the same pattern as an assignment, VAR=$(... && ...), it became a different animal. An assignment takes the exit status of the command substitution, and under set -e a non-zero status anywhere aborts the job. So on every run where the config was empty, which was most of them, the test returned false, the assignment inherited that false, and set -e killed the job stone dead. Same $(...), two completely different fates, decided entirely by whether it sat to the right of an = or got handed to a command as an argument. Go and tofu never used the assignment form, so only rust fell down the hole.

The fix was to stop being clever and write the boring if:

GITLEAKS_CONFIG=""
if [ -n "$CONFIG" ]; then
 GITLEAKS_CONFIG="--config $CONFIG"
fi

Boring shell is good shell.

What a scanner is actually looking at

The whole mess came from one unspoken assumption: that a tool called to scan “my change” was scanning my change. It was scanning a checkout, and a checkout on a shared runner is a much bigger, messier thing than the diff in front of you. None of the pieces were broken. gitleaks did its job, GIT_STRATEGY: fetch did its job, my fixtures were exactly where I’d left them. They just added up to a red pipeline that had nothing to do with the code I was trying to ship. I’d spent a good chunk of the day proving my innocence to a scanner that was only ever doing as it was told… and the one thing I’d actually got wrong was being sure I already knew what it was looking at.

Bought, not stolen

Tue, 23 Jun 2026 00:00:00 +0000

The malware that spent months wearing Microsoft’s trust didn’t steal a thing. No cracked certificate authority, no private key lifted off some breached vendor. There was, in effect, a shop: you uploaded your malware to a website, paid somewhere between five and nine thousand dollars, and got it back signed with a real, valid certificate. The same kind that vouches for the software you actually want on your machine.

The crew running that shop is one Microsoft’s Digital Crimes Unit tracks as Fox Tempest, and the certificates weren’t forged. They were minted through Microsoft’s own code-signing service, which Fox Tempest abused under a pile of fake identities and impersonated companies, then resold to anyone with the money. The malware they dressed up was the usual rogues’ gallery, the Oyster backdoor, the Lumma and Vidar infostealers, often got up as spoofed Teams or AnyDesk installers. In May, Microsoft pulled the operation apart: more than a thousand fraudulent certificates revoked, the infrastructure seized, a lawsuit filed against Fox Tempest and the ransomware crew behind Rhysida that had been paying for the service. For the better part of a year, a valid Microsoft signature was a product you could check out of a basket.

The reflex is to file this under “another supply-chain breach”. It isn’t one. Nothing was breached. The system did exactly what it was built to do, for a paying customer who’d lied about who they were. That’s the part worth sitting with.

We have been here before

None of this is new. I went back and read a Trend Micro write-up on code-signing abuse from 2018, and it could have gone out last week. Stuxnet carried valid signatures from stolen Realtek and JMicron certificates. After Sony Pictures was ransacked in 2014, the attackers signed their Destover malware with Sony’s own keys. The names rotate and the methods rotate, the story doesn’t: there is a trust mark, there is money in wearing it, so somebody works out how to wear it.

I’ve started thinking of them as honest villains. Not honest with their victims, obviously. Honest in their incentives: a rational operator who finds a loophole and works it for everything it’s worth, because there’s profit on the other side of it. You can be as furious at Fox Tempest as you like, and I am, prosecution is exactly right, but the honest villain is never the surprise. The honest villain is a constant. Build a gate worth getting through and one will turn up to test it… every single time! That isn’t cynicism. It might be the most dependable law we’ve got.

A seal was never a promise

Here’s the thing we keep forgetting about a signature, and it goes back a very long way.

For most of human history, trust was blood. You trusted the people you were bound to, by birth and by the pacts of kinship and marriage that held a tribe together. It worked, for tens of thousands of years, because the circle was small enough that everyone you needed to trust was someone you knew, or someone known to someone you knew. The bond and the accountability were the same thing. Betray the tribe and the tribe knew precisely whose door to come to.

Then we outgrew the tribe. We started trading with strangers, across distances and across lifetimes, with people we would never meet and could never vouch for by blood. And blood stopped being enough. So we built stand-ins for it. The wax seal pressed into a letter. The signature at the foot of a contract. The notary, the stamp, the certificate. Each one a small portable proxy for the bond of kinship we’d walked away from.

But look at what a seal actually did. A seal on a letter never told you the letter was true. It told you whose seal it was, which is to say it told you who to hold responsible if the letter turned out to be a lie. It was never a guarantee of honesty. It was a marker of accountability. It pointed at a person.

A code signature is the newest seal in that very old line, and it does the same single job. It does not tell you the software is safe. It was never built to. It tells you who signed it, which is to say who to come to when it isn’t. That’s the whole of it. The padlock in the browser, the green tick on the installer, the verified signature on a binary, none of them ever meant “this is good”. They meant “here is a name attached to this”.

The part you cannot sell

Which brings me to the uncomfortable bit, the one that points back at me as much as at Redmond.

If a signature is a name accepting responsibility, then the power to sign and the blame for what you sign are the same object. You don’t get one without the other. You cannot hold out the seal and quietly keep back the accountability behind it, because the accountability is the only thing the seal was ever made of.

That is exactly what came apart here. Microsoft holds enormous power as a signing authority, the power to make code look trustworthy to millions of machines. With that power comes the plain duty to check who you’re handing it to. They took the first part and skimped on the second. They sold the seal and skipped the diligence the seal is supposed to stand for, and the honest villain simply walked through the gap between the two.

And I have to hold myself to that very same rule, or I’ve no business naming it. I sign my own releases. Every go-tool-base release carries an OpenPGP signature over its checksums, made by a key that never leaves AWS KMS, with the public key published off-platform so the release host can’t quietly swap it. I do it so the people who use my tools, and they’re a varied bunch, can trust that what they’re running genuinely came from me.

But trusting me was never really the question. Here’s the one that keeps me up: what happens the day I’m the weak point? If a contributor slips something rotten into my code, or one of my own AI agents writes something it shouldn’t, and my automation dutifully signs it, then the signature does its job perfectly and the whole house of cards comes down. My users can trust me all they like. The seal will still say “Matt”, and it will be telling the truth, and that is precisely the problem.

So the accountability can’t be delegated, and I don’t try to. Nothing reaches my releasable branches without my own eyes on it first. No merge request, no commit, not from a contributor and not from one of my agents. The vigilance is mine, singularly, and that’s deliberate, because the blame is mine too, singularly, and they’re the same coin. It’s an easier thing to say as a one-man outfit than it’ll be if that ever changes. But the principle doesn’t get cheaper at scale. It just gets harder to honour, which is a very different thing from optional.

What standing behind it looks like

If you want to see the duty done properly, look at how OpenAI handled the Axios incident earlier this year. A poisoned dependency had got at the material used to sign their macOS app. They had every reason to believe their certificates were fine. They revoked them anyway, and rebuilt, because the cost of being wrong was their users’ machines and their own name on the door. That’s what holding the power looks like. You act on the possibility of compromise, not the proof of it, because by the time you’ve got proof it’s already on somebody’s laptop.

It’s also why I’ve come to treat trust as layers rather than a single tick in a box. The release signature is one layer. Notarisation is another: every macOS binary go-tool-base ships is notarised by Apple as well, and has been for a good while. There will be more, and the Rust side of my tooling has its own signing coming very soon. None of them is the answer on its own. Each one is just another check, another name standing behind the thing. And the part I care about most, as someone who builds tools other people build on, is handing that same machinery to them, so they can stand behind their own releases for their own customers instead of trusting that someone, somewhere up the chain, did the diligence.

Vigilance, still

I’d love to tell you there’s a clean technical fix on the way, a trust system the honest villain can’t game… There isn’t. We’ve tried plenty, and some have aged better than others. (I’m looking at you, blockchain, the confident answer to a question almost nobody was asking!) Whatever we build next, the same loop runs: the mechanism gets more elaborate, so the attacks on it get more insidious, so the mechanism gets more elaborate again. We have spent the whole of human history moving trust further and further from the blood bond that used to ground it, and every step we take away from a person who will answer for it, the honest villain takes right alongside us.

Maybe there’s a version of the future where this stops… some distant place where nobody needs to cheat because nobody wants for anything, and greed has quietly retired. I’m not holding my breath. Until then the job is the plodding one it has always been: stay vigilant, act before you’re certain, and keep trust as close as you can to a person who’ll stand behind it. Because that is the only thing a seal ever was. Not a promise that the thing is good. A name, and someone willing to answer to it.

A signing key that never leaves KMS

Thu, 11 Jun 2026 00:00:00 +0000

The last post in this series walked through how a tool verifies a release signature the platform can’t forge. That post had a loose end dangling off the back of it, and I knew it the whole time I was writing. Because a signature has to be produced by a private key… and a private signing key is the single worst thing in this entire story to lose. Steal it, and you sign malware that sails through every check I spent two posts building, signature and all. So where does that key live? The answer I landed on is the one this whole post is about: inside AWS KMS, and it never comes out.

The only key you can’t steal

Think about where a signing key normally ends up. A file on a build server. A secret in CI. A key on the release engineer’s laptop, “just for the release, I’ll delete it after”. Every one of those is a copy, and every copy is one more thing somebody can read, exfiltrate, or quietly clone while your back is turned. You can wrap them in passphrases and vaults and rotation policies all you like, and you’re still standing guard over a thing that exists in a place you don’t fully control.

The way out is almost annoyingly simple to state: the only key nobody can steal is the one that was never anywhere to be stolen from. So don’t hold the key at all. Let something else hold it, somewhere it has no export path, and ask that thing to sign for you.

That thing is AWS KMS. This is the infrastructure side of the question I opened the signing series with, finally answered with real Terraform.

A key that’s born in the box and stays there

The signing key is an asymmetric KMS key, and the module that provisions it is small enough to read in one sitting:

resource "aws_kms_key" "this" {
 description = var.description
 key_usage = "SIGN_VERIFY"
 customer_master_key_spec = var.key_spec # default RSA_4096

 # Asymmetric SIGN_VERIFY keys do not support KMS-managed rotation;
 # rotation is handled by minting a new key (alias = `<name>-v2`) and
 # publishing the v2 public key alongside the v1 key (dual-sign window).
 enable_key_rotation = false

 policy = data.aws_iam_policy_document.key_policy.json
}

The private half of that key is generated inside KMS and there is no API that hands it back to you. You don’t sign with it the way you’d sign with a file. You call kms:Sign: the bytes you want signed go up, a signature comes back down, and the key itself never moves. An attacker who completely owns my CI, my account, my laptop, can ask KMS to sign things for as long as their access lasts… but they can’t walk off with the key and keep signing forever. The blast radius is “while I’m compromised”, not “until I rotate a key I didn’t know had leaked three years ago”.

Why RSA-4096 and not the Ed25519 I’d normally reach for? Because KMS asymmetric signing doesn’t offer Ed25519, and OpenPGP’s packet format is tied to the algorithm that signed it, so the choice of key spec ripples all the way out to the signature on the wire. RSA-4096 is the strong option KMS does offer, so RSA-4096 is what the workflow is built around. A constraint of the box shaped the cryptography, not the other way round, and I’d rather say so than pretend I picked RSA on purpose.

Minting an OpenPGP key from a key you can’t hold

Here’s the part I find genuinely neat. OpenPGP wants a private key to self-sign its own public key when you generate it. And I don’t have a private key in any form I can hand to a library… it’s sitting in KMS, behind a door with no handle on my side. So how do you produce a valid OpenPGP public key at all?

go-tool-base leans on a small Go interface, crypto.Signer: anything that can return its public key and sign a digest. A KMS-backed signer satisfies it by turning each Sign call into a kms:Sign request. Then pkg/openpgpkey builds the OpenPGP entity around that signer:

func Entity(signer crypto.Signer, name, email string, creationTime time.Time) (*openpgp.Entity, error) {
	rsaPub, ok := signer.Public().(*rsa.PublicKey)
	// ...
	pubPkt := packet.NewRSAPublicKey(creationTime, rsaPub)
	// Construct the private-key packet directly (rather than
	// packet.NewSignerPrivateKey, which panics on opaque signers):
	// the crypto.Signer drives the actual signing, so a KMS-backed
	// signer works here.
	privPkt := &packet.PrivateKey{PublicKey: *pubPkt, PrivateKey: signer}
	// ...
}

Look at that PrivateKey packet. The field where OpenPGP expects the secret key material holds the crypto.Signer instead, which is to say, a remote handle to KMS. When the entity self-signs its public key, that self-signature is computed by KMS. gtb keys mint runs exactly this and writes out an ASCII-armored OpenPGP public key, and at no point did a single byte of private key material exist on the machine that minted it. The OpenPGP “private key” is a phone line to a vault, not a key.

That public key is what gets published off-platform over WKD and baked into the binary, the two trust anchors that post cross-checks.

Access without a human and without a standing key

A key that never leaves KMS is only as good as the rules about who may call kms:Sign. The signer role is deliberately narrow: it can call kms:Sign and kms:GetPublicKey on this one key and nothing else, and it is assumable only over OIDC from specific CI subjects, the same keyless federation the rest of the estate runs on. No human holds it. No long-lived access key sits in a CI variable waiting to leak. A release job federates in for its few minutes, signs, and the credentials evaporate with the runner.

So the chain of “who can sign a release” has no standing secret in it anywhere. Not a key file, not an access key, not a console user. Just a short-lived token, scoped to two API calls, on a key that can’t be exported.

The real cost: rotation is manual

This isn’t free, and the bit it taxes you on is rotation. KMS won’t auto-rotate an asymmetric SIGN_VERIFY key, which is why the module sets enable_key_rotation = false rather than leaving a default on. Rotating means minting a new key (a -v2 alias), publishing its public key alongside the old one, and running a dual-sign window long enough that clients have picked up the new anchor before you retire the old. It’s manual, it’s a runbook, and pretending otherwise would be the kind of thing this series exists to argue against. The trade I made was: a key with no exfiltration path, in exchange for rotation I have to do by hand. For a release-signing key, that’s the right side of the trade.

Why this is a command and not a script I hid

The origin of all this is a good deal less tidy than the result. I was working through the key-generation runbook, creating the offline rotation key with a gpg command I’d copied straight off my own page… and it just hung. No error, no prompt, just a cursor blinking while gpg waited on something it never bothered to mention.

My first instinct was the lazy one: drop the minting script into a scripts folder in my infra repo and never speak of it again. Then it nagged. That repo’s private, so the recipe would live somewhere nobody else could ever reach, and I’d already half-promised myself a tutorial walking people through this exact setup. So it shouldn’t sit in infra at all. It should be a gtb command, with a pluggable backend so anyone can swap my KMS for whatever provider they happen to run.

The deeper objection is the one that actually shaped it, though. I didn’t want to be shelling out to gpg by hand in the first place. gtb is a tool I hand to other people, and every time it drops to the shell for some gpg incantation, that’s an environment I’m asking the next person to reproduce, a dependency to install, a fiddly step to get subtly wrong, all before they can sign a single thing. The aim was to keep as much of this inside the box as I could: mint the key, build the WKD tree, produce the signature, all in pure Go, with no gpg on the path and no gpg-wks-client either.

So gtb keys mint pulls the public half out of your KMS key and frames it as OpenPGP, the trick from earlier; gtb keys wkd builds the tree ready to upload; and gtb sign produces the detached signature through that same remote round-trip. What comes out is an entirely ordinary OpenPGP signature gpg --verify is happy with, so you’re not locked into anything of mine. And none of it is just for me: build your tool on go-tool-base and the same handful of commands stands you up with this exact model, pointed at your own KMS. No cloud KMS to hand? There’s a local backend, a plain key on disk, to wire the whole thing together on your laptop first. These are commands for you, the person shipping the tool. Your users never run mytool keys mint… they just get updates that quietly check themselves, which was the whole idea two posts ago.

That setup deserves a walkthrough of its own, and it’ll get one. For now, the ergonomics were the point, not a nicety bolted on afterwards. The safest setup in the world is no use to anyone if it takes a PhD to stand up.

Where this leaves the whole story

Step back and the full loop is finally closed. The private key is born in KMS and never leaves it. Its public key is minted from it, with KMS computing its own self-signature. That public key is published off-platform and embedded in the binary. Releases are signed by KMS, reached only through short-lived OIDC federation. And the client verifies against the embedded and WKD keys cross-checked against each other. At no single point in that chain is there a thing an attacker can grab that lets them forge a release, and the most dangerous thing of all, the private key, has no theft path because it has no export path.

That’s the thread running through the whole signing series, from the very first checksum to here: the strongest control isn’t a better lock on the key. It’s arranging things so the key was never somewhere you could lose it. Nobody is coming to clean your supply chain, so the least you can do is leave it nothing worth stealing.

A signature the platform can't forge

Tue, 09 Jun 2026 00:00:00 +0000

A self-updating tool has a chicken-and-egg problem baked into it. The thing doing the updating is the thing being updated, so when it reaches out and pulls down a newer version of itself, it’s the one that has to decide whether to trust what just landed. No human in the loop, nobody to ask. I’ve been closing that gap in go-tool-base’s self-updater in two phases. The first gave it a checksum: download the new binary, hash it, compare it against the release’s checksums.txt. That catches the accidents, the truncated download, the flipped bit on a dodgy mirror. And I said at the time, plainly, that it does nothing about a determined attacker who owns the release platform… the checksums file sits right next to the binary, so whoever can swap one can swap both. I left that as an IOU. This second phase is me paying it.

The thing a checksum can’t do

A checksum is a promise that the bytes you got match the manifest. It says nothing about who wrote the manifest. So if GitLab, or my account, or a leaked CI token gets compromised, the attacker rewrites the binary and the checksums.txt in the same breath, and the hash matches perfectly, because they’re the one who computed it. It’s the same wall I keep walking into whenever I think about supply-chain trust: a checksum is only ever as good as whatever’s standing behind it, and the thing standing behind a checksum is the very platform that just handed you the file. Same hands, both times.

To get past that, you need a signature whose root of trust lives somewhere the platform can’t reach.

The crypto is the easy part

Here’s the bit that caught me slightly off guard while I was building this: the cryptography is the easy part. Verifying a detached OpenPGP signature is a library call, and go-tool-base’s TrustSet wraps it up in one method:

func (t *TrustSet) VerifyManifestSignature(manifest, signature []byte) error {
	// ...
	signer, err := openpgp.CheckArmoredDetachedSignature(
		t.entities, bytes.NewReader(manifest), bytes.NewReader(signature), nil)
	if err != nil {
		return errors.Wrap(ErrSignatureInvalid, err.Error())
	}
	if signer == nil {
		return errors.Wrap(ErrSignatureInvalid, "no signer in trust set matched")
	}
	return nil
}

Hand it the manifest, the detached signature, and a set of trusted public keys (the entities), and it tells you whether any one of them signed it. That’s the whole of the cryptography, and it’s genuinely not where the hard work lives.

The hard work is that set of trusted public keys. Where do they come from? Because if the answer is “we ship them right next to the binary”, well… you’re straight back to the checksum problem. Whoever can swap the binary can swap the key too, sign with their own, and the check waves it through none the wiser.

Pulling the two questions apart

So the design splits along exactly that seam. The verification half is fixed, and deliberately boring (the method above). The trust anchor, the actual keys, comes from a swappable KeyResolver:

// The interface separates "where the trust anchor comes from" from "how a
// signature is verified against it", so SelfUpdater can be wired with
// whichever resolver chain a tool needs without changing verification logic.
type KeyResolver interface {
	Name() string
	Resolve(ctx context.Context) (*TrustSet, error)
}

That little seam is really the whole game. Everything interesting about standing up to a compromised platform comes down to which resolver you hand the updater, and the verification code never has to know the difference.

Three answers to “where does the key live”

The first option is to embed it. Bake the public key straight into the binary at build time (NewEmbeddedResolver), so it rides along inside a release you already trusted enough to run. Tidy and self-contained. The catch is that a future malicious release could embed a different key, so on its own, embedding really just trusts whoever cut the most recent binary.

The second is WKD, the Web Key Directory. Fetch the key over HTTPS from a well-known path on a domain you control (NewWKDResolver), nothing to do with where the release itself is hosted. Now the key isn’t in the binary at all, so poisoning a release doesn’t touch it. You haven’t made the problem disappear, mind… you’ve moved the trust onto your domain’s host and its DNS. A different blast radius, but a blast radius all the same.

The third option is to do both, and make them agree. Run embedded and WKD, and insist they agree:

func (c *CompositeResolver) Resolve(ctx context.Context) (*TrustSet, error) {
	// ... run each child resolver concurrently ...
	if err := checkAgreement(successes); err != nil {
		return nil, err // ErrKeyResolverMismatch
	}
	return successes[0].ts, nil
}

Think of it as the two-key rule on a safe deposit box, or two witnesses who’ve never met telling you the same story. One source on its own you might quietly doubt. But if the key baked into the binary and the key sitting on my domain hand back the same fingerprint, that agreement is worth a great deal more than either of them alone. And if they ever come back different, that’s not a maybe, that’s an alarm: ErrKeyResolverMismatch. Poison one source and the mismatch is the thing that gives the game away.

That composite is the real answer, and it’s why the interface exists at all. There’s nothing a single attacker can get their hands on that holds the whole thing up by itself. The key is baked into a release you trusted, and fetched from a domain well off the release platform, and the two have to match before a single byte of the update is allowed through.

The separation is the whole point

It’s easy to nod along at “two sources” and miss the part that actually does the work. The agreement between the embedded key and the WKD key is only worth something if an attacker can’t reach both of them from the same place. If the key I bake into the binary and the key I serve over WKD both came out of the same release pipeline, whoever owns that pipeline swaps the pair of them, the fingerprints still match, and the cross-check happily waves the forgery through. Same hands, both times. Again.

So they don’t share a pipeline, and that’s the entire design, not an accident of how things ended up. The binary, and the key embedded in it, are built and signed in GitLab CI, which federates into AWS KMS to do the signing itself. The WKD key lives somewhere else completely: a Cloudflare Pages site serving openpgpkey.phpboyscout.uk, deployed by hand at rotation time with the Wrangler CLI and a token allowed to do nothing but edit that one Pages project. No Git integration, no webhook, nothing that lets a push to the repo or a run of the release pipeline so much as touch it. The Cloudflare account is even administered under a different email and a different second factor from the GitLab and AWS ones, so the three anchors really are independent rather than just feeling that way.

Which is what makes them fail independently, and that independence is the only thing that makes the agreement worth checking. To forge a release that survives the cross-check, an attacker doesn’t have to beat one system, they have to beat two unrelated ones, on different platforms, behind different credentials, in the same window, without either of them noticing.

There’s a quieter benefit in the cadence, too. Releases go out constantly and automatically; the WKD key changes rarely, and only ever by hand. So the busy, automated path, the one an attacker is most likely to prise open, is exactly the one with no power to rewrite the key everyone checks against.

Requiring it, without breaking everyone

Now, a check nobody ever switches on is just theatre. But switch it on before the keys are actually out there in people’s installs, and you’ve handed everyone a self-inflicted outage instead. So the default is deliberately timid. The framework ships DefaultRequireSignature = false: a tool built on go-tool-base doesn’t suddenly start rejecting its own updates the day its author bumps the framework version.

The tool author flips it to true in main(), but only after they’ve shipped a release that embeds the key, so every install out there already holds the trust anchor before the first release that insists on one. Ship the key, then turn the lock: the same leave-yourself-a-way-back discipline as any migration you’d like to still have a job after. And the end user still gets an override (update.require_signature, or an env var) for the day it all goes sideways and they need out.

What it actually buys

The first phase stopped accidents. This one stops the platform. And not because the cryptography is clever, OpenPGP checks the signature in a single call, but because the trust anchor is arranged so that nothing the attacker can actually reach holds the whole thing up on its own. A signature only ever proves the sender, never the contents. All of this is really about making “the sender” something a compromised release host can’t quietly fake its way into being.

Which leaves one last thread dangling. The verifying key gets fetched from somewhere, fine… but the signing key, the private half that actually produces these signatures, has to live somewhere the platform can’t reach either, or none of the rest holds up. That’s the capstone, and where this series ends: where that key lives, and why it never leaves the box it’s born in.

Anything under an 8

Mon, 08 Jun 2026 00:00:00 +0000

I read the news about the National Vulnerability Database over a coffee that went cold while I sat there muttering at my phone. The short version: the NVD, the free public catalogue that quietly props up half the security tooling you and I run every day, is going under in slow motion. And the more I dug into why, the worse the taste in my mouth got.

I’m an open-source person. I think of myself as part of that community, and the NVD is one of those public goods the whole community leans on without ever really thinking about it. So my first reaction wasn’t clever or measured. It was a kick in the teeth.

The carcass and the vultures

Here’s where things actually are. In February 2024 the NVD had around 13,000 unprocessed vulnerabilities sitting in a queue waiting to be analysed. By the end of 2025 that backlog had passed 27,000. This April, NIST effectively admitted it can’t dig out: everything published before 1 March 2026 that hadn’t been enriched got swept into a bucket marked “Not Scheduled”, and going forward only the highest-risk entries get the full treatment. The rest you’re on your own with.

The reasons are grimly ordinary. The Cybersecurity and Infrastructure Security Agency stopped funding the programme in 2024. The enrichment contract lapsed that same February, and despite NIST having two years’ notice it needed a replacement, the database limped along understaffed until late November. And the volume kept climbing regardless: 48,185 CVEs in 2025, roughly 131 a day, with forecasts of the annual figure topping 60,000, getting on for ten times what it was a decade ago. No money, a fumbled handover, and a firehose. That’s the whole story.

The bit that turns my stomach is what comes next. When a free public good fails, the gap doesn’t stay empty. It gets filled, and it gets filled by people selling something. There are already commercial vulnerability databases that are better resourced and more current than the NVD, and the moment the free one is visibly on the floor, every one of them sees a market. Plenty of those subscriptions cost more in a year than a small open-source project will see in donations in its lifetime. So the catalogue the little projects relied on most is exactly the one about to be priced out of their reach. Vultures circling a carcass, and the carcass is something we all built on.

The number we never checked

And then I read the part that stopped me blaming everyone else.

A Department of Commerce Inspector General audit went through the NVD’s work and found that NIST’s own severity scores matched independent assessors only 12% of the time. Read that again. Not that NIST was wrong 88% of the time, that’s not quite what it says, but that two competent parties looking at the same vulnerability landed on the same severity barely one time in eight. The score was never an objective fact handed down from on high. It was always an estimate, a judgement call, the kind of thing reasonable people disagree about most of the time.

Which is awkward, because I have spent years treating that number as gospel. And I know I’m not alone, because I’ve watched whole engineering organisations do the same thing in writing. More than one large employer I’ve had bakes the CVSS score straight into policy: anything scored 8 or above blocks the build and gets a meeting, and anything under an 8 goes through at an engineer’s discretion. When time is money, and it always is in those places, “it’s only a 6.4, ship it” is the easiest decision you’ll make all week. I’ve made it. I’ve made it without opening the advisory, without checking whether the vulnerable code path was even reachable in what we’d built, on the strength of a single number that, it turns out, two experts wouldn’t have agreed on anyway.

So before I get cross about the funding, I have to sit with my own part in this. We took a contestable estimate and bolted it to the door as a gatekeeper. We turned “a rough signal worth a closer look” into “the closer look”, and then we stopped looking. The database didn’t promise us a safety net. We just decided it was one and stopped checking underneath.

Don’t blame the robots for this one

There’s an easy villain on offer here, and I want to wave you off it. It would be tidy to say AI did this, that the flood drowning the NVD is a tide of machine-generated slop, the same dynamic I wrote about when curl’s bug bounty buckled under unverifiable reports. It’s tempting, it’s topical, and it’s mostly wrong.

The people who actually crunch the numbers are clear that the surge is largely legitimate growth. There are now more than 484 CVE Numbering Authorities, far more organisations reporting far more bugs far more thoroughly than they did a decade ago. That isn’t a quality collapse, it’s the system working as designed and simply getting bigger than its funding. Pinning it on AI would be scapegoating, and scapegoating the robots for an underfunding-and-mismanagement problem is just a way of letting the people who defunded it off the hook.

None of which means AI gets a free pass. It just isn’t the arsonist. The same machine-assisted discovery tools that found genuine bugs are also forecast to push CVE volumes higher still, and yes, one of the tools named in that forecast is the very one I poked fun at over curl. AI is an accelerant on a fire that was already burning for thoroughly human reasons. It’s a beat in this story, not the spine.

The version I’m betting on

Where does this leave the working engineer? In a harder spot than before, because the easy answer stopped being easy. My usual line, the one I keep ending these pieces on, is that the diligence is the job: pin, lock, audit, and read the actual advisory instead of trusting a number. All of that still holds. But it just got more expensive, because the data underneath the diligence is thinner and, as it turns out, was shakier than we let ourselves believe.

So I’m not going to pretend there’s a clean fix. This problem won’t solve itself, and it won’t be solved by any one of us. It needs all of us to actually support the services we depend on, with money, with contributions, with attention, so the public goods that underpin our craft are still standing in ten years. That’s the dull, grown-up part.

But I’ll end this one looking up rather than down, because for once I can. I think the next few years bend towards safer software almost in spite of us. Modern languages are quietly closing off whole categories of vulnerability at the source: every memory-safety bug that a borrow checker refuses to compile is one that never reaches a database to be mis-scored in the first place, which is rather the point of building a framework that contains no unsafe. Used with proper guidance instead of left to spew slop, AI can be a genuine help finding and triaging the things that do slip through. And the junior engineers we keep sawing off the bottom rung are exactly the people who, mentored by the greybeards before they retire, could build the next generation of vulnerability identification that the current model clearly can’t sustain.

As for the vultures… it’s a coin toss. A lot of firms will look at the NVD on its back and see a land grab. I’d love to be proved an optimist and watch at least one of them stand tall, take all that better-resourced data and open it to open-source projects for nothing, because it’s the right thing to do and because the whole industry drinks from that well. One of them doing the decent thing would be worth more than all the press releases about responsible AI put together.

The catalogue is wobbling. The number was never as solid as we treated it. Neither of those is the end of the world, as long as we stop outsourcing our judgement to a free service we never funded and never checked, and start paying, in every sense, for the foundations we build on. Boring, unfashionable, and the only thing that ever works. I think we’re up to it.

Nobody's coming to clean your supply chain

Fri, 29 May 2026 00:00:00 +0000

Pick a week in May 2026 and there’s a supply-chain attack in it. On the 11th someone owned TanStack’s CI and pushed 84 poisoned package versions in six minutes. On the 14th, three malicious versions of node-ipc, a library with ten million weekly downloads, shipped an identical credential-stealer. Days later it was @antv, cascading down into a charting library a million projects depend on. Each one runs its payload the moment you install it, then quietly tries to publish itself from your machine.

You’ve heard this story so many times the outrage has worn smooth. So let me point at the one detail that should still make you sit up: the TanStack packages carried valid signing provenance. Real attestation, pointing at the real pipeline. The seal was genuine. The contents were poison.

A signature proves the sender, not the contents

I’ve spent a fair while building integrity and signing into my own tools, so this one stings a little. Signing is a trust mechanism, and a good one. It’s how I prove a binary you downloaded was built and published by me and nobody else, and in a world with this many ways to be impersonated, that matters more than ever.

But TanStack shows the limit in neon. If the pipeline doing the signing is itself compromised, the signature is still perfectly valid. It just now certifies a lie. Provenance answers “did this come from where it claims?” It does not answer “is what’s inside safe?”, and we have spent a few years quietly letting people treat those as the same question.

They aren’t. A signature is a promise about the sender. The thing we actually need is a promise about the contents: that whoever signed has done the diligence, the testing, the vetting, to vouch for what’s in the tin. A signature without that behind it isn’t a safety certificate. It’s a tamper-proof seal on a poisoned jar.

It was never just npm

It’s tempting to file all this under “npm being npm”. Resist it, because it’s a category error. The thing that makes these attacks work, a stranger’s code running on your machine as a side effect of installing or building, is not an npm bug. It’s a near-universal design choice.

Ecosystem	Untrusted code on install/build?	Mechanism
npm (JS)	Yes, at install (dependencies too)	`pre`/`postinstall` scripts
PyPI (Python)	sdist yes, wheel no	`setup.py`; wheels forbid hooks
RubyGems	Yes, at install	native-extension build (`extconf.rb`)
cargo (Rust)	Yes, at build	`build.rs` and proc-macros
Composer (PHP)	Dependencies: no	only the root project’s scripts run, by design
Maven/Gradle (JVM)	Yes, at build	build scripts and plugins
NuGet (.NET)	Modern: no	`install.ps1`, legacy format only
Go (modules)	No	no install or build hooks

(Lifecycle hooks across ecosystems are catalogued at ecosyste.ms if you want the receipts.)

Read that and the lesson isn’t “npm is uniquely bad”, it’s “this was a choice, and several ecosystems chose differently”. Go runs no install or build hooks at all. PHP’s Composer flatly refuses to run a dependency’s scripts, only your own project’s. Python’s wheel format forbids install hooks. The hook was never inevitable.

And yes, that includes my own back yard. cargo’s build.rs is the same gun fired at build time instead of install time, and the TrapDoor campaign used exactly that to rifle through keystores on crates.io this year. Rust isn’t safe here. It’s a smaller, better-policed target, which is a different thing, and I’d rather say so than pretend one of my favourite languages is above it.

No registry can hand you a clean package

Here’s the uncomfortable core. Not one of these registries can guarantee the package you pull is clean. They can sign it, scan it, attest its origin and mandate 2FA on maintainers, and they should do all of that. But none of it is a guarantee, because the failure modes are endless and attackers keep finding new ones. A maintainer account gets phished. A CI token leaks. A trusted contributor turns. A dependency four levels down quietly changes hands.

So the onus lands, and will keep landing for a good while yet, on the consuming engineer. That isn’t a comfortable answer or a clever one. It’s the true one.

And it’s a genuinely rotten spot to stand in, because the advice contradicts itself. Patch slowly and you’re scolded for running known-vulnerable dependencies. Patch the instant a release drops and you’ve skipped the bedding-in that might have caught a poisoned one. There’s no setting on that dial that’s safe, only trade-offs you have to actually think about. Add CI that leaks credentials it never needed, and a dependency tree thousands of strangers deep, and you can see why there’s no single villain to blame and no single switch to flip.

The boring discipline that actually helps

What’s left isn’t heroic, it’s hygiene, and it’s the boring, necessary stuff I keep banging on about. Pin your CI actions to commit SHAs so a moved tag can’t swap code under you. Commit your lockfiles. Run the auditors, cargo-audit, pip-audit, govulncheck, npm audit, or Google’s cross-ecosystem OSV-Scanner, on every build. Gate the dependency tree and give every exception an expiry date so “we’ll deal with it later” can’t quietly become “never”. Keep the tree small: every crate you don’t add is a stranger you don’t have to trust.

None of that is a solution. All of it is diligence, and diligence is the only thing that was ever going to stand behind the signature. When I sign a release, the cryptography is the easy part. The promise underneath it, that I pinned, locked, audited, vetted and tested before I put my name on it, is the part worth anything. That’s the contract. The signature is just how I countersign it.

The encouraging note is that the structural defences exist and they work. Go’s checksum database and its refusal to run hooks. Composer declining to trust a dependency’s scripts. Python’s wheels. cargo-vet and cargo-deny giving you somewhere to record human judgement at scale. More ecosystems should steal these shamelessly, because a registry that makes the safe path the default does the working engineer a far bigger favour than one that leaves it all to discipline.

The same shape, a third time

If this feels familiar, it should. I wrote recently about a bug bounty that collapsed because the cost of slop was deferred, and about a junior pipeline being cut because the bill lands years later. Supply-chain security is the same shape a third time. The convenience is now, the catastrophe is later, and the only thing standing in the gap is an engineer paying attention, doing the dull work, refusing to be rushed into trusting something they haven’t checked.

There is no clean package waiting to be found, no registry about to solve this for us, no signature that means “safe” all on its own. There’s the diligence you do before you put your name to something, and the judgement to know when an install is asking you to trust more than you should. For a good while yet, that is the whole job. Boring, unfashionable, and the only thing that works.