Debugging on PHP Boy Scout

The secret that wasn't on my branch

Sat, 04 Jul 2026 00:00:00 +0000

My CI went red on a secret I’d never committed.

Not a close call, not a near-miss I’d half-forgotten about. gitleaks, the secret scanner, failed a merge request of mine on a private key that was not on my branch, was not in my change, and as far as I could tell had nothing to do with me at all. The job was adamant. I was baffled. Somewhere in between was a lesson about what a secret scanner actually scans.

Prove it isn’t yours

First rule of being accused: don’t get defensive, get evidence. The scanner pointed at a couple of commits carrying a test private key and a PEM block in a spec document. I genuinely didn’t recognise them, but “I don’t recognise it” is a feeling, not a fact, and feelings don’t reopen a pipeline.

Git will tell you the truth if you ask it precisely. The question is: are these flagged commits actually part of my branch?

git merge-base --is-ancestor <flagged-sha> HEAD

That asks “is this commit an ancestor of where I am?”. The answer came back no. The commits the scanner was choking on were not in my history. They weren’t mine.

So whose were they? A bit of digging turned them up on a completely separate, still-unmerged branch, where someone (me, a few days earlier, on a different feature) had committed a throwaway test key and a PEM example in a spec, on purpose, as fixtures. Deliberate, harmless, and nowhere near the branch under review. And yet here they were, failing a merge request that had never touched them.

What gitleaks actually scans

Here’s the bit I’d taken for granted. I assumed gitleaks detect scanned my change. It doesn’t. With no further instruction it scans the whole history reachable in the checkout it’s handed.

And the checkout it’s handed is where the second half of the surprise lives. GitLab runners default to GIT_STRATEGY: fetch, which reuses the runner’s working directory between jobs rather than cloning fresh every time. It’s faster, and most of the time you never notice. But it means a shared runner accumulates refs from every branch it has ever built. My MR’s job happened to run on a runner that had, at some point, built that other branch, so the fixtures were sitting right there in the local object store, fair game for a scanner walking the whole graph.

So gitleaks did exactly what I’d asked it to do, which was “scan everything”, and “everything” turned out to be a great deal more than my change. It walked the lot and dutifully reported fixtures from a branch I wasn’t even proposing to merge. The scanner wasn’t wrong. My idea of what it was looking at was.

Unblock now, fix properly after

Two problems on two timescales. I needed the MR to merge today, and I needed this to never happen again. Those want different fixes.

The immediate one: tell gitleaks those specific fixtures are known and intended. They’re test material, they’re meant to be there, so they go in the allowlist:

paths = [
 # ...
 '''internal/cmd/keys/keys_test\.go''',
 '''docs/development/specs/2026-06-08-keys-mint-command\.md''',
]

That unblocks the merge. It does nothing about the root cause, which is that the scan was looking at the wrong commits in the first place. Allowlisting individual false positives as they crop up is closing the stable door after the horse has bolted, one horse at a time, forever.

The real fix lives in the shared CI component, not in any one repo. Scope the scan to the commits the merge request actually introduces (cicd v0.10.3):

if [ -n "$CI_MERGE_REQUEST_DIFF_BASE_SHA" ]; then
 gitleaks detect --source . --verbose --redact \
 --log-opts="$CI_MERGE_REQUEST_DIFF_BASE_SHA..$CI_COMMIT_SHA"
else
 gitleaks detect --source . --verbose --redact
fi

--log-opts is handed straight through to git log, so that range, base-of-the-MR to tip, is the exact set of commits the merge request adds and nothing else. On a merge request the scan now sees only what you’re proposing to merge. Off a merge request (a plain branch or a tag pipeline) it falls back to the full scan, because there you genuinely do want the lot. The before-and-after in the job log tells the whole story: the entire accumulated history on one side, the handful of commits you actually wrote on the other.

Fixing the fix

There’s a tax on touching CI shell, and I paid it. The change went into the go, rust and tofu security templates. Go and tofu went green. Rust failed.

The rust template had built its optional --config flag the clever way, inline:

... $([ -n "$CONFIG" ] && echo "--config $CONFIG")

As a command argument, that substitution is harmless: its exit status is thrown away, so nobody cares that the test inside returns false when there’s no config. But when I rewrote the block and reached for the same pattern as an assignment, VAR=$(... && ...), it became a different animal. An assignment takes the exit status of the command substitution, and under set -e a non-zero status anywhere aborts the job. So on every run where the config was empty, which was most of them, the test returned false, the assignment inherited that false, and set -e killed the job stone dead. Same $(...), two completely different fates, decided entirely by whether it sat to the right of an = or got handed to a command as an argument. Go and tofu never used the assignment form, so only rust fell down the hole.

The fix was to stop being clever and write the boring if:

GITLEAKS_CONFIG=""
if [ -n "$CONFIG" ]; then
 GITLEAKS_CONFIG="--config $CONFIG"
fi

Boring shell is good shell.

What a scanner is actually looking at

The whole mess came from one unspoken assumption: that a tool called to scan “my change” was scanning my change. It was scanning a checkout, and a checkout on a shared runner is a much bigger, messier thing than the diff in front of you. None of the pieces were broken. gitleaks did its job, GIT_STRATEGY: fetch did its job, my fixtures were exactly where I’d left them. They just added up to a red pipeline that had nothing to do with the code I was trying to ship. I’d spent a good chunk of the day proving my innocence to a scanner that was only ever doing as it was told… and the one thing I’d actually got wrong was being sure I already knew what it was looking at.

The cobra hook I was sure I'd enabled

Wed, 24 Jun 2026 00:00:00 +0000

It came out of an audit. I’d recently pointed a small army of review agents at the whole go-tool-base codebase, back before that became a political problem, and one of the findings was that a subcommand could quietly skip the framework’s own start-up code. My first reaction was the dangerous one: surely not… we switched that on ages ago. So I asked for a second pair of eyes on the exact line.

There was no line. I was certain I’d enabled it, but I had simply never done it.

What the start-up hook is for

go-tool-base is built on cobra, the library most Go command-line tools are built on. In cobra, a command can carry a PersistentPreRunE: a function that runs before the command itself, and that, the name strongly implies, persists down to the command’s children. Think of it as the “before you do anything, get the tool ready” step.

go-tool-base uses exactly one of them, on the root command, to do all the humdrum setup: load and merge configuration, set up logging, ask about telemetry the first time you run, wire up the telemetry collector, and check whether there’s a newer release to install. Everything the tool does afterwards leans on that having happened. By the time your actual command runs, props.Config is meant to be populated and the collector is meant to exist.

The reasonable assumption (the one I’d made, anyway) is that “persistent” means it cascades. Define it once at the root and every mytool foo bar three levels down gets it for free.

“Persistent” promises less than it says

Here is the catch, and it is a good one to file away if you ever build a command tree. cobra runs only the nearest PersistentPreRunE it finds, walking up from the command you actually invoked. If a subcommand defines its own, that one runs and the root’s does not. Not as well as. Instead of. There’s no warning and no error; the child’s hook simply wins, and the parent’s is passed over in silence.

So the moment any command below the root declared its own PersistentPreRunE, the entire start-up for that branch, the config, the logging, the telemetry, the update check, would just not happen. props.Config would be nil. The collector would be nil. The first you’d hear of it is a nil-pointer panic a long way from the cause, or, worse, no panic at all and a tool running happily unconfigured.

EnableTraverseRunHooks is cobra’s opt-in to the behaviour most people assume is already the default: run every PersistentPreRunE from the root down to the leaf, in order. I’d assumed it was the default. It is not, and I’d never turned it on.

A landmine nobody had stepped on

The saving grace was that nothing was actually broken yet. In go-tool-base’s own command tree, the root is the only command that defines a persistent pre-run, so “root to leaf” and “nearest only” happen to produce the identical result. The flag being off changed nothing I could observe.

The bug was latent. It was a trap laid for the first person to do something entirely reasonable: add a PersistentPreRunE to one of their own subcommands. go-tool-base is a framework other tools are built on, so that person was never going to be me. The instant a downstream author did the obvious thing, their config and telemetry would vanish for that branch of their tool and nothing would tell them why.

That is the kind of bug I least like shipping. It compiled. It passed the tests. It would have demoed perfectly. And it sat there waiting to hand a stranger a debugging session with no breadcrumbs, for the crime of using a standard cobra feature the obvious way.

One line, and a note for whoever’s next

The fix is the line I was so sure already existed (root.go):

// Run every parent PersistentPreRunE from root to leaf rather than only the
// closest one. Without this, a downstream subcommand that defines its own
// PersistentPreRunE silently shadows the root bootstrap (config load,
// telemetry, update check) for that subtree. With it set, the framework
// bootstrap always runs first and a child hook runs after it.
cobra.EnableTraverseRunHooks = true

With it set, cobra runs the root start-up first and then the child’s hook, in order, so a downstream command adds to the setup instead of replacing it.

I didn’t want to stop there, because the next author to add a child hook still deserves to understand the ordering. So the change also drops a one-time debug line if it spots any command in the tree carrying its own PersistentPreRunE (the same file), saying out loud what’s going on:

l.Debug("a downstream command defines its own PersistentPreRunE; " +
	"it runs AFTER the framework bootstrap (config load, telemetry, update check), not instead of it")

And, belt and braces, the collector now defaults to a no-op so the few paths that do legitimately return early, like init and help, still satisfy the “always non-nil” promise the rest of the code relies on. The whole thing shipped with a pair of regression tests that assert the bootstrap really does run when a child hook is present, and that it runs first. It’s all written up in a short spec and landed in one commit.

Trust, but grep

There are two things worth taking away. The cobra one is portable: if you rely on PersistentPreRunE cascading down a command tree, set EnableTraverseRunHooks, because “persistent” means less than it sounds and the nearest hook wins by default.

The other is the one I keep having to relearn. The settings I’m most certain about are the ones I never check, precisely because the certainty is what stops me looking. Somewhere along the line I’d promoted “I meant to” straight to “I did”, with nothing in between… and then defended it out loud before I’d even gone to look. A review agent is good at exactly that blind spot: it has no memory of intending to do something, only the code in front of it. The best thing the audit turned up wasn’t a clever bug. It was a flag that was never there. Leaving the campsite better than you found it has to include the traps nobody’s stepped on yet.