Testing on PHP Boy Scout

When you hand the same key to every call

Tue, 16 Jun 2026 00:00:00 +0000

I was building a tutorial, the kind where the whole point is that the reader runs every command and it just works. So I generated a fresh project with go-tool-base, added a command, then added a command underneath that command, and hit build. It didn’t.

pkg/cmd/hello/cmd.go: props.ChildCmd undefined
 (type *props.Props has no field or method ChildCmd)

My own generator, in my own framework, had just written code that referenced a thing that didn’t exist… which is a special kind of embarrassing.

A bug with a two-month alibi

git blame walked me straight to the commit that introduced the command middleware system back in March. Middleware here is the web-style idea of wrapping a command’s run function with cross-cutting behaviour, timing, auth, recovery, that sort of thing. To wire it in, the generator started emitting this for a nested command:

setup.AddCommandWithMiddleware(cmd, child.NewCmdChild(p), props.ChildCmd)

where the line before had simply been:

cmd.AddCommand(child.NewCmdChild(p))

The catch is that third argument. props.ChildCmd is meant to be one of a set of constants, but those constants are hand-declared for the framework’s built-in commands only (UpdateCmd, DocsCmd, and friends). The generator never declares one for a user’s child command, so the generated parent referenced a name that nothing had ever declared. Undefined. Won’t compile.

Here’s the part that should worry you more than the bug. It shipped in March and nobody noticed until late May. Partly because it only bites nested commands, a command under another user command; top-level commands register by a different path and were fine. But mostly because the generator’s tests checked the generated code as text, asserting that it contained the right strings, and never once ran go build on the result. CI was green for two months on code that could not compile. We were grading the essay without ever reading it aloud.

What the key actually was

Once I stopped staring at the missing name, the real problem came into focus… and it wasn’t the missing constant at all.

That third argument is a middleware lookup key. The framework keeps a table of middleware registered against each key, and the key tells it which to apply. It is not an on/off switch and it is not optional, so the generator had to supply one at every registration site. It was being asked to guess, on every call, a value it had no reliable way to produce for a user command.

And the tell was sitting right there in the same generator: everywhere else, the idiom was props.FeatureCmd("name"), a function that derives a key from a string. The nested-registration path was the one place that assumed a hand-declared constant instead. One call site out of step with all the others.

That is the actual lesson, and it has nothing to do with cobra or codegen. When you find yourself threading the same derived value through every single call site, and getting it wrong, the value is in the wrong place. The feature key was never the caller’s business. It belonged to the command.

Changing my mind about cobra

This is where I had to eat a helping of my own opinion.

go-tool-base is built on cobra, the de-facto Go library for building command trees, and I like it a great deal. I had deliberately not wrapped it. Every abstraction over a good library is a tax the reader pays, so my standing rule was: use cobra directly, don’t hide it behind something of mine.

The trouble is the middleware pattern kept growing, and the bigger it got the more plainly “don’t abstract cobra” was a position I was holding past its evidence. The very thing I’d refused to build was the thing the design had come to need. It helped that, having recently moved the project to GitLab, the version had reset to a 0.x prerelease, which makes a breaking change cheap. The window to stop patching and do it properly was open, and not for long.

Go’s composition model made it almost painless. You can embed a pointer to one struct inside another and the outer type inherits all the inner one’s methods for free, which is about as close to monkey-patching as a statically typed language gets. So a setup.Command became a cobra command plus the feature it belongs to:

type Command struct {
	*cobra.Command
	Feature props.FeatureCmd
}

func Wrap(feature props.FeatureCmd, cmd *cobra.Command) *Command {
	return &Command{Command: cmd, Feature: feature}
}

func (c *Command) Register(children ...*Command) {
	for _, child := range children {
		if child.Command == nil {
			continue
		}
		if child.RunE != nil {
			child.RunE = Chain(child.Feature, child.RunE)
		}
		c.AddCommand(child.Command)
	}
}

Because *cobra.Command is embedded, a setup.Command is a cobra command for every method cobra offers; the one place you need the raw pointer, you reach for .Command. The generated command now carries its own identity:

func NewCmdChild(props *props.Props) *setup.Command {
	cmd := &cobra.Command{Use: "child", RunE: ...}
	return setup.Wrap(props.FeatureCmd("child"), cmd)
}

and the parent registers it with nothing threaded through the call:

parent.Register(child.NewCmdChild(p))

The feature key now lives on the command, derived from the command’s own name, which is the one place the generator can always produce it correctly. The bug isn’t so much fixed as made unsayable: there’s no call site left to write the wrong thing into. The wiring got cleaner on the way past, too, each command’s run is wrapped exactly once with its own feature, instead of the old recursive pass that re-applied the parent’s feature down the whole subtree. And the old free function stays on as a deprecated shim that just calls Register, so nothing downstream breaks before v1.0.

Changing my mind about the tests

The redesign was the satisfying fix. The test was the important one.

The reason a non-compiling generator sailed through CI for two months is that its tests read the generated source as text. A generator is a program that writes a program, and we were checking that it wrote the expected words without ever asking whether the words formed a working program. So the redesign shipped with a different kind of test, one that scaffolds a real project, adds a nested command, and actually builds it. Its own comment says the quiet part out loud:

The previous test suite asserted file-content shapes but never tried to go build the generated module, so the nested-command path that referenced undefined props.<Name>Cmd symbols compiled cleanly in tests and broke only when downstream users built their tools.

It’s gated behind an integration flag, because it shells out to the Go toolchain and that’s too heavy for every unit run, but it closes the exact gap that hid the bug. The only real test of a code generator is whether its output compiles.

What it comes down to

Three times in one bug I had to change my mind. I’d decided cobra shouldn’t be abstracted; the evidence said abstract it. I’d reached for the one-line patch; the evidence said redesign. I’d trusted tests that read the generated code; the evidence said build it. None of those were comfortable, and the version reset is the only reason the timing was kind.

Both technical lessons are worth keeping. When the same derived value is threaded through every call, the abstraction is in the wrong place. And the only proof that something which writes code actually works is to compile what it writes. But the one underneath both, the one I apparently have to keep relearning, is simpler than either: don’t get so attached to an implementation that you can’t change your mind when the evidence says it doesn’t fit. The framework is better for it, and it now has a composition seam to hang the features cobra doesn’t give us natively. A nested command that wouldn’t build was just the thing that finally made me look.

From allow_failure to blocking

Sat, 30 May 2026 00:00:00 +0000

There’s a special kind of CI job that everyone on a team quietly learns to ignore: the one marked allow_failure: true. It runs, it goes red, the pipeline goes green anyway, and after the third time you stop looking at it. I inherited six of those when I moved rust-tool-base’s CI to GitLab. Over a few days I turned three of them into real gates, and the interesting part was never the YAML. It was working out which ones had earned the right to block, and which hadn’t.

What allow_failure actually buys you

allow_failure: true is genuinely useful, and quietly corrosive. It lets a job report a problem without stopping the pipeline, which is exactly right for a check that’s noisy, not yet stable, or guarding against something you can’t fix this minute. The trouble is that a warning nobody is forced to act on is a warning nobody acts on. Leave a job advisory long enough and it becomes scenery: red, ignored, pointless. So an advisory check is really a promise, “I’ll make this blocking once it’s trustworthy”, and a promise you only ever mean to keep is just a lie you haven’t noticed yet.

When I migrated rust-tool-base from GitHub Actions to GitLab CI, the move landed six jobs as allow_failure: true: the macOS and Windows tests, the integration tests, cargo-audit, trivy, and coverage. That wasn’t laziness. A migration is the wrong moment to also be fighting flaky gates. But it left me holding six promises to either keep or admit I wasn’t going to.

A check earns the right to block

Here’s the rule I settled on. A check earns the right to fail your build when two things are true: it’s meaningful (a red result is a real problem, not noise) and it’s reliable (it goes red only when there genuinely is a problem, and it can actually run to completion). Flip a check to blocking before both hold and you haven’t raised the bar, you’ve taught the team to force-merge past red, which is worse than no gate at all, because now the red means nothing.

Three of my six crossed that line within a few days. Three deliberately didn’t. The reasons are the whole story.

trivy: blocked once there was nothing to block on

trivy scans the dependency tree for HIGH and CRITICAL advisories. It went across as advisory for an honest reason: the Cargo.lock at migration time already carried two known HIGH/CRITICAL advisories I hadn’t cleared yet, a path-traversal in gix-validate and a DNS-rebinding issue in rmcp. Make trivy blocking with those sitting there and the pipeline is red from day one, over problems I already knew about and was already fixing. So it stayed advisory until the dependency bumps cleared both, and then the allow_failure line came out. The gate never changed. The tree underneath it got clean enough to stand on.

integration-tests: blocked once it could actually run

The integration tests stand up a real Gitea in a Docker-in-Docker service and talk to it. They were advisory for a different reason: they couldn’t reliably run. dind needs a privileged runner, and the suite was resolving the container host with a hardcoded 127.0.0.1 that didn’t hold everywhere. Blocking a job that fails for infrastructure reasons rather than code reasons is the fastest way to make people distrust the entire pipeline. So the fix wasn’t in the YAML, it was making the thing dependable: privileged set on the runner, and the host resolved through the test library’s own get_host() instead of a hardcoded address. Once it ran the same way every time, it earned the gate.

coverage: blocked once it could run at all, then once it cleared the bar

Coverage is the two-step one, and my favourite, because it nearly didn’t make it for a thoroughly undramatic reason: it ran out of memory. cargo llvm-cov instruments every test binary, and linking hundreds of instrumented object files needs more RAM than the shared medium runner had, so the job bus-errored on the link. I tagged it onto a larger runner, and then the shared SaaS runners were switched off entirely, so the tag matched nothing and the job sat pending forever.

The fix was a self-hosted homelab runner with the RAM the instrumented link actually needs. I moved coverage there but kept it advisory for one run, to confirm the box could finish the build before I trusted it. It did, at 73.22% line coverage, so I set the gate to fail under 70% and made it blocking. Three points of headroom: enough that ordinary churn won’t trip it, tight enough that a real drop will. A coverage gate pinned to the current number is a tripwire that fires on the very next commit; set it a touch below and it catches regressions instead of normal life.

The three I left advisory, on purpose

The point was never “block everything”. Three jobs are still allow_failure in the current pipeline, deliberately. The macOS and Windows tests run on SaaS runners that bill by the minute; they’re worth running, not worth blocking every merge of a Linux-first project over a quota I’m choosing to ration. And cargo-audit stays advisory because cargo-deny already does the blocking advisory check: cargo-audit is a second opinion from a different database, and a second opinion that can veto isn’t a second opinion, it’s a duplicate gate that will eventually disagree with the first and block you on the difference.

That’s the same rule from the other side. Those three haven’t earned the right to block, because blocking them would cost more than it ever caught.

The upshot

allow_failure: true is fine as a waiting room and corrosive as a destination. Every advisory check is a promise to make it blocking once it’s both meaningful and reliable, and the job is to keep the promise or admit you won’t. trivy earned its gate when the advisories cleared, the integration tests when they ran the same way every time, coverage when it had a runner with enough memory and a threshold set just below the current mark. The three I left advisory earned that standing too, by costing more to block than they’d catch. The YAML is one deleted line per job. Knowing which line to delete, and when, is the whole skill.

Process isolation won't save you from the filesystem

Sat, 25 Apr 2026 00:00:00 +0000

A test that passed every single time I ran it on its own, and failed maybe one run in five when I ran the whole suite. The failure was always the same: the self-update test downloaded a release archive, went to extract it, and found the archive corrupt. Half-written. As if something had been scribbling in the file while it read it. Something had.

The comfort I was leaning on

The self-update tests are heavier than a unit test wants to be. They stand up a fake release, download the artefact, verify its checksum, extract it, swap a binary. Real files, real I/O. So they’d been built to run as separate processes, not just separate threads, each one its own little world.

And I’d quietly filed that under “solved”. Separate processes don’t share an address space. One can’t reach into another’s memory and corrupt a value mid-read. That whole category of data race, the kind you reach for a mutex to fix, simply can’t happen across a process boundary. So I’d stopped thinking about concurrency in these tests at all, because I’d convinced myself the isolation was total.

It wasn’t total. It was isolation of memory, and I’d let myself hear it as isolation of everything.

Two processes, one path

The thing two processes very much do still share is the filesystem. And the self-update flow, sensibly, caches its download rather than re-fetching it. The default cache directory is computed from the tool’s name and the release version, in crates/rtb-update/src/flow.rs:

pub fn cache_dir_for(tool_name: &str, version: &str) -> PathBuf {
 let base = directories::ProjectDirs::from("", "", tool_name)
 .map_or_else(std::env::temp_dir, |p| p.cache_dir().to_path_buf());
 base.join("update").join(version)
}

Read that with two parallel test processes in mind. They’re testing the same tool, against the same fake release tag. So tool_name matches and version matches, which means cache_dir_for hands both of them the identical path. Two processes, isolated in every way that involves memory, both downloading and extracting into one shared directory on disk, at the same time. One writes the archive while the other is partway through reading it, and you get exactly the corrupt half-written file the test kept tripping over.

Process isolation did nothing here, because the contention was never in memory. It was on a path string that came out the same for both of them.

Once it’s framed as “they share a path”, the fix writes itself: don’t share the path. Give each invocation its own cache directory. The updater builder already had the seam for it, and the doc comment now says exactly why it’s there, in crates/rtb-update/src/updater.rs:

/// Tools call this when they want isolation per-invocation
/// (e.g. CI runners, tests with parallel processes) or to honour
/// a user-supplied `--cache-dir` flag.
pub fn cache_dir(mut self, cache_dir: impl Into<PathBuf>) -> Self {
 self.cache_dir = Some(cache_dir.into());
 self
}

Each test now builds its updater with cache_dir(its_own_tempdir), so two parallel processes land on two different directories and never meet. No lock, no serialisation, no clever cross-process file mutex. Just the realisation that the shared thing was a directory, and the cure for shared mutable state is usually to stop sharing it, not to guard it.

The fix that turned out to be a feature

The part I’m quietly pleased about is that this didn’t stay a test-only hack. The override I needed to isolate the tests is exactly the override a real tool wants for its own reasons. A CI runner doing self-update wants a writable cache path it controls, not wherever directories-rs decides the system cache lives. A user might reasonably want to point the whole thing somewhere specific. That’s a --cache-dir flag, and cache_dir() is precisely the hook you’d wire it to.

So the thing I added to stop a flaky test is the same thing a downstream tool reaches for to expose --cache-dir. The test forced the seam to exist, and the seam was worth having anyway. I’ll take that trade every time over a fix that only the test suite benefits from.

What it comes down to

I’d treated “separate processes” as a synonym for “can’t race”, and it isn’t. Processes don’t share memory, so the memory races are gone. They absolutely still share the filesystem, the network, every named resource the OS will hand to anyone who asks for it by the same name. My two test processes computed the same cache path from the same tool and tag, and raced on the files in it, and no amount of address-space isolation was ever going to touch that.

Shared mutable state on disk is still shared mutable state. The fix wasn’t a bigger hammer, it was giving each process its own directory and letting the isolation I thought I already had actually be true.

The test-mocking pattern that races

Thu, 16 Apr 2026 00:00:00 +0000

I’m going to tell you about a bug go-tool-base shipped, because it’s one of those bugs that’s so reasonable-looking you’ll find it in textbooks, conference talks, and an awful lot of otherwise excellent Go code. We had it too. It passed every test on my laptop, every single time, and then quietly fell over on CI while blaming an innocent bystander.

It’s the classic Go trick for mocking a dependency, and it races.

A pattern that looks completely reasonable

Here’s a thing you need to do constantly in Go tests: stop a function from really shelling out. It calls exec.LookPath to find a binary, or exec.Command to run one, and your test very much does not want it touching the real $PATH or spawning a real process.

The Go community has a well-worn answer. Hoist the function into a package-level variable, call that, and let tests reassign it:

// production code
var execLookPath = exec.LookPath

func findTool() (string, error) {
 return execLookPath("sometool")
}

// test
func TestFindTool(t *testing.T) {
 old := execLookPath
 defer func() { execLookPath = old }()
 execLookPath = func(string) (string, error) {
 return "/fake/path", nil
 }
 // ...assert...
}

It’s tidy. No interface to thread through, no constructor to change. You’ll find it in a great deal of Go code, including some very respectable Go code indeed. go-tool-base had it too.

And it works. It works on your machine, it works in code review, it works the first hundred times CI runs it. Which is precisely what makes it dangerous, because it’s wrong, and it’s just been biding its time.

Add one line and it detonates

Go’s t.Parallel() is more or less free performance. Mark your tests with it and the runner overlaps them instead of plodding through one at a time. On a package with a few hundred tests it’s a real, worthwhile speed-up, so naturally you reach for it.

Now picture two tests, both using the pattern above, both marked t.Parallel(). They run concurrently. Test A assigns its fake to execLookPath. Test B assigns its fake to execLookPath. Test A reads execLookPath expecting its own fake. Two goroutines, one variable, writes and reads with nothing synchronising them. That’s a textbook data race, and the textbook is right: the behaviour is undefined. Test A might see B’s fake. The deferred restore might land in the wrong order and leave the variable pointing at a fake after both tests have finished, poisoning a third one for good measure.

The truly nasty part is the intermittency. Whether the race actually bites depends on goroutine scheduling, which depends on machine load and core count. Your laptop running eight tests at once might never lose the coin-toss. A CI runner under load, scheduling differently, loses it and fails a test that has nothing obviously to do with the change in the commit. You re-run the pipeline, it passes, everyone shrugs and moves on. A test suite that fails one run in twenty trains your team to ignore it, and an ignored CI failure is worse than no CI at all.

I can tell you this one from direct, slightly embarrassed experience, because go-tool-base shipped exactly this bug and CI caught it the honest way: green on the laptop, red on the runner, with the failure cheerfully pointing at innocent bystander tests rather than the global that was actually the culprit. go test -race will name it for you if you crank the parallelism up high enough to lose the toss reliably… but you have to go looking, and you only go looking once it’s already ruined an afternoon.

The fix isn’t synchronisation, it’s structure

The instinct is to slap a mutex around the variable. Resist it. A mutex makes the race defined, but it doesn’t make the design any good. You’ve still got global mutable state, you’ve just queued the fight instead of cancelling it. And tests that serialise on a shared lock aren’t really parallel any more, so you’ve also handed back the speed-up you came for in the first place.

The real fix is to not have a shared variable at all. The dependency was always an input to the code; the package-level var was just a way of avoiding saying so out loud. So say it. Inject it.

A struct field:

type Finder struct {
 lookPath func(string) (string, error) // defaults to exec.LookPath
}

func (f *Finder) find() (string, error) {
 return f.lookPath("sometool")
}

Or a functional option, if you’d rather keep the zero value clean. Either way, each test constructs its own Finder with its own fake. There’s no shared variable, so there’s no race, and t.Parallel() is free again because the tests genuinely don’t touch each other.

go-tool-base wrote this straight into its standing rules: no package-level mocking hooks, full stop. Dependencies come in through struct fields, functional options, or config fields. (The same injection discipline that makes Props so testable, applied one rung further down.) And to stop everyone hand-rolling the same exec fakes, there’s a small internal package, internal/exectest, with ready-made LookPath and CommandContext doubles you construct per-test. The pattern is gone, and the door it came in through is shut.

The rule worth taking away

A package-level variable that tests reassign is shared mutable state. It reads as a harmless convenience because in a single-threaded test run it behaves like one. t.Parallel() is the thing that reveals it was never harmless, only unobserved.

The general lesson is older than Go: if a value is an input to your code, make it an input. Smuggling it in as a global is borrowing test-time convenience against a debt that comes due, with interest, the day someone wants their tests to run in parallel. Pay cash. Inject the dependency.

Worth remembering

Mocking via a reassignable package-level variable is a beloved Go shortcut and a latent data race. It survives because single-threaded test runs hide it; t.Parallel() exposes it as intermittent, bystander-blaming CI flake that’s miserable to trace. A mutex only makes the bad design defined. The fix is structural: inject the dependency as a struct field or functional option, so each test owns its own double and there’s no shared state to race over. go-tool-base banned the global-hook pattern outright and ships internal/exectest so nobody’s tempted back to it.

If a piece of code depends on something, let it say so in its signature. Your future self, staring at a CI failure that flatly refuses to reproduce, will thank you.

Testing code that calls an LLM: yes, you actually can

Wed, 08 Apr 2026 00:00:00 +0000

“You can’t test code that calls an AI.” I’ve heard it said with great confidence, and it’s half right, which is the most dangerous kind of right. You genuinely can’t assert on what a non-deterministic model says. But the model isn’t your code, and the bits sitting either side of it most certainly are.

“You can’t test AI code”

It’s a fair worry. Your command calls an LLM. The LLM returns something slightly different every run. A test that asserts response == "..." is broken before you’ve finished typing it. So the conclusion arrives quickly: the AI path can’t be tested, leave it uncovered.

Which is a shame, because the AI call is usually the riskiest line in the whole command.

The conclusion is also wrong. It mistakes “I can’t test the model” for “I can’t test my code”. The model is not your code. Your code is the two pieces sitting on either side of it.

Your code is a prompt and a handler

Strip the command down to what it actually does:

It builds a prompt. It assembles a system prompt, the user’s input, perhaps some context, and sends it.
The model does something. This is not your code.
It takes the response and does something with it. It parses it, branches on it, prints it, stores it.

Steps one and three are entirely yours, and entirely deterministic. The same inputs build the same prompt and handle the same response the same way, every single time. That’s testable. Step two is the only part that isn’t, and step two was never yours to test in the first place.

So the job is to pin step two to a known value, and then test one and three properly.

Test the prompt: snapshot it

Step one produces a prompt, and a prompt is just a string, which means you can pin it.

Both frameworks lean on snapshot testing here. go-tool-base uses a golden-file approach: the prompt your code generates is recorded to a file, and the test re-generates it and compares against that file. rust-tool-base does the same with insta, snapshotting the request body the client would send.

The reason this matters is that the prompt is load-bearing and quietly easy to break. You refactor how context gets assembled. Without noticing, you’ve changed the wording, or the ordering, or dropped a line the model was leaning on. Nothing fails to compile. The behaviour just drifts, silently.

A snapshot test catches exactly that. It fails, it shows you the diff between the old prompt and the new one, and it makes you stop and make a decision. Was this change intended? If yes, you accept the new snapshot and move on. If no, you’ve just caught a bug before it shipped. Either way the prompt never changes by accident, which for AI code is most of the battle.

Test the handler: mock the response

Step three needs a response to handle, and in a unit test you don’t get that response from the real model. You supply it.

go-tool-base ships generated mocks for the ChatClient interface. A test builds a mock client, tells it “when Ask is called, return this canned value”, and runs the command against it:

mockClient := mock_chat.NewMockChatClient(t)
mockClient.EXPECT().
 Ask(mock.Anything, mock.Anything, mock.AnythingOfType("*main.Analysis")).
 RunAndReturn(func(_ context.Context, _ string, target any) error {
 *(target.(*Analysis)) = Analysis{Severity: "critical"}
 return nil
 })

Because the interface is only four methods, that mock is trivial to set up and complete by construction. rust-tool-base takes the same idea one layer down: HTTP-bound tests use wiremock, which stands up a fake server returning a canned response body. The client makes a real HTTP request; it just goes to a fake endpoint the test controls.

Either way, step two is now fixed to a value you chose, which makes step three deterministic. And that unlocks the tests that actually matter: given a malformed response, does the command fail gracefully? Given a rate-limit error, an empty answer, a field missing? Those are the cases a live model almost never hands you on demand, and a mock hands you every time, on the first run.

This is, incidentally, the same discipline as the test-mocking work elsewhere in the framework: the dependency is injected, so the test gets to decide what it does.

What you deliberately don’t test

One boundary worth stating. None of this tests whether the model gives good answers. That question is real, but it’s a different activity (evaluations, run as their own suite) and not something to mix into the unit tests.

The unit suite’s job is your code: that it builds a sound prompt, and that it handles every shape of response correctly, including the ugly ones. Keep that well away from “is the model clever today”. A unit test that depends on the model being clever is a unit test that fails when the weather changes, and a flaky test just teaches people to ignore the whole suite.

What it comes down to

Code that calls an LLM is testable; the model is not, and those are different statements. Your code is a prompt builder and a response handler, both deterministic, with the model sat in between.

go-tool-base and rust-tool-base converge on the same approach. Snapshot the prompt, with golden files or insta, so a refactor can’t change what you send without a test noticing. Mock the response, with generated ChatClient mocks or a wiremock server, so tests run with no network and you can feed in the malformed and error cases a real model won’t reliably produce. Leave “are the answers any good” to a separate evaluation suite. Test the two halves you own, and the non-determinism in the middle stops being an excuse to leave the riskiest line uncovered.

BDD where it earns its place, and nowhere else

Sat, 28 Mar 2026 00:00:00 +0000

I have a slightly complicated relationship with BDD. I’ve watched it turn a tangled test suite into something the whole team could read and reason about, and I’ve watched it turn a perfectly good unit test into a paragraph of ceremonial English that nobody benefits from. So when go-tool-base brought in Cucumber-style BDD, the interesting decision wasn’t adopting it. It was being ruthless about where not to.

Two tests that hurt for different reasons

Most of go-tool-base’s tests are ordinary table-driven Go tests, and they’re absolutely fine. A function, a slice of input/expected pairs, a loop. Nobody needs Gherkin to understand a parser test.

But two areas were genuinely painful, and they were painful in the same way: the test had become harder to understand than the thing it was testing.

The first was pkg/controls, the service-lifecycle package. It runs a small state machine (Unknown, Running, Stopping, Stopped) with signal handling, health monitoring, restart policies and graceful shutdown all woven through it. The integration tests for graceful shutdown had grown to over three hundred lines of imperative goroutine and channel coordination. They worked. But reviewing them was a slog, and a test you can’t review with confidence is a test you can’t trust when it fails. The behaviour being checked, “when a shutdown signal arrives mid-startup, the controller stops cleanly”, was a simple sentence buried under a heap of synchronisation scaffolding.

The second was the CLI itself. init, update, doctor are user workflows. “Given a config file with a custom value, when I run init, then the custom value survives the merge.” That’s already a Given/When/Then; it just happened to be written out as Go.

Godog, and the line I drew

Godog is the official Go implementation of Cucumber. You write .feature files in plain Gherkin and bind each step to a Go function. The shutdown scenario stops being three hundred lines of channels and becomes this:

Scenario: graceful shutdown completes within the deadline
 Given a controller with two registered services
 When a shutdown signal is received
 Then both services stop in registration order
 And the controller reports a clean shutdown

The goroutine choreography doesn’t vanish, of course. It moves into the step definitions, written once and reused. What changes is that the scenario is now readable by someone who’s never opened the file before, including someone from an ops team who’ll never write a line of Go but absolutely has opinions about how shutdown should behave.

Here’s the part I want to dwell on, because it’s the part most BDD adoptions get wrong. The first design decision written down for this work was: strategic, not universal. Use Godog only where BDD adds clarity. Keep table-driven Go tests as the baseline everywhere else.

That sounds obvious written down. It is not obvious in practice, because BDD has a gravitational pull. Once a team has feature files, there’s a powerful urge to express everything as feature files, for consistency. And that’s how you end up with Gherkin scenarios for a pure function (Given the number 2, When I double it, Then I get 4) which is pure ceremony. You’ve wrapped a one-line table test in a paragraph of English and a step-definition indirection, and made it actively worse.

The test for whether BDD belongs is this: is this test a narrative, or is it a matrix?

A matrix is the same logic with many input/output pairs. That’s a table-driven test, that’s most unit tests, and Gherkin actively harms them. A narrative is a sequence of steps where the ordering and the state between steps is the thing under test, and that’s where Gherkin pays for itself. Lifecycle transitions are narratives. A user running three commands in sequence is a narrative. Doubling a number is not.

go-tool-base drew that line and stuck to it. Feature files live in features/ at the project root, where a non-Go developer can find and read them. Step definitions live in test/e2e/, kept well away from the unit tests. And the unit tests stayed exactly what they were, because they were already the right tool.

Made to fit, not bolted on

A couple of smaller decisions kept the BDD layer from feeling like a foreign object.

It runs under go test. There’s no separate Cucumber runner to install or remember. A godog.TestSuite is invoked from an ordinary TestFeatures(t *testing.T), so the BDD scenarios run in the same go test ./... as everything else. CI didn’t need a new concept bolted onto it.

And the CLI end-to-end tests build the gtb binary once and reuse it across every scenario. Compiling a binary per scenario would make the suite slow enough that people would quietly start skipping it, and a test suite people skip is just decoration. Build once, test many.

Stepping back

go-tool-base brought in Godog for BDD, but the decision worth writing about is the restraint. BDD was applied to exactly two things: the service-lifecycle state machine, where a 300-line goroutine tangle became a four-line scenario anyone can review, and CLI workflows, which are Given/When/Then by their very nature. Everywhere else, table-driven Go tests remained the baseline, because wrapping a matrix test in Gherkin makes it worse, not better.

The useful rule: BDD fits a narrative, ordered steps with meaningful state in between, and fights a matrix. Adopt it as a scalpel for the narratives. Resist the pull to turn it into a religion.

Loaded Testing

Sat, 30 Jun 2012 00:00:00 +0000

I recently had to do some load testing for a site recently that would allow me to test in excess of 100k requests in a 60 second period…

So I decided to do some testing using JMeter as it seemed like a suitable tool for doing what I needed and I had used it for some simpler testing in the past.

After a little fumbling around I managed to get a test plan designed that would simulate 10k users actually navigating the site and adding to a cart etc, with a number of various interactions. It wasnt perfect but it would correctly simulate over 100k requests.

So feeling quite pleased with myself I started the test from my laptop. Now I’m not a big gamer, I’m known to play a little World or Warcraft from time to time but that’s about it. So when it comes to computing power i tend to opt for battery life over sheer grunt.

Suffice to say, my laptop fell flat on its face, and if it hadn’t it turns out that the connection I was using just wasn’t up to the task of handling that much traffic adequately.

So plan B…

I quickly fired up the largest AWS instance available and got a copy of jmeter installed. A little tinkering with my test plan and some googling on how to run jmeter without a gui and a quick

./jmeter -n -t test-plan.jmx

and it appeared to be running.

(Please bear in mind that I’m being overly kind… it took a LOT of tinkering and twice as much Googling to work out how to get the test results out so i could actually get some idea of WTF was happening during the test)

So… client “happy”… I decided to go and find a better way to do my load testing in the future.

Sticking with JMeter I managed to find this gem of a page

http://jmeter.apache.org/usermanual/remote-test.html

tl;dr > use your local install of jmeter to trigger tests to run on one or more remote “nodes” and then have all the results sent to your local install.

So I set to work!

Building a Node

First I need to set up an AWS instance that we can use and duplicate so I can quickly build a cluster of nodes on demand. I’m a big fan of Ubuntu so I spin up a micro instance of 12.04 server. Next I shell into the instance and install the default Java runtime from apt

apt-get install openjdk-7-jre

Yes I know there are other more appropriate runtimes, but i dont really care… i just need it to work and it does.

next I grab a copy of the latest stable from http://jmeter.apache.org/download_jmeter.cgi and un-tar it to /usr/local/jmeter

(N.B. JMeter is available through the apt but I had issues with that version and you need to make sure that both your local version and all the nodes run the same version of jmeter)

We can now test that the install is working running /usr/local/jmeter/bin/jmeter-server and you should get some output that looks similar to

Created remote object: UnicastServerRef [liveRef: [endpoint:[10.???.???.???:38939](local),objID:[46522b57:138381f1023:-7fff, 2635011707874933136]]]

Which tells us that the server is running.

BUT unfortunately its not going to work just yet. Because we are using Amazons EC2 we are going to relying on their NAT for routing. Out of the box JMeter just wont work properly.

However there is something we can do to combat this. We can set the parameter RMI_HOST_DEF that the /usr/local/jmeter/bin/jmeter-server script will include in starting the server.

export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

I’ll explain what we are doing here. Amazon have been quite clever by providing a meta-data endpoint that you can poll from within your instance to get key pieces of data… Including the public dns record.

We can use this endpoint and using wget pipe that into the RMI_HOST_DEF param (ensuring that we prepend -D) and then export that so it becomes available to the /usr/local/jmeter/bin/jmeter-server script.

Now to get the server to start on boot.

a quick upstart script should solve this

# Upstart script to initialise jmeter-server

description "JMeter Server"
author "Dev in Charge "

start on started networking
stop on stopping networking
stop on stopping shutdown

console output

script
 # get the current public DNS record
 export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

 # start jmeter in server mde
 /usr/local/jmeter/bin/jmeter-server
end script

saving this to /etc/init/jmeter-server.conf will mean that it will auto-start jmeter-server on boot and allow you to manually control the process using start jmeter-server and stop jmeter-server

and thats it… instance configured

All you need to do now is save the instance as an AMI and you have an on-demand image for spinning up a cluster of remote JMeter servers for you to play with.

Configuring your local installation

Now that the server side is working we need to configure our local installation to allow it to connect.

First things first however, make sure you are using the same version of JMeter as you are running on the server.

We need to edit the jmeter.properties file that can be found in the bin folder of the installtion you downloaded. Look for the parameter remote_hosts This needs to be set with the public dns of the remote server(s) your connecting to. for example

remote_hosts=ec2-176-34-164-170.eu-west-1.compute.amazonaws.com,ec2-123-34-456-789.eu-west-1.compute.amazonaws.com

Thats your local version configured. You will now be able to tell your local version to run tests on any or all of your specified remotes.

However if your like me you work behind a router/firewall. If so this isnt the end of the story. When you send a test plan to a remote from your local install it will also send the IP address of your local machine for it to send the results back to. JMeter does this by looking up where your current hostname resolves to. In my circumstance it resolved to 127.0.1.1. The reason it did this is down to the fact my systems host file had the line

127.0.1.1 devincharge.local

To resolve this I had to change it to my external IP address

89.345.871.79 devincharge.local

And set up port forwarding from my router to my local machine for all ports from 1024 to 65535. Now, you can if you want use specific ports so you dont have to port forward everything from your router, but i’ll leave that for you to lookup as there are plenty resources on how to do this for you to google and I’ve waffled on for far too long already.

Happy testing

Testing on PHP Boy Scout

When you hand the same key to every call

A bug with a two-month alibi

What the key actually was

Changing my mind about cobra

Changing my mind about the tests

What it comes down to

From allow_failure to blocking

What allow_failure actually buys you

A check earns the right to block

trivy: blocked once there was nothing to block on

integration-tests: blocked once it could actually run

coverage: blocked once it could run at all, then once it cleared the bar

The three I left advisory, on purpose

The upshot

Process isolation won't save you from the filesystem

The comfort I was leaning on

Two processes, one path

The fix is to stop sharing the path

The fix that turned out to be a feature

What it comes down to

The test-mocking pattern that races

A pattern that looks completely reasonable

Add one line and it detonates

The fix isn’t synchronisation, it’s structure

The rule worth taking away

Worth remembering

Testing code that calls an LLM: yes, you actually can

“You can’t test AI code”

Your code is a prompt and a handler

Test the prompt: snapshot it

Test the handler: mock the response

What you deliberately don’t test

What it comes down to

BDD where it earns its place, and nowhere else

Two tests that hurt for different reasons

Godog, and the line I drew

Made to fit, not bolted on

Stepping back

Loaded Testing

Building a Node

Configuring your local installation