Which tests would you cut when the bill arrives?
The runner fleet cost $270 I didn't have, so it went off. The tests stayed on, and the savings came from everything I'd been paying for twice.
Tags
12 pages
Testing
The release that OOM-killed its own compiler
A wedding-photo culler ran out of memory compiling an LLM SDK it has never once called. The release had been running for thirty-eight minutes at that point.
Prove the bug before you fix it
A model handed a bug will hand you back a plausible fix and a green test suite in about a minute. That tells you almost nothing.
FFmpeg thinks it has a disk
FFmpeg assumes a disk it can seek around in. Giving it a convincing in-memory filesystem, and testing the behaviour that would hurt most first.
The scaffolder that won't hand you code that doesn't compile
A Rust scaffolder with an AI codegen path that drafts a real command, then refuses to hand it over until it compiles and passes lint.
When you hand the same key to every call
A generated nested command would not compile, and the cause was a scaffolder handing every call the same key regardless of depth.
From allow_failure to blocking
Turning three advisory CI jobs into real gates once they were trustworthy enough to block a merge, and how to tell when they are.
Process isolation won't save you from the filesystem
A flaky self-update test: separate processes still shared a computed cache path and raced on disk. Process isolation is not filesystem isolation.
The test-mocking pattern that races
Reassigning a package-level function variable in tests is a latent data race that t.Parallel exposes. The fix is structural, not another lock.
Testing code that calls an LLM: yes, you actually can
You cannot test the model, but you can test your code: snapshot the prompt as a golden file and mock the response to cover the ugly cases.
