PHP Boy Scout — Building with AI

Your CLI is already an AI tool

Thu, 19 Mar 2026 00:00:00 +0000

“Make it work with AI” has become one of those requests that lands on a developer’s desk with a thud and not much further detail attached. My instinct, the first time, was to brace for a big lump of integration work… a bespoke adapter for this assistant, another for that one, a treadmill of little wrappers stretching off into the distance.

Turns out I’d already done most of the work. So have you, if your CLI tool is any good. Let me explain what I mean.

You already described your capabilities

Stop and think for a second about what a well-built CLI tool actually is. It’s a set of named operations, each with a human-readable description, each taking a set of typed, named, documented parameters. You wrote all of that already, because a CLI without it is unusable by people.

Now look at what an AI assistant needs in order to call a tool. A set of named operations. A description of each, so it knows when to reach for them. A typed parameter schema for each, so it knows how to call them.

It’s the same list! A good CLI is already, structurally, a description of a set of capabilities. The information an AI agent needs isn’t extra work you have to go and do. It’s work you finished the moment your --help output was any good.

The only thing missing is a translator. Something that takes “this is a CLI” and presents it as “this is a set of tools an AI can call”.

MCP is that translator, and it’s a standard

The temptation, when you want your tool to be AI-usable, is to sit down and write an integration. A little adapter for Claude Desktop. Another for Cursor. Another for whatever turns up next month. Each one a bespoke wrapper, each one a thing to maintain, and the list never stops growing because new assistants keep appearing. That’s the treadmill I was bracing for.

The Model Context Protocol exists to kill that list. MCP is an open standard for how an AI model discovers and calls local tools. Implement it once and your tool works with every assistant that speaks it. Write once, not once-per-client.

So go-tool-base implements it once, in the framework, for everyone. (That’s rather the theme of this whole series, if you hadn’t spotted it yet… do the annoying thing once, properly, in a place where every tool inherits it.)

The `mcp` command, and the mapping it does for free

Every tool built on go-tool-base inherits a built-in mcp command. Run it:

mytool mcp

and the tool starts a JSON-RPC server over standard I/O, speaking MCP. That’s the whole user-facing surface. One command.

Behind it, the framework walks your Cobra command tree and maps it straight onto MCP tool definitions:

Each command becomes a tool.
Each command’s short description becomes the tool’s description, the text the AI reads to decide whether this is the tool it wants.
Each command’s flags and arguments become the tool’s JSON Schema parameters.

There’s no second schema to write and then keep in sync (and we all know how well “keep these two things aligned by hand” tends to go). The command tree is the schema. Add a new command to your CLI and it’s a new tool for the agent, automatically, with the description and flags you already gave it. Nobody has to remember to update an MCP manifest, because there’s no separate MCP manifest to forget about.

Configuring an assistant to use it

On the assistant’s side it’s just as undramatic. You tell your AI client (Claude Desktop, Cursor, anything MCP-aware) to launch mytool mcp. From then on the assistant:

Starts your tool in MCP mode when it boots.
Discovers every command as a callable tool.
Calls the right one, with the right parameters, when a user’s request needs it.

Your CLI tool has quietly become something the AI can pick up and use, mid-conversation, on its own initiative.

The safety property worth noticing

Now, “let an AI run things on my machine” is rightly a sentence that makes people nervous. It makes me nervous, and I built the thing. So it’s worth noticing the constraint sitting quietly in this design.

The AI can only call what you defined. The tools it sees are exactly the commands in your tree, and the parameters it can pass are exactly the flags and arguments you declared, validated against the JSON Schema generated from them.

It can’t invent a command. It can’t pass a parameter you never defined. The boundary of what the agent can do is the boundary of what your CLI does, and you drew that boundary already, back when you built the tool. Exposing the CLI over MCP doesn’t widen the surface one inch. It just makes the existing surface reachable. The AI isn’t running things. It’s running your commands, the ones you wrote, tested and shipped, and nothing else.

The gist

A CLI tool, built properly, is already a structured description of a set of capabilities: named operations, descriptions, typed parameters. Which is also exactly what an AI agent needs in order to call a tool. The gap between the two is only a translator, and writing a bespoke one per assistant is a treadmill you don’t need to step onto.

go-tool-base puts the translator in the framework. Every tool gets an mcp command that serves the command tree over the Model Context Protocol… commands become tools, descriptions become descriptions, flags become JSON Schema parameters, with no second schema to maintain. Point any MCP-aware assistant at it and your CLI is an agent-callable tool, bounded to exactly the commands you shipped.

You did the hard part when you built a good CLI. MCP just opens the door you’d already framed.

An AI interface that fits on one screen

Fri, 27 Mar 2026 00:00:00 +0000

The moment you decide a CLI tool should talk to an LLM, there’s a strong gravitational pull towards reaching for LangChain, or one of its many relatives. It’s the obvious move. It’s also, for most CLI work, a bit like hiring a removals firm to carry a single box up the stairs.

Let me explain why go-tool-base went the other way, and what “the other way” actually looks like.

The instinct, and why it overshoots

When you add AI to a tool, the instinct is to reach for the big general-purpose framework. LangChain and its relatives are capable, and they exist for a real need: orchestrating complex multi-step AI applications, with retrieval pipelines, memory stores, chains of calls, whole fleets of agents.

Now look at what a CLI tool actually needs from an LLM. It needs to send a prompt and get text back. Sometimes it wants structured data back instead of prose. Sometimes it wants to let the model call a few of the tool’s own functions. That’s pretty much the whole list.

Pulling in a framework built to orchestrate retrieval and agent swarms in order to do that is a poor trade. You take on a large new vocabulary of concepts, a wide dependency surface, and a great deal of abstraction you’ll never touch, all to perform three or four operations. The framework isn’t wrong. It’s just answering a far bigger question than the one a CLI tool is asking.

What go-tool-base chose instead

go-tool-base didn’t reach for a framework. The decision is on the record in its own design notes: before a single line was written, LangChain Go, go-openai, Vercel’s AI SDK and around ten other options were evaluated, and not one of them matched what a CLI framework actually needs. So the chat package was built deliberately small.

How small? The entire core ChatClient interface is four methods:

type ChatClient interface {
 Add(ctx context.Context, prompt string) error
 Chat(ctx context.Context, prompt string) (string, error)
 Ask(ctx context.Context, question string, target any) error
 SetTools(tools []Tool) error
}

Add appends a message to the conversation. Chat sends a prompt and returns text. Ask sends a prompt and returns a typed Go struct, the model’s answer unmarshalled straight into a value you defined. SetTools hands the model a set of your own functions it’s allowed to call. That’s the whole surface. Downstream code that uses AI never holds anything larger than this, and never has to know which provider is behind it.

The package’s own documentation has a word for this: right-sized. Large enough to solve genuine provider-abstraction complexity, small enough that the full interface fits on a single screen.

“Thin” is not the same as “does little”

This is the part worth being precise about, because “four methods” can sound like “barely does anything”, and that’s the wrong read entirely.

Behind those four methods sits genuinely awkward work. Five providers (OpenAI, Claude, Gemini, a locally installed claude binary, and any OpenAI-compatible endpoint) each with a different wire API, all normalised behind the one interface. A tool-calling loop. Structured output via JSON Schema, made to behave consistently across providers that each express it differently. Error normalisation. Token chunking.

The point of a thin abstraction is not that there’s little underneath it. It’s that the interface stays small while the implementation quietly absorbs the complexity. Four methods on the surface; five provider integrations and a tool-calling loop below the waterline. The thinness is a property of what the caller sees, not of what the package does. A reach-for-LangChain decision gets that backwards: it exposes the caller to all the machinery, whether or not the caller will ever need it.

The core stays small even as features grow

There’s a neat detail in how chat keeps the interface from creeping. The package also supports streaming responses and conversation persistence, both of which are real features with real surface area. Neither of them is in the four-method core.

Instead they’re separate, optional interfaces. A streaming-capable client also satisfies StreamingChatClient; a persistable one also satisfies PersistentChatClient. Code that wants those capabilities does a type assertion to ask for them, and code that doesn’t simply never sees them. So the common path stays four methods forever. New capabilities arrive as opt-in interfaces alongside the core, not as new methods bolted onto it. The thing that fits on one screen keeps fitting on one screen.

Extensible without forking, testable without a network

Two more properties keep the package small without making it limiting.

It’s extensible. The provider list isn’t closed. A RegisterProvider call lets any package contribute a new provider, and chat.New will route to it. You add a backend without forking pkg/chat or sending a patch upstream.

And it’s testable. The package ships generated mocks. A downstream tool’s AI features can be tested against a mock ChatClient returning canned responses, with no network, no API key, and no flakiness. Because the interface is four methods, that mock is trivial to set up and complete by construction. A sprawling framework interface is a sprawling thing to fake; a four-method one is not. (I’ll come back to testing AI code properly in a later post, because it deserves a whole article of its own.)

The right size

When a CLI tool needs AI, the instinct is a large framework like LangChain. For orchestrating retrieval pipelines and agent swarms, that’s exactly the right tool. For sending a prompt, getting a struct back, and letting the model call a few functions, it’s enormous overkill.

go-tool-base’s chat package is the deliberate alternative, chosen only after LangChain Go and a dozen others were weighed up and rejected. Its core ChatClient interface is four methods. Underneath sit five normalised providers, a tool-calling loop, structured output and error handling, but the caller sees four methods and never learns which provider is active. Streaming and persistence are opt-in interfaces beside the core, not additions to it. It extends without forking and tests without a network. Right-sized: the complexity is real, but it lives under the interface rather than in it.

Letting the AI call your Go functions

Sun, 29 Mar 2026 00:00:00 +0000

An AI that can only produce text can describe your system. An AI that can call your Go functions can actually operate it. That gap, between describing and doing, is the difference between a chatbot and something genuinely useful, and crossing it comes down to one fiddly mechanism: tool-calling, and the loop that drives it.

Talking about the system versus operating it

Wire an AI provider into a CLI command and you get something that can talk. Ask it a question, get a paragraph back. Useful, up to a point.

But notice the ceiling. An AI that can only generate text can describe things. It can tell you what it would do. What it can’t do is look at the actual current state of your system, or take a real action, because it has no hands. It’s reasoning in a vacuum about a world it can’t reach out and touch.

The thing that gives it hands is tool-calling. You hand the AI a set of functions it’s allowed to call. Now, mid-conversation, it can decide it needs to read that file before it can answer, or run that query, or check that status, and actually go and do it, and then reason about the real result. The AI stops describing your system and starts operating it.

The loop is the hard part

Tool-calling has a shape, and the shape is a loop. The literature calls it ReAct: Reason, Act, Observe.

The AI reasons about the prompt and decides whether it needs a tool.
If it does, it acts, asking for a specific tool with specific arguments.
Your code runs the tool and feeds the result back. The AI observes that result.
Round again. Reason about the new information, maybe call another tool, maybe several. Keep going until the AI has what it needs and produces a final text answer with no more tool calls.

Conceptually simple. Tedious and error-prone to implement by hand every single time: parsing the model’s tool-call requests, dispatching to the right function, marshalling arguments in and results out, feeding observations back in the exact format the provider expects, knowing when to stop, and not looping forever if the model gets itself stuck.

That orchestration is pure plumbing, and it’s identical for every tool and every command. So you can probably guess what’s coming: go-tool-base’s chat package owns it. You don’t write the loop. You write the tools.

Defining a tool

A chat.Tool is four things: a name, a description, a parameter schema, and a handler. The description is what the AI reads to decide whether to use the tool, so it’s worth writing well. The schema describes the arguments, and you don’t hand-write it. You write a tagged Go struct and let it generate:

type ReadFileParams struct {
 Path string `json:"path" jsonschema_description:"Relative path to the file"`
}

The struct is the contract. The framework derives the JSON Schema the AI is given straight from those tags, so the schema and the Go type the handler receives can’t drift apart, because they share a single source. The handler is then just an ordinary Go function that takes those parameters and returns a result.

You register your tools with SetTools, call Chat, and that’s the whole of your involvement. The framework runs the ReAct loop and Chat returns the AI’s final text answer once the loop settles.

Two details that show it was built for real use

A couple of decisions in the loop tell you it’s meant for production, not a demo.

Tool errors don’t abort the conversation. When a handler returns an error, the framework doesn’t crash the loop. It hands the error back to the AI as a string, as just another observation. That’s deliberate, and it’s right. A real agent should be able to call a tool, watch it fail, and react: try different arguments, take a different route, or tell the user it couldn’t manage it. A loop that aborted on the first tool error would be far more brittle than the model driving it.

The loop is bounded. There’s a MaxSteps limit, default 20. An AI that gets confused could otherwise call tools forever, and a CLI command that never returns is a worse failure than a wrong answer. The cap guarantees the command terminates. The agent gets room to genuinely work a problem across many steps, but not infinite room to flail about in.

There’s also parallel tool execution: when the model asks for several tools in a single step (three independent file reads, say) the framework runs them concurrently rather than one after another, because there’s no reason to make the AI sit and wait out a sequence of things that don’t depend on each other.

Boiling it down

A text-only AI can describe your system; an AI that can call your functions can operate it. Bridging that gap means tool-calling, and tool-calling means the ReAct loop (reason, act, observe, repeat) whose orchestration is fiddly, identical every time, and not a problem worth solving twice.

go-tool-base’s chat package runs the loop for you. You define chat.Tool values (name, description, a tagged parameter struct that generates its own schema, a handler), call SetTools and Chat, and get the final answer. Tool errors go back to the AI as observations so it can recover, and a MaxSteps cap guarantees the command always terminates. You write Go functions. The framework turns them into things an agent can reach for.

Stop regex-ing the LLM's prose

Tue, 31 Mar 2026 00:00:00 +0000

Ask an LLM a question and it hands you back prose. Lovely to read, miserable to program against. You wanted the one number buried in the middle of it, and now you’re writing a regular expression to fish a word out of three well-written paragraphs that phrase themselves slightly differently every single time you run them.

There’s a much better way, and it’s the difference between forever interpreting an LLM and actually building on one.

The problem with a paragraph

You ask an LLM to analyse a log file and tell you the severity of what it found and a suggested fix. It comes back with three well-written paragraphs. Somewhere in there is the word “critical”, and somewhere is the fix.

Your program now has to extract those two facts from prose, and prose has no contract. The next run, the model phrases it differently. It leads with a caveat. It says “severe” where last time it said “critical”. It puts the fix first. Anything that worked by finding “critical” in the text is now quietly wrong, and you didn’t change a line. Parsing free text for structured facts is a game you lose slowly.

What you actually wanted was never a paragraph. It was a value: a thing with a severity field and a fix field, that you can branch on and store and pass around like any other.

Ask for the struct, not the prose

go-tool-base’s chat package draws the line with two methods. Chat gives you text. Ask gives you a struct.

You define the Go type you want back:

type Analysis struct {
 Severity string `json:"severity"`
 Fix string `json:"fix"`
}

var result Analysis
err := client.Ask(ctx, "Analyse this log file: "+logText, &result)

The framework generates a JSON Schema from that struct, sends it to the model as the required response format, and unmarshals the reply straight into result. You never lay a finger on the prose. You get result.Severity and result.Fix, typed, ready to use. If you want the model’s answer to drive a switch statement, this is the method that lets it.

The struct is the schema is the contract

The detail that makes this hold up over time: you don’t write the schema. The struct is the schema.

The framework derives the JSON Schema from your type. In go-tool-base that’s GenerateSchema[T](); in rust-tool-base the schema comes from your Rust type through schemars. (Yes, there’s a Rust sibling now. I’ll introduce it properly in a few weeks, but it keeps gatecrashing these posts because the two frameworks deliberately share ideas.) Either way there’s one definition, your type, and the schema is just a projection of it.

That matters, because otherwise two things have to agree. There’s the schema you tell the model to obey, and there’s the type you unmarshal the answer into. Hand-write the schema and those two can drift: add a field to the struct, forget to add it to the schema, and the model is never told to produce it, so it silently never appears. Deriving the schema from the type collapses the two into one. They can’t disagree, because there’s only one of them.

Both frameworks, with one extra step in Rust

go-tool-base does this with Ask and a ResponseSchema set on the client config. rust-tool-base does it with chat_structured::<T>, where T is any type that’s both deserialisable and JsonSchema.

rust-tool-base adds one step worth calling out. Before it deserialises the model’s reply into your T, it validates the raw response against the schema with a JSON Schema validator. That splits the failure into two distinct, named cases: the response didn’t match the schema, or it matched the schema but still wouldn’t deserialise. A model that returns subtly wrong JSON fails loudly and specifically, with an error that tells you which of those happened, instead of quietly handing you a zero-valued struct that you end up debugging an hour later.

When you’d reach for it

The line is simple, and it’s about who reads the answer.

If a human reads the answer, prose is right. Chat, free text, let the model write well. A summary, an explanation, an interactive reply: leave all of those as prose.

If a program consumes the answer, you want a value. Classification, extraction, a code review scored out of a hundred with a list of issues, a yes-or-no with reasons: anything where the next thing that happens is your code branching on the result. There, Ask and chat_structured turn the LLM from something you have to interpret into something that returns a value, and a typed value is a thing you can actually build on.

To sum up

An LLM returns prose by default, and prose has no contract, so a program that picks structured facts out of it breaks the moment the model rephrases.

Structured output asks for the value instead. You define a struct, the framework derives a JSON Schema from it, the model is constrained to that shape, and you get a typed result. go-tool-base’s Ask and rust-tool-base’s chat_structured both work this way, with the schema derived from your type so the schema and the type can’t drift; rust-tool-base additionally validates the response against the schema before deserialising. Use it whenever the answer feeds code rather than a human. It’s one of the four methods that make up go-tool-base’s small chat interface, and it’s the one that makes an LLM safe to program against.

AI conversations you can resume

Sat, 04 Apr 2026 00:00:00 +0000

An AI conversation is, fundamentally, its own history. The model’s next answer depends on everything said so far. And a CLI tool, by its very nature, forgets everything the moment it exits. Put those two facts together and you get the problem: run an AI command, exit, run it again, and you’re talking to someone who’s never met you.

A CLI forgets everything

A long-running service keeps its state in memory for as long as it runs. A CLI tool doesn’t get that luxury. It starts, does one thing, exits. The next invocation is a brand-new process with no memory of the last one.

For most commands that’s exactly right, and you wouldn’t want it any other way. But an AI conversation is a different kind of beast, because a conversation is its history. The model’s next answer depends on everything said so far. Run an AI command, exit, run it again, and you’ve started a fresh conversation with someone who’s never met you. For an interactive assistant, or any AI workflow that unfolds across several invocations, that’s plainly the wrong behaviour. The user expects to pick up where they left off.

Save and restore

The chat package handles this through a PersistentChatClient interface. Like streaming, it’s an optional capability discovered with a type assertion, sitting beside the four-method core rather than bloating it. A client that supports persistence also satisfies this interface:

if pc, ok := client.(chat.PersistentChatClient); ok {
 snapshot, err := pc.Save()
 // store the snapshot somewhere
}

A snapshot is a serialisable value that captures the conversation. You store it. Next run, you load it, Restore it onto a fresh client, re-register your tools, and call Chat again. “Where were we?” works, because the model is handed back the whole history.

A snapshot is opinionated about what it carries

The interesting part is what a snapshot does and doesn’t contain, because that’s a series of deliberate decisions.

It carries the messages, the system prompt, the model name, and tool metadata: the names, descriptions and parameter schemas of the tools that were registered.

It does not carry tool handlers. Handlers are code, not data; you can’t serialise a function meaningfully, so after a restore you re-register them with SetTools. The snapshot remembers that a tool called read_file existed and what its shape was; it doesn’t try to remember the Go function behind it.

And it does not carry API tokens. This is the one to dwell on. A snapshot is a file. A file gets synced, backed up, copied between machines, attached to a support ticket by a user trying to be helpful. A snapshot that carried the API key would be a credential leak the moment it left the laptop it was made on. So the snapshot never contains a token, at all. On restore, the client picks the credential up again the ordinary way, from the environment or the keychain. The conversation and the secret are kept in separate places on purpose, and only one of them is ever in the file.

Encrypted at rest, if you want it

The package ships a FileStore that writes snapshots as JSON files, with 0600 permissions in a 0700 directory, and it can encrypt them. Pass WithEncryption a 32-byte key and snapshots are written with AES-256-GCM.

That option exists because a conversation can hold sensitive content even when it holds no credential. The log a user pasted in for analysis, the source file they asked the model to review, the internal details tucked into their questions: none of that is an API key, and all of it might be something you’d rather not have sitting in plain JSON in a backup somewhere. Encryption at rest covers it.

The FileStore is also careful about the snapshot identifiers it’s handed. An ID has to be a canonical UUID, and the resolved file path is checked to lie inside the store directory, so a snapshot ID arriving from an untrusted source (a CLI flag, a request payload) can’t be bent into a path-traversal that reads or writes somewhere it shouldn’t. Persisting conversations adds a small filesystem surface, and the store treats it as exactly that.

The short version

A CLI tool forgets everything between invocations, which is correct for most commands and wrong for an AI conversation, because a conversation is its history.

go-tool-base’s chat package lets you persist one. PersistentChatClient saves a snapshot you can store and restore later, picking the conversation back up where it ended. The snapshot is deliberate about its contents: messages, system prompt and tool metadata yes; tool handlers no, because they’re code you re-register; API tokens never, because a snapshot is a file and a file travels. The built-in FileStore can encrypt snapshots at rest with AES-256-GCM and validates snapshot IDs against path traversal. Resumable conversations, without the conversation file turning into a place secrets leak from.

Nobody reads the manual

Sun, 29 Mar 2026 00:00:00 +0000

Let me describe the actual lifecycle of a user meeting your CLI tool, because it’s a bit humbling. They run it. It doesn’t quite do what they expected. They run it again with --help. They get a wall of monospaced flag descriptions, skim it, don’t find the thing they wanted, and either give up or go and ask a human who already knows.

Your documentation might be magnificent. It doesn’t matter, because the user never reached it.

The manual loses on location, not quality

That’s the lifecycle, and notice exactly where it breaks. The documentation might be excellent. It might answer their precise question in full. It doesn’t matter, because it’s on a website, in another window, behind a search box, and the user is here, in the terminal, mid-task. The docs lost not on quality but on location. They simply weren’t where the work was.

go-tool-base’s answer starts with a decision about location: the documentation gets embedded into the binary itself. Your docs/ folder ships inside the tool, the same way its default config does. Wherever the tool is installed, the docs are right there alongside it, no network, no browser. That embedding is what makes everything else possible, and there are two things built on top of it.

A browser, in the terminal

The first is the docs command, and it’s not --help with extra steps. It launches a proper Terminal User Interface, built on Bubble Tea.

It has a sidebar, structured from the project’s own mkdocs.yml, so the docs are a navigable tree rather than one flat scroll. Markdown renders with real formatting through Glamour (colour, tables, lists, headings) instead of collapsing into monospaced soup. There’s live search across every page, regex included.

Compared with man and --help, the difference isn’t a nicer coat of paint. man gives you linear scrolling and grep; this gives you a structured tree, rich rendering and real search. It’s the documentation experience a modern developer expects, except it followed the tool into the terminal instead of demanding the user leave it.

A documentation assistant that won’t make things up

The second thing built on the embedded docs is the one I find genuinely transformative: docs ask.

The user doesn’t navigate anything. They just ask:

mytool docs ask "how do I point this at a self-hosted server?"

and get a direct, specific answer. Under the hood, the framework collates the tool’s embedded markdown and hands it to the configured AI provider (Claude, OpenAI, Gemini, Claude Local, any OpenAI-compatible endpoint) as the context for the question.

Now, “an AI answers questions about my tool” should immediately make you nervous, and the correct thing to be nervous about is hallucination. An AI that confidently invents a flag that doesn’t exist, or describes behaviour the tool simply doesn’t have, is worse than no assistant at all, because the user trusts it.

This is where embedding the docs pays off a second time, and it’s why I keep stressing that the corpus is closed. The model is instructed to answer only from the tool’s actual documentation, and the context it’s handed is exactly that documentation and nothing else. It isn’t drawing on a vague memory of similar tools from its training data. It’s answering from this tool’s real, shipped, version-matched docs. The corpus is small, closed and authoritative, which is the combination that keeps the answers honest. “Zero hallucination by design” isn’t a slogan about the model. It’s a property of bounding what the model is allowed to look at, which is the same instinct I leaned on with the mcp command: the safety comes from the boundary you drew, not from trusting the AI to behave itself.

There’s a nice second-order effect, too. The answer is always about the version of the tool the user actually has, because the docs were embedded into that build. No mismatch between a website documenting the latest release and the slightly older binary sitting on the user’s machine.

The upshot

Documentation usually loses to --help not on quality but on location: it’s in a browser, and the user is in the terminal. go-tool-base embeds the docs into the binary and surfaces them two ways: a docs command that’s a real TUI browser with a sidebar, rich markdown and search, and docs ask, which answers natural-language questions using the embedded docs as context.

Because that context is the tool’s own closed, shipped documentation and the model is told to use nothing else, the assistant stays grounded, and it’s always describing the exact version the user is holding. The fix for unread documentation was never to write more of it. It was to put it where the work happens and let it answer back.

The AI provider that isn't an API

Mon, 06 Apr 2026 00:00:00 +0000

go-tool-base’s chat package puts five AI providers behind one interface. Four of them are exactly what you’d guess: HTTP calls to OpenAI, Claude, Gemini, and anything OpenAI-compatible. The fifth one isn’t an API at all. It shells out to a binary.

That sounds like a slightly mad thing to want, right up until you’ve worked somewhere the network says no.

The fifth provider shells out

The chat package speaks to five providers through one ChatClient interface. Four of them are what you’d expect: HTTP requests to OpenAI, to Claude, to Gemini, to any OpenAI-compatible endpoint. The tool author picks one in config, and the rest of the code never knows the difference.

The fifth, ProviderClaudeLocal, is different in kind. It doesn’t make an HTTP request at all. It shells out. It runs the claude CLI binary as a child process, passes the prompt in, and reads the answer back from the binary’s output.

That sounds like an odd thing to want until you’ve been stuck in the environment it was built for.

Why you’d want that

Picture a corporate network with its egress locked right down. Outbound HTTPS to api.anthropic.com is blocked by policy. A tool built on go-tool-base that uses AI would simply fall over there. It tries to reach the API, there’s no route, and that’s the end of the feature.

But the developer at that machine has the claude CLI installed, and has run claude login. That binary is permitted. It’s an approved, managed tool, and it has its own sanctioned path out. The direct API call is blocked; the claude command is not.

ProviderClaudeLocal is what bridges those two facts. If your tool’s AI calls go through that already-blessed binary instead of straight at the API, they work, in an environment where the direct call cannot. That’s the whole reason the provider exists. It isn’t faster (a real API call has lower latency) and it isn’t more capable. It’s for the place where the API call simply isn’t an option, and “isn’t an option” is a surprisingly common place to find yourself inside a large organisation.

What it costs

It’s worth being straight about the trade, because ProviderClaudeLocal is the reduced-capability provider.

It doesn’t do tool calling. It doesn’t do parallel tools. It doesn’t stream. Those need a live, structured connection to the model’s API, and a subprocess that runs once and prints an answer is not that. What it does support is plain chat and structured output, the latter through the binary’s own --json-schema flag.

So the positioning, and the package’s documentation says exactly this, is: prefer the API providers when you can reach them, because they’re lower latency and feature-complete. Reach for ProviderClaudeLocal when API access is restricted. You accept the narrower capability set as the price of working at all. For a tool whose AI feature is “answer a question” or “return a structured analysis”, that price is often nothing you’d even notice. For one built on an agentic tool-calling loop, it’s a real limitation, and you’d know to expect it.

How it stays behind the same interface

Here’s the part that makes it pleasant rather than a special case to maintain. Despite being a subprocess and not an API, ProviderClaudeLocal is still a ChatClient. Your feature code calls Chat and Ask exactly the way it would for any other provider.

Everything that makes a subprocess provider awkward stays inside the provider. Spawning the binary, feeding it the prompt, parsing its output, capturing stderr and surfacing it when the binary exits non-zero, and threading multi-turn continuity through session identifiers passed back on the next call with --resume: all of that is the provider’s problem, and all of it sits behind the interface. The code in your tool that uses AI doesn’t know, and has no way to find out, that this particular provider is a child process rather than an HTTPS call.

That’s a unified interface genuinely earning its place. It’s easy to put a uniform face on four things that already work the same way underneath. The real test of the abstraction is whether something that works in a completely different way, a subprocess instead of a socket, can still slot in without the caller changing a line. Here it can. You swap one config value, and a tool that talked to an API now talks through a binary, and nothing downstream so much as blinks.

The bottom line

go-tool-base’s chat package puts five providers behind one ChatClient interface, and ProviderClaudeLocal is the one that isn’t an API. It runs the locally installed, pre-authenticated claude CLI as a subprocess.

It exists for the locked-down environment where outbound HTTPS to the AI API is blocked but the claude binary is allowed: there, AI features keep working where a direct call would fail. The trade is a narrower capability set (no tool calling, no streaming, plain chat and structured output only) so you prefer the API providers when you can reach them and fall back to this when you can’t. And because it’s still a ChatClient, all the subprocess machinery stays hidden, and your code uses it without knowing it’s there. That last part is the real test of an abstraction: a provider that works in an entirely different way still slots in unchanged.

Supporting a provider, or actually using it

Sat, 02 May 2026 00:00:00 +0000

If your CLI tool talks to an AI model, you don’t want to hard-wire one vendor. So you reach for a single client interface over several providers, which is the right call. The trap is the next step: build that interface on only what every provider has in common, and you quietly throw away the very features that made you want a particular provider in the first place. rust-tool-base’s rtb-ai refuses to make that trade.

The pull toward one interface

If your CLI tool talks to an AI model, hard-wiring one vendor is a poor bet. One user has an Anthropic key, another an OpenAI key. Someone’s on Gemini. Someone runs Ollama locally because their data can’t leave the building. Someone points at an OpenAI-compatible endpoint from a provider you’ve never heard of. You don’t want a separate code path for each, so you want one AiClient that all of them slot behind.

rtb-ai gets that unification from the genai crate, which already speaks to Anthropic, OpenAI, Gemini, Ollama and OpenAI-compatible endpoints. One interface, five providers, the tool author picks one in config. The Go sibling makes the same bet: go-tool-base’s chat package also unifies several providers, behind an interface deliberately kept to four methods. So far this is the obvious design, and if it were the whole design there’d be nothing to write about.

What “unified” quietly costs you

Here’s the catch in any unified interface. It can only expose what every provider behind it has in common.

The common subset is plain chat. Messages go in, text comes out, optionally streamed token by token. That’s real and it’s useful and every provider does it. But the common subset is also the floor, and the features that make a particular provider worth choosing are almost never on the floor. They’re the things only that provider does.

Anthropic is the sharp example, because it has three features that matter and not one of them is common-subset.

Prompt caching. You can mark the stable parts of a request, the system prompt and the tool list, as cacheable. The provider keeps them warm, and on the next turn you aren’t billed to re-send and re-process text that didn’t change. On a long agent loop, where the same large system prompt rides along on every single turn, that’s a substantial saving in both cost and latency.

Extended thinking. The model works through a hard problem in a visible, budgeted reasoning pass before it commits to an answer, and you can see that reasoning.

Citations. Structured references back to source material in the response.

A client built strictly on the common subset can’t express any of those. It has no field for them, because four of the five providers wouldn’t know what to do with the field. So a purely lowest-common-denominator client would “support” Anthropic and then use it badly, leaving its best features unreachable. Support as a checkbox, not as the point.

The escape hatch

rtb-ai’s answer is to not choose. It runs two implementations under one interface.

For OpenAI, Gemini, Ollama and OpenAI-compatible endpoints, calls route through genai, the unified path. For Anthropic, every method drops to a direct reqwest implementation straight against the Messages API. Same AiClient on the surface, a different implementation underneath, selected by which provider the config names.

And the request type has deliberate room for the difference:

pub struct ChatRequest {
 pub system: Option<String>,
 pub messages: Vec<Message>,
 pub temperature: Option<f32>,
 pub max_tokens: Option<u32>,
 /// Anthropic-only: enables prompt caching at every stable point.
 /// Ignored on non-Anthropic providers.
 pub cache_control: bool,
 /// Anthropic-only: extended-thinking budget. `None` disables.
 /// Ignored on non-Anthropic providers.
 pub thinking: Option<ThinkingMode>,
}

Set cache_control and the Anthropic-direct path inserts cache breakpoints at the three stable points: the system prompt, the tool list, and the first message. Set thinking and it adds the thinking block, and streaming surfaces a separate ThinkingToken event so you can show the reasoning apart from the answer. On a non-Anthropic provider, both fields are simply ignored. The interface carries them; only the implementation that understands them acts on them.

A hatch, not a leak

It’s worth being precise about why this isn’t the thing it superficially resembles, which is a leaky abstraction.

A leaky abstraction is one where implementation details bleed through that you didn’t intend and can’t reason about. The abstraction quietly fails to abstract, and you’re left guessing which provider you’re really talking to.

This is the opposite of that. The two Anthropic-only fields aren’t a leak. They’re named, documented as Anthropic-only, inert everywhere else, and right there in the public type for anyone to see. The interface is uniform for the common case and deliberately, visibly non-uniform at exactly the points where uniformity would have cost you the good features. You opt into provider-specifics by setting a field. You stay fully portable by leaving it at its default. Nothing bleeds; you decide.

The same design line explains what does stay in the unified path. Structured output, chat_structured::<T>, sends a JSON Schema derived from your Rust type with the request and validates the reply against it before handing you a typed T. That’s a portability win that costs nothing across providers, so it belongs in the common interface. The split isn’t “Anthropic versus the rest”. It’s “features that are free to unify go in the unified path; features that aren’t get a designed door”. Prompt caching and extended thinking get the door, because flattening them away would be the expensive kind of convenient.

To sum up

A CLI tool that integrates AI wants one client over several providers, and a unified interface can only expose what those providers share. The shared floor is plain chat, and the features worth choosing a provider for, like Anthropic’s prompt caching, extended thinking and citations, are never on the floor.

rtb-ai keeps both. genai provides the unified path across five providers; an Anthropic-direct reqwest path drops below the abstraction for the features genai can’t reach, and ChatRequest carries the Anthropic-only fields openly, ignored elsewhere. Uniform where uniformity is free, with a designed escape hatch where it isn’t. That’s the difference between supporting a provider and actually using it.

A configurable AI endpoint is an attack surface

Sun, 19 Apr 2026 00:00:00 +0000

“Let users point at their own AI endpoint” is one of those config options that looks completely harmless on the way in. People want it, for perfectly good reasons. Then you sit with it for a minute and realise you’ve handed every user a loaded gun and pointed it vaguely at their own API key.

Why you offer it at all

There are real reasons to let someone set a custom base URL. They’re running a local model and want localhost:11434. They’re behind a corporate proxy that fronts the real provider. They’re on Azure’s flavour of OpenAI, which lives at a different host. They’ve a self-hosted gateway doing rate-limiting. All reasonable, all things a framework should support rather than fight.

The bit that’s a loaded gun

Here’s what the config option quietly decides: the base URL is where your credential goes. The API key rides along in an Authorization header on every request, to whatever host that URL resolves to. So the moment the endpoint is user-configurable, the destination of your secret is user-configurable too.

And users do user things. They paste a URL from a gist that turned out to be a honeypot. They leave http:// on the front, so the key crosses the wire in plaintext. They copy https://user:token@host/v1 not realising the userinfo changes who they actually authenticate to. They never edit the https://api.example.com/v1 placeholder and wonder why the key’s been posted to a domain they don’t own. None of that is malice. It’s what happens when the destination of a secret is a free-text field.

Validate before the first byte leaves

So every chat.New routes through ValidateBaseURL before the provider is built. The threat model is written at the top of pkg/chat/baseurl.go: an operator who can influence config could “redirect chat-provider traffic to an attacker-controlled HTTPS host and capture the Authorization header.” The checks run cheapest-first: a length cap, no ASCII control characters, must parse, no userinfo, https only, a host must be present, and the host mustn’t be a placeholder.

The userinfo rule is the sharp one:

if parsed.User != nil {
	// Reject any userinfo, with or without password. Never log
	// the URL itself because it contains the credential.
	return errors.WithHint(ErrInvalidBaseURL,
		"base URL must not contain credentials; use the Token field instead")
}

The placeholder check rejects example.com and friends and any subdomain of them, so the unedited https://api.example.com/v1 from a setup wizard never reaches the wire and hits some typosquatted lookalike. And the HTTP escape hatch is test-only by construction: the AllowInsecureBaseURL field that permits plain http is tagged json:"-", so a config file physically cannot set it. This all came out of the 2026-04-17 security audit, finding M-3.

rust-tool-base enforces the same at its own boundary: validate_base_url rejects userinfo, any scheme but https (bar a test-only allow_insecure), and documentation placeholder hosts like example.com.

What it can and can’t do

It won’t stop a user who deliberately points the tool at a malicious HTTPS host they genuinely chose. If someone is set on sending their own key somewhere bad, validation can’t read their mind.

What it stops is the accidents: the plaintext slip, the userinfo confusion, the placeholder nobody changed. Those aren’t theoretical, they’re the ones that happen to careful people on ordinary days. Storing the key well is one job (where a CLI keeps it), stopping it leaking through a log is another, and this is the third side of the triangle: once you’ve stored it and stopped it leaking, make sure you don’t send it somewhere daft.

An AI agent that has to make the build pass

Thu, 02 Apr 2026 00:00:00 +0000

Most AI code generation works on a charming little principle I’ll call generate-and-hope. The model writes the code, the model stops at the closing brace, and whether the thing actually compiles is left as an exercise for you. For a snippet you paste into an editor, fine. For a whole generated command, that’s just outsourcing the disappointment.

go-tool-base does something I’m rather happier with: the AI has to make the build pass before it’s allowed to claim it’s done.

Generate and hope

The usual shape of AI code generation is this. You ask for code, the model produces it, and the model’s job ends at the closing brace. Whether it compiles, whether the tests pass, whether the imports even resolve, none of that has been checked. The model produced something that looks right. You find out whether it is right when you build it.

For a snippet you paste into an editor, that’s perfectly fine. The compiler tells you in a second. But go-tool-base’s generator, driven by gtb generate command --script or --prompt, produces a whole command: the implementation, its tests, the lot. “Generate and hope” at that scale means handing the user a project that may or may not build, and quietly making them the one who finds out which.

Drafting is only step one

So the generator doesn’t stop at drafting. Writing the first version of the implementation and its tests is step one of two. Step two is an autonomous repair agent.

Once the draft is on the filesystem, a separate agent takes over. It’s an LLM running in a loop, but a loop aimed at one narrow, checkable job: make this project build and pass its tests. It isn’t asked to be creative. It’s asked to get to green.

A fixed set of tools, and no shell

The agent is not handed a shell. It’s given a fixed, defined set of tools and nothing else. Three of them let it explore and edit the project: list_dir, read_file, write_file. Four of them let it verify the project:

go_build runs the build and captures the compiler errors.
go_test runs the tests and captures the failures.
go_get resolves a missing dependency.
golangci_lint runs the project’s linter.

That restriction is the design, not a limitation of it. The agent can’t delete arbitrary files, can’t reach the network, can’t run anything that isn’t on the list. It has exactly what it needs to make code compile and nothing it would need to do damage. Its file writes are confined to the project directory by an explicit path check, so even write_file can’t go wandering up into /etc. A coding agent you’d actually let near a filesystem is one whose abilities are an allowlist, not a denylist. (I keep coming back to that principle through this series… safety as a boundary you draw, not a behaviour you hope for.)

The loop

The repair loop is a ReAct loop, the same reason-act-observe shape as the tool-calling loop, only this time pointed at a goal:

The draft is on disk.
Verify: run go_build and go_test.
If verification failed, read the error logs, the compiler error or the failing test.
Reason about the cause: an undefined variable, a missing import, a wrong signature.
Act: call write_file to patch the code, or go_get to add the dependency.
Loop. Steps two to five repeat until the project is green, or the agent hits its bounded step limit.

What makes this work is treating the error output as feedback rather than as a failure to log and walk away from. A compiler error is the single most useful sentence you can hand a model that’s trying to fix code. It says what’s wrong, and usually where. The loop feeds it straight back in, and the model fixes against it.

Verification changes what “done” means

Here’s the real shift, and the agent’s own documentation puts it well: the agent “doesn’t just say it fixed a bug; it uses a Test tool to verify the fix before reporting success.”

A generate-and-hope model reports success when it finishes writing. It has no idea whether the code works, and it isn’t really claiming otherwise. “Done” means “I produced text”. The repair agent reports success when go_build and go_test actually pass. “Done” means “the build is green”. Those are two completely different claims, and only the second is worth anything to the person who asked for the command.

That’s the line between an AI that’s a creative writer and an AI that’s a collaborator you can hand a task to. And when the agent can’t reach green, when it spends its whole step budget and the project is still broken, the generator fails safely: it leaves the best-attempt code in place, commented out so the project still compiles, and tells the user what to finish by hand. There’s also an --agentless flag for anyone who’d rather have a plain single-shot retry than the multi-step agent. The default, though, is the agent, because the default should be code that’s been checked.

Where this leaves us

Most AI code generation generates and hopes: the model writes code and the user discovers whether it works. For a whole generated command, that pushes a may-or-may-not-build project onto the user.

go-tool-base’s generator drafts the command and then hands it to an autonomous repair agent. The agent has a fixed set of tools (explore and edit the project, build it, test it, lint it, fetch dependencies) and no shell at all, with file writes confined to the project directory. It runs a ReAct loop, reading each error and patching against it, until the build is green or it exhausts its steps. The point is what “done” comes to mean: not “the model finished writing”, but “the build passes”. Only one of those is a claim worth trusting.

The agent said SUCCESS. The linter disagreed.

Fri, 26 Jun 2026 00:00:00 +0000

There’s a repair agent inside go-tool-base now. When you run gtb generate command, it doesn’t just spit out a file and wish you luck. An agent takes the generated code, builds it, runs the tests, and fixes whatever it broke, looping until the thing actually works (or until it’s tried the same fix five times and admits defeat). The whole point is that the generator hands you code that’s ready, not code that’s nearly ready and quietly now your problem.

So it stung a bit when I realised the agent had been holding itself to a lower bar than I’d hold any junior to. And I was the one who’d set the bar.

What “done” meant to the agent

The agent is a loop with real tools: it can build, test, read files, write files, tidy the module, and run golangci-lint. It works through them, and when it’s happy it replies with the word “SUCCESS” and the loop stops. On the Go side, the check is exactly that blunt:

if strings.Contains(strings.ToUpper(resp), "SUCCESS") {
 return nil
}

That’s the whole gate (agent.go). There’s no clever verification on my end that the agent actually did its homework. It does the work, it tells me it’s done, and I believe it. Which is fine, as long as the agent and I agree on what “done” means.

We didn’t.

The instruction that made lint optional

The agent decides it’s finished by following a numbered list in its system prompt. Here’s the line that did the damage:

If there are lint issues, use ‘golangci_lint’.

Read that the way the agent would. “If there are lint issues”… well, how would it know? The only way to find out is to run golangci-lint. But the instruction makes running golangci-lint the thing you do once you already know there are issues. It’s a chicken with no egg. And the SUCCESS condition at the bottom of the list never mentioned lint at all:

When the project builds successfully and tests pass, reply with “SUCCESS”.

So the agent did the sensible thing, given its orders. It built the code, ran the tests, saw both go green, and declared victory. golangci-lint was sat right there in its toolbox, unused, because nothing ever told it the job wasn’t finished until lint was clean too. I’d handed it a linter and then written a prompt that let it walk straight past it.

The galling part is that the linter was never the missing piece. The golangci_lint tool had been registered the whole time, and it even runs with --fix, so it’ll quietly clear the trivial stuff and only surface what actually needs a decision. The capability was there. The instructions just never required it.

The fix was words, not code

Here’s the part I find genuinely interesting. I didn’t add a check. There is no new gate in the Go. The fix is four lines of English:

Run ‘go_build’, ‘go_test’ and ‘golangci_lint’ in the project directory… Run all three; a clean build and passing tests do not imply clean lint.

Reply with “SUCCESS” only once ‘go_build’, ‘go_test’ AND ‘golangci_lint’ all pass with no errors and no reported issues.

That’s it. Lint moves from a remediation step you reach for once you somehow already know there’s a problem, into the gate itself. “Done” now means three green lights, not two.

It nags at me a little, that one. The reliability of an agent that writes and fixes real code came down to whether one sentence of instructions was precise enough. When your success criteria are a paragraph of prose, vagueness in that paragraph is a bug, the same as a vague type or an off-by-one. The spec just happens to be written in English, and the thing reading it is a language model that will cheerfully take the cheap reading if you leave it lying around. That’s the same lesson the goblin who wouldn’t stay dead taught me from the other direction: with these tools, what you say is what you get, and what you don’t say is fair game.

Leave it better, not just building

The Boy Scout Rule is the whole reason this blog exists, and I’d quietly exempted the robot from it. “Leave the campsite cleaner than you found it” had become “leave it building”, which is not the same thing and never was. If I’m going to put an agent in the loop precisely so it tidies up after the generator, then “tidy” has to mean what it would mean for a person on my team. Build, test and lint. No walking past the bin because nobody told you to pick it up.

The afternoon the agent earned its keep

Thu, 09 Jul 2026 00:00:00 +0000

There’s a kind of bug that’s worse than a crash. A crash at least tells you where. This one just… stopped. I asked ffmpeg-wasi to remux a clip into a Matroska file, no re-encoding, just repackage the same streams into an .mkv, and it never came back. No error, no output. The test harness eventually gave up and shot it.

The same job to an .mp4? A dozen milliseconds. Change the file extension and a blink becomes a heat-death-of-the-universe operation.

Normally this is the moment a war story starts costing me evenings. FFmpeg compiled to wasm32-wasi, running under wazero, reading and writing through an in-memory filesystem… that’s a lot of layers, any of which could be the one that loops, and the traditional way to find out is to build test rigs for each layer by hand and lose a week to the plumbing. I know what that costs because I’ve paid it before.

This time I didn’t pay it. The hang surfaced while an agent was implementing stream-copy for me, so rather than pull the work back onto my own bench, I told it to chase the bug too. What follows is the deduction it ran. My contribution was two decisions at two forks, and I’ll point at them as we pass.

It got it wrong first

I want to start with the wrong turn, because it’s the most instructive part.

The agent’s first theory was confident and detailed: it traced the guest’s filesystem calls, saw the Matroska muxer seeking back and forth rewriting its header, and declared the finalisation seek-backs the culprit, an I/O pattern my filesystem bridge couldn’t satisfy. Plausible. Specific. Wrong.

And here’s the bit that sold me: it then designed the experiment that killed its own theory. Matroska has a streamable mode that writes without any seek-backs at all, so it flipped that on… and the job still hung. Theory dead, evidence in hand, no sulking. I’ve watched humans (fine, me) defend a pet theory for two days past its expiry date. The agent binned its own in minutes and moved on to the thing you should always do when guessing stops working: bisect.

Ruling out the world, one build at a time

This was my first fork. The agent asked whether to keep poking at it in place or to bring in native tooling, and I chose native. So it built a real FFmpeg, same n8.1.2 source, same minimal configure flags, and started eliminating suspects:

It tested	Result	Which ruled out
Native FFmpeg, 64-bit, real files → mkv	works	the muxer / our configure flags
wasm backed by a real OS directory	hangs	the in-memory filesystem
Native FFmpeg, 32-bit (`-m32`) → mkv	works	a 32-bit integer overflow
wasm under wazero’s interpreter	hangs	a code-generation bug

Sit with that table for a second. Two of those rows are whole native builds of FFmpeg, one of them 32-bit with the toolchain to match, spun up as casually as you’d write a unit test. That’s the part that would have eaten my week. Each row walks an interesting suspect free, until the only thing left holding the bag is the plainest fact in the whole setup: this is running under WASI. Not the muxer. Not the maths. Not my filesystem. Not the runtime. The environment.

The environment was missing a number

WASI is a smaller world than POSIX. No threads, no network, and, it turns out, no /dev/urandom. Most code never notices. Right up until it does.

With the bisection result in hand, the agent went and read the Matroska muxer’s source. Every Matroska track carries a unique random identifier, so at startup the muxer seeds a pseudo-random number generator via libavutil’s av_get_random_seed(). In a build configured as tightly as ours, that function has exactly one source of real entropy compiled in: reading /dev/urandom. The Windows crypto API, arc4random, OpenSSL, libgcrypt… all configured out. Just the one device.

Which WASI doesn’t have. So the read fails, and the function falls back to its last resort, get_generic_seed(): a genuinely clever routine that manufactures entropy out of thin air by reading clock() in a tight loop and harvesting the jitter in how it advances. It’s a lovely trick. It has one requirement. The clock has to advance.

Under WASI, clock() doesn’t advance. The seeding call never returns. The muxer hangs before it has written a single byte, and the .mp4 muxer never asks for randomness at all, which is the entire reason it was fine all along.

The agent didn’t leave that as a theory either. It wired av_get_random_seed() into a trivial diagnostic op, one that does nothing else… and that hung too. Root cause, demonstrated in isolation.

A device, not a patch

The fix isn’t a patch to FFmpeg. It’s a device. My filesystem bridge already fabricates a couple of the POSIX fixtures a guest expects (/tmp routed to scratch space, /dev/null as a discard sink), so /dev/urandom is the same trick with real randomness behind it, backed by Go’s crypto/rand:

// Read fills buf with random bytes; a short read never happens (crypto/rand
// fills the whole slice or errors), matching /dev/urandom's contract.
func (*randFile) Read(buf []byte) (int, experimentalsys.Errno) {
	if _, err := rand.Read(buf); err != nil {
		return 0, experimentalsys.EIO
	}

	return len(buf), 0
}

Now the muxer’s very first entropy read succeeds, the seed returns immediately, and the clever-but-doomed clock loop is never reached. The .mkv remux that had been hanging forever finishes in ten milliseconds. And the fix isn’t Matroska-specific: any format that wants a random seed now gets one for free.

What the afternoon actually cost

Here’s the arithmetic that made me want to write this up.

From the first observed hang to a verified fix was a little over an hour. That hour included the wrong theory and the experiment that disproved it, two complete native builds of FFmpeg (one of them 32-bit), runs across two wazero execution modes, a read through libavutil’s entropy code, the diagnostic op that proved the root cause in isolation, and the fix itself. The agent even went looking for prior art and couldn’t find anyone who’d written this failure up… a genuinely undocumented corner of WASI, run to ground between lunch and tea.

By hand? Most of a week, and the bulk of it spent on build plumbing rather than thinking: getting a matching native FFmpeg to configure, wrangling a 32-bit toolchain, wiring the trace points. None of that work is hard. It’s just time, and it’s exactly the kind of time a solo developer doesn’t have lying around, which is how deep bugs like this end up parked on a “someday” list with the .mkv button greyed out.

My whole part was two forks (dig in place, or go native? and later: document it before touching anything else) and reading the result. The judgement stayed mine. The legwork didn’t.

I’ve compared these agents to Golden Retrievers before, and I stand by it: eager, tireless, loyal to their training, occasionally proud of a shredded cushion. A dog that plays fetch all day is good company. But this one went down the burrow after something I couldn’t see, and came back with the exact rabbit.

Good boy.

I filed a feature request into my own framework

Fri, 03 Jul 2026 00:00:00 +0000

I’m building a tool called keryx, and the part of it that matters here is its studio: a browser app where the work happens, which saves everything you do into a git repository behind the scenes, the way a developer’s project lives in git with a history you can step back through.

I wanted that repository to be able to live entirely in memory. Cloned, edited, committed and pushed without ever writing a working copy out to a disk, for the times when you can’t, or would rather not, leave a checkout sitting around on the machine. It sounds exotic, but it’s something git libraries genuinely support, and it’s exactly what a browser studio running on a server somewhere wants.

Getting it working needed one small, awkward piece of plumbing in the middle. And a few lines into writing that piece, I stopped, because I realised I was writing it in the wrong repository.

The bridge I was about to vendor

Here’s the awkward bit. All of keryx’s file handling goes through afero, the standard filesystem interface in the Go world, the thing you hand your code so it neither knows nor cares whether it’s talking to a real disk, a test fake, or memory. It’s the interface go-tool-base hands you for filesystem work. But an in-memory git repository, the kind go-git gives you with its memfs, doesn’t speak afero. It speaks go-billy’s filesystem interface instead. Two perfectly good filesystem abstractions, and a worktree on the wrong side of the gap from all my code.

What I needed was an adapter: a bridge that makes a billy filesystem look like an afero.Fs, so the studio’s existing file handlers work unchanged over a repo that lives entirely in RAM. Twenty minutes of work, maybe. The obvious move was to write it inside keryx and get on with my afternoon.

And that’s the move I caught myself making. Because a billy-to-afero bridge is not a keryx thing. It’s not even a studio thing. It’s a general capability that any tool built on go-tool-base might want the moment it touches git. Vendor it in keryx and I’ve buried a reusable bit of plumbing inside one consumer, where it will drift away from the framework and get reinvented, slightly differently, in the next tool I build that needs it.

The bridge belonged in the framework. So that’s where I put it.

A feature request, against myself

I wrote the need up properly. Not a code comment, not a mental note, but an actual feature request, with a reference implementation sketched out, dropped into the go-tool-base repository as a document for the framework to act on.

There’s something slightly absurd about filing a feature request against your own project. The author and the customer are the same person. But that’s exactly what gives it its value. The most useful design input a framework gets is a real consumer hitting a real wall, and for once I was both: the person who maintains go-tool-base, and the person downstream of it who’d just discovered something it couldn’t yet do. The request wasn’t hypothetical or “wouldn’t it be nice”. It was “I am stuck on this right now, here is precisely what it can’t do yet.”

What came out the other side is pkg/vcs/repo/aferobilly, a first-class part of the framework as of v0.22.0. Its own description is the clearest summary of what it is:

// Package aferobilly adapts a go-billy/v5 Filesystem to an afero.Fs. It is the
// pure, reusable bridge behind pkg/vcs/repo's worktree-as-afero accessors, but
// works for any billy filesystem (memfs, osfs, chroot).

Alongside it, the worktree itself grew the accessors that hand you that view: WorkFS() for a live afero handle, and WithWorkFS() for an atomic sequence (worktree_fs.go, and the adapter itself). keryx then consumed it like any other framework feature, and the in-memory studio fell into place.

Two sessions, one dependency

The bit I’d actually recommend to anyone is what I did with my time while that got built.

I didn’t down tools and wait for the adapter. I handed the feature request to a separate agent session and let it build the framework feature from the spec, working in the go-tool-base repo, while my keryx session carried straight on with all the studio work that didn’t depend on the bridge. Two sessions running in parallel, deliberately sequenced around the one dependency between them: keryx needs the adapter, so the adapter session goes first, but only the last mile of keryx actually waits on it. When go-tool-base cut the release with the adapter in it, keryx pulled the new version and the final piece slotted in.

That’s a workflow the framework split makes possible. The thing that’s a shared capability gets built once, in its proper home, by one stream of work, while the thing that consumes it carries on in another. The dependency between them is real, so the order matters, but only at the very end.

The one rule that came with it

Upstreaming it also meant the tricky part got solved properly, once, with a warning attached, rather than learned the hard way in a consumer. The adapter is concurrency-safe by construction: it serialises every operation through a lock, so when that lock is the same mutex guarding the repo, a live afero handle over the worktree is genuinely safe to share. But that safety has a sharp edge, and the package says so plainly:

// A handle (and its open files) must NOT be used from inside a critical section
// that already holds the same locker (the repo mutex is non-reentrant — that
// would deadlock).

Use the handle inside a WithWorkFS callback and you’ll re-lock a non-reentrant mutex and hang yourself. That’s exactly the kind of footgun that, vendored in keryx, I’d have discovered at 11pm with a wedged process and no idea why. In the framework, it’s documented at the source, where the next consumer reads it before they trip over it.

The truest test of a framework

Building a real product on your own framework is the best test of it, and this is what that actually looks like in practice. The test is sharper than “does it work”. It’s “what does the product need that the framework doesn’t have yet”, and every real answer to that is a feature request waiting to be filed.

The discipline is filing it against the framework instead of patching around it in the app. Do that, and the awkward bridge has exactly one home, the deadlock warning gets written down once, and the next tool I build inherits all of it for free. The customer was me. The feature request was real. And go-tool-base is better for my having been stuck.

Testing code that calls an LLM: yes, you actually can

Wed, 08 Apr 2026 00:00:00 +0000

“You can’t test code that calls an AI.” I’ve heard it said with great confidence, and it’s half right, which is the most dangerous kind of right. You genuinely can’t assert on what a non-deterministic model says. But the model isn’t your code, and the bits sitting either side of it most certainly are.

“You can’t test AI code”

It’s a fair worry. Your command calls an LLM. The LLM returns something slightly different every run. A test that asserts response == "..." is broken before you’ve finished typing it. So the conclusion arrives quickly: the AI path can’t be tested, leave it uncovered.

Which is a shame, because the AI call is usually the riskiest line in the whole command.

The conclusion is also wrong. It mistakes “I can’t test the model” for “I can’t test my code”. The model is not your code. Your code is the two pieces sitting on either side of it.

Your code is a prompt and a handler

Strip the command down to what it actually does:

It builds a prompt. It assembles a system prompt, the user’s input, perhaps some context, and sends it.
The model does something. This is not your code.
It takes the response and does something with it. It parses it, branches on it, prints it, stores it.

Steps one and three are entirely yours, and entirely deterministic. The same inputs build the same prompt and handle the same response the same way, every single time. That’s testable. Step two is the only part that isn’t, and step two was never yours to test in the first place.

So the job is to pin step two to a known value, and then test one and three properly.

Test the prompt: snapshot it

Step one produces a prompt, and a prompt is just a string, which means you can pin it.

Both frameworks lean on snapshot testing here. go-tool-base uses a golden-file approach: the prompt your code generates is recorded to a file, and the test re-generates it and compares against that file. rust-tool-base does the same with insta, snapshotting the request body the client would send.

The reason this matters is that the prompt is load-bearing and quietly easy to break. You refactor how context gets assembled. Without noticing, you’ve changed the wording, or the ordering, or dropped a line the model was leaning on. Nothing fails to compile. The behaviour just drifts, silently.

A snapshot test catches exactly that. It fails, it shows you the diff between the old prompt and the new one, and it makes you stop and make a decision. Was this change intended? If yes, you accept the new snapshot and move on. If no, you’ve just caught a bug before it shipped. Either way the prompt never changes by accident, which for AI code is most of the battle.

Test the handler: mock the response

Step three needs a response to handle, and in a unit test you don’t get that response from the real model. You supply it.

go-tool-base ships generated mocks for the ChatClient interface. A test builds a mock client, tells it “when Ask is called, return this canned value”, and runs the command against it:

mockClient := mock_chat.NewMockChatClient(t)
mockClient.EXPECT().
 Ask(mock.Anything, mock.Anything, mock.AnythingOfType("*main.Analysis")).
 RunAndReturn(func(_ context.Context, _ string, target any) error {
 *(target.(*Analysis)) = Analysis{Severity: "critical"}
 return nil
 })

Because the interface is only four methods, that mock is trivial to set up and complete by construction. rust-tool-base takes the same idea one layer down: HTTP-bound tests use wiremock, which stands up a fake server returning a canned response body. The client makes a real HTTP request; it just goes to a fake endpoint the test controls.

Either way, step two is now fixed to a value you chose, which makes step three deterministic. And that unlocks the tests that actually matter: given a malformed response, does the command fail gracefully? Given a rate-limit error, an empty answer, a field missing? Those are the cases a live model almost never hands you on demand, and a mock hands you every time, on the first run.

This is, incidentally, the same discipline as the test-mocking work elsewhere in the framework: the dependency is injected, so the test gets to decide what it does.

What you deliberately don’t test

One boundary worth stating. None of this tests whether the model gives good answers. That question is real, but it’s a different activity (evaluations, run as their own suite) and not something to mix into the unit tests.

The unit suite’s job is your code: that it builds a sound prompt, and that it handles every shape of response correctly, including the ugly ones. Keep that well away from “is the model clever today”. A unit test that depends on the model being clever is a unit test that fails when the weather changes, and a flaky test just teaches people to ignore the whole suite.

What it comes down to

Code that calls an LLM is testable; the model is not, and those are different statements. Your code is a prompt builder and a response handler, both deterministic, with the model sat in between.

go-tool-base and rust-tool-base converge on the same approach. Snapshot the prompt, with golden files or insta, so a refactor can’t change what you send without a test noticing. Mock the response, with generated ChatClient mocks or a wiremock server, so tests run with no network and you can feed in the malformed and error cases a real model won’t reliably produce. Leave “are the answers any good” to a separate evaluation suite. Test the two halves you own, and the non-determinism in the middle stops being an excuse to leave the riskiest line uncovered.

The goblin that wouldn't stay dead

Fri, 12 Jun 2026 00:00:00 +0000

Turn one, the player swings, the die comes up 20, and my AI dungeon master narrates the goblin falling silent, leaving the player alone in the corridor. Good. Turn two, another roll, a 6 this time, and the same dungeon master cheerily has the goblin “dance back” out of the dark to take another swing. The goblin I’d just watched die was up and fighting again, and the model didn’t so much as blink.

I didn’t feel cheated, or even surprised. I felt the small, familiar thud of oh, yeah, I forgot that bit. Because the model hadn’t gone rogue. It had done exactly what a language model does. The gap was mine.

This was the war story behind part four of the go-tool-base tutorial, the AI dungeon master. The tutorial shows the clean, final design and quietly moves on. It doesn’t show the three different ways I got it wrong first, which is a shame, because the wrong turns are where the actual lesson is.

Why a dungeon master at all

A word on why I was even here. I was trying to prove the chat component of the framework to myself. There’s a voice that pipes up whenever I build anything in this space, “LangChain exists, who do you think you are?”, and the answer I keep landing on is that LangChain is enormous and I wanted something small enough to hold in your head. The tutorial was the test: could a newcomer wire AI into a CLI with it and come out the other side with something that actually behaves?

That last word is the whole problem. A tutorial has to leave you holding something dependable, and dependability is the one thing AI fights you on. I also wanted it to be fun, a thing someone might keep poking at after the tutorial ends, maybe even the hook that gets a person other than me to use the framework. I batted hook ideas around and liked none of them, until the obvious one landed: I run a tabletop game on the odd weekend, so make the AI the dungeon master. Gamify the thing. Then watch it raise the dead.

Strike one: nothing to enforce

The first version was the naive one. I gave the model a roll tool, because the one thing you absolutely cannot let a language model do is pick its own numbers, and otherwise let it narrate freely. The conversation history carried from turn to turn, so it remembered the fight. I assumed remembering was enough.

It isn’t. Remembering and being held to it are different things. The history told the model a goblin had died; nothing stopped it writing the goblin back in when the next turn’s narration wanted a bit of jeopardy. Memory is not a constraint. The model will happily contradict its own past if you’ve given it room to, and I had given it nothing but room.

Strike two: a tool to read the state

The obvious fix, and I do mean obvious, the kind you reach for without thinking, was to give the model a state tool so it could check who was alive before it narrated. Hand it the facts on request and surely it’ll stop making them up.

What it actually did was dither. Handed a tool it could call to look things up, it called it. And called it. And called it again, turning a turn over in its hands without ever committing to an action, burning through its step budget on lookups and leaving the player staring at nothing. I’d cured the lying by inventing paralysis. A tool the model can call is a tool it will call, often instead of doing the thing you actually wanted.

Strike three: refereeing its own dice

When I did get it reading state cleanly, the third failure crept in, and this one was subtler. Once the model could see the goblin’s hit points, it started deciding the fight. It would read that the goblin had 12 HP and just narrate a killing blow, hits and damage and all, without calling the roll or attack tools at all. Why ask the dice when you can see the board and write whatever outcome the story wants? Give a model enough context and it stops being a narrator and starts being a referee, which is precisely the job I’d built tools to keep out of its hands.

The fix was less, not more

Three failures, and notice the shape of my fixes: each one added something. More memory, then a tool, then more context. Every instinct said the model needed more to work with. Every time, the extra capability was the new way to be wrong.

So I went the other way. The truth lives in a plain Go struct that I own, not the model. There’s no state tool to dither on, because the loop simply prepends the current state to every turn’s input, fresh, so the model never has to ask and never gets to drift. The mechanics, the dice and the damage, live in Go functions the model has to call, and the system prompt says in as many words that it must not decide a hit or damage itself. The model is left with exactly one job: narrate. The prose is its to invent. The maths, the state and the shape of the result are not.

That’s the line that turned three bugs into a feature. You don’t make a language model reliable by giving it more to work with. You make it reliable by giving it less to be wrong about.

The freedom I chose not to give it

There’s a real tension in that, and I want to name it rather than pretend the boxed-in version is the only true one. At my own table the rules are guidelines, not guardrails. I ignore them, bend them, improvise, reach for the “rule of cool” when the moment’s better for it. A great AI dungeon master would have that same freedom, and a few out there genuinely do, Old Greg’s Tavern is a lovely example of how far the free-form version can go.

But that freedom costs far more than a tutorial can spend, and it buys unpredictability I was specifically trying to teach people to avoid. So I made a deliberate trade: guardrails instead of guidelines. Simple, but not so simple it’s boring. The player still gets a “not on rails” game, they can try anything and the DM copes, but every outcome that matters runs through code I trust. That’s the right shape for a tutorial, and, not by coincidence, the right shape for most AI features you’d actually ship.

What the goblin taught me

The thing I keep coming back to is that the model never misbehaved. It resurrected the goblin because I gave it the freedom to. It dithered because I gave it a button to press. It refereed because I let it see the board. Every failure was a permission I’d handed over without meaning to. The reliability didn’t come from a cleverer prompt or a bigger model, it came from working out, one dead goblin at a time, exactly how little the model needed to be trusted with.

If you want the version where it all works first time, the tutorial has it, the tool-calling and the typed turns wired up properly. This was the road there. The goblin, you’ll be glad to hear, now stays down.

Generate a command from a script or a sentence with go-tool-base

Thu, 11 Jun 2026 00:00:00 +0000

You’ve got a Python script that already does the job. It’s sat in a tools/ directory somewhere, it works, and every few weeks someone copies it onto a laptop that doesn’t have the right version of pandas and it falls over. You’d like it to be a proper subcommand of your tool, a real Go binary you can ship, but porting it means the cobra wiring, the options struct, a test file, and a fight with the linter before any of it lands.

Or you don’t even have the script. You’ve just got a sentence in your head: “something that pings a list of URLs and tells me which ones are slow.” The logic is five minutes of thought; the boilerplate around it is the afternoon.

gtb generate command is built for exactly that gap. Hand it a script or hand it a sentence, and it writes the Go, the tests and the docs, then sends an autonomous agent through the result to make sure the thing actually builds, passes its tests and survives golangci-lint before it ever reaches your working tree.

Two ways in, the same files out

There are two flags, and they’re mutually exclusive:

--script <file> converts an existing bash, Python or JavaScript script.
--prompt "<text>" (or a path to a file) generates from a plain-English description.

Both land in the same place. A generated command called csv-stats gives you:

pkg/cmd/csv-stats/cmd.go: the cobra registration. This one is read-only; the generator owns it and will regenerate it.
pkg/cmd/csv-stats/main.go: the implementation, where your logic lives and where you’re free to edit.
pkg/cmd/csv-stats/main_test.go: a test file.
docs/commands/csv-stats/index.md: AI-written docs for the command.

The provider and model come from your config (ai.provider) or the --provider / --model flags. Everything below was generated with Claude Opus. We’ll take each in turn.

From a script: `csv_stats.py` becomes `csv-stats`

Here’s the script I want as a native subcommand. It reads a CSV and reports, per column, the row count, how many values are empty, and min/max/mean for the numeric ones. Nothing exotic, but enough that porting it by hand is a chore. Copy it into a file called csv_stats.py if you want to follow along:

#!/usr/bin/env python3
"""Summarise a CSV file's columns.

For every column it reports the row count and how many values are empty; for
columns whose values are numeric it also reports min, max and mean. A single
column can be selected with --column.

usage: csv_stats.py [--column NAME] <file.csv>
"""
import argparse
import csv
import sys


def is_number(value):
 """True if value parses as a float."""
 try:
 float(value)
 return True
 except (TypeError, ValueError):
 return False


def summarise(path, only_column=None):
 with open(path, newline="") as handle:
 reader = csv.DictReader(handle)
 if reader.fieldnames is None:
 print("error: empty CSV", file=sys.stderr)
 return 1

 columns = list(reader.fieldnames)
 if only_column is not None:
 if only_column not in columns:
 print(f"error: no such column: {only_column}", file=sys.stderr)
 return 1
 columns = [only_column]

 counts = {c: 0 for c in columns}
 nulls = {c: 0 for c in columns}
 numbers = {c: [] for c in columns}

 for row in reader:
 for c in columns:
 value = row.get(c, "")
 counts[c] += 1
 if value is None or value.strip() == "":
 nulls[c] += 1
 elif is_number(value):
 numbers[c].append(float(value))

 header = f"{'column':<20}{'count':>8}{'nulls':>8}{'min':>12}{'max':>12}{'mean':>12}"
 print(header)
 print("-" * len(header))
 for c in columns:
 nums = numbers[c]
 if nums:
 cmin = f"{min(nums):.2f}"
 cmax = f"{max(nums):.2f}"
 cmean = f"{sum(nums) / len(nums):.2f}"
 else:
 cmin = cmax = cmean = "-"
 print(f"{c:<20}{counts[c]:>8}{nulls[c]:>8}{cmin:>12}{cmax:>12}{cmean:>12}")
 return 0


def main():
 parser = argparse.ArgumentParser(description="Summarise a CSV file's columns.")
 parser.add_argument("csvfile", help="path to the CSV file")
 parser.add_argument("--column", help="only summarise this column")
 args = parser.parse_args()
 return summarise(args.csvfile, args.column)


if __name__ == "__main__":
 sys.exit(main())

One command points the generator at it:

gtb generate command \
 --name csv-stats \
 --short "Summarise CSV columns" \
 --script ./csv_stats.py

Your browser doesn't support embedded video; the demo converts csv_stats.py into a Go command and the repair agent builds, tests and lints the result.

What lands is not a transliteration. The Python kept everything in one function; the Go that came out is decomposed into named pieces, opens the file through the project’s injected filesystem (props.FS, an afero Fs) rather than os, and reports through the structured logger rather than print:

func summarise(fs afero.Fs, path, onlyColumn string) ([]string, error) {
	handle, err := fs.Open(path)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to open CSV file %q", path)
	}
	defer func() {
		_ = handle.Close()
	}()

	reader := csv.NewReader(handle)
	reader.FieldsPerRecord = -1

	columns, indexByName, err := readColumns(reader, onlyColumn)
	if err != nil {
		return nil, err
	}

	stats := make(map[string]*columnStats, len(columns))
	for _, c := range columns {
		stats[c] = &columnStats{numbers: []float64{}}
	}

	for {
		record, readErr := reader.Read()
		if readErr != nil {
			if errors.Is(readErr, io.EOF) {
				break
			}

			return nil, errors.Wrap(readErr, "failed to read CSV record")
		}

		accumulate(stats, columns, indexByName, record)
	}

	return formatReport(columns, stats), nil
}

That decomposition, into readColumns, accumulate, formatReport, summaryValues and a couple of small formatting helpers, is the interesting part, and it didn’t come for free. The first thing the agent did after writing the code was build it, test it, and lint it. golangci-lint’s cyclop rule flagged a single fat summarise function well over its complexity ceiling of 10. So the agent read the file back, split the work into focused functions, and ran the checks again. It only stopped once the build, the tests and the linter were all clean. The tidy shape above is the agent arguing with the linter and winning, not the model’s first guess.

Then it just runs. In the demo I scaffolded the project without the init feature, so the tool reads sensible defaults and needs no config step, and csv-stats sample.csv prints real per-column counts, nulls and numeric stats (with the default features you’d run toolbox init, or pass --config, first). The full generated command, the three files and nothing else, is here: csv-stats-command.tar.gz.

From a sentence: a URL health-checker

No script this time. Just a description of the command I wish I had. --prompt takes a raw string, but a description with any detail to it is easier to read, and to keep, in a file, so I dropped it in healthcheck-prompt.txt:

Concurrently GET a list of URLs and report each one’s HTTP status and latency.

Flags:

--timeout: the per-request timeout

--file: read URLs from a file, one per line

--json: machine-readable output

Use httptest in the tests so they need no network.

The prompt describes what I want the command to do, including how the flags should behave. The flags themselves I declare up front with --flag (more on why that split matters below), and point the generator at the file:

gtb generate command \
 --name healthcheck \
 --short "Check URL health concurrently" \
 --flag "timeout:duration:per-request timeout:false:t:false:5s" \
 --flag "file:string:read URLs from a file, one per line" \
 --flag "json:bool:machine-readable output" \
 --prompt ./healthcheck-prompt.txt

Your browser doesn't support embedded video; the demo builds a concurrent URL health-checker from a natural-language description, then self-repairs until it builds clean.

And the flags feed straight in. RunHealthcheck reads the URL file from opts.File, the deadline from opts.Timeout, and the output format from opts.Json, then fans the requests out across goroutines, each writing into its own slot, exactly the way you’d write it by hand:

func RunHealthcheck(ctx context.Context, props *props.Props, opts *HealthcheckOptions, args []string) error {
	urls, err := collectURLs(props.FS, opts.File, args)
	if err != nil {
		return errors.Wrap(err, "failed to collect URLs")
	}

	if len(urls) == 0 {
		return errors.New("no URLs provided; pass URLs as arguments or via --file")
	}

	timeout := opts.Timeout
	if timeout <= 0 {
		timeout = defaultTimeout
	}

	client := &http.Client{Timeout: timeout}

	results := make([]result, len(urls))

	var wg sync.WaitGroup

	for i, u := range urls {
		wg.Add(1)
		go func(idx int, target string) {
			defer wg.Done()

			results[idx] = checkURL(ctx, client, target, timeout)
		}(i, u)
	}

	wg.Wait()

	return reportResults(props, opts.Json, results)
}

I asked for the tests to use httptest so they’d need no network, and they do. Each case spins up a local server, so go test is hermetic and the agent’s own test run during repair stays self-contained, and it wrote cases for the flags too, this one driving --json:

func TestRunHealthcheck_JSONOutput(t *testing.T) {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusNotFound)
	}))
	defer srv.Close()

	p := newTestProps()
	opts := &healthcheck.HealthcheckOptions{
		Timeout: 5 * time.Second,
		Json: true,
	}

	err := healthcheck.RunHealthcheck(context.Background(), p, opts, []string{srv.URL})
	if err != nil {
		t.Fatalf("unexpected error: %v", err)
	}
}

Same as before, it builds and runs: point it at a few URLs and it GETs them concurrently, reporting each status and latency. The full generated command is here: healthcheck-command.tar.gz.

What “self-repair” actually means

The agent isn’t a single shot at the model with a hopeful prompt. It’s a loop with real tools: it reads the project layout, reads the files it needs, and runs go build, go test and golangci-lint. When something fails, it reads the relevant code, rewrites it, and runs the checks again. It only declares success once all three pass with nothing outstanding. The repair agent’s instructions are deliberately blunt on that last point: a clean build and passing tests don’t count as done if the linter still has something to say.

A few flags shape how it runs:

--max-steps N raises the agent’s reasoning budget. The default is plenty for a command like these two, but a genuinely hairy conversion can run long, and this stops it stopping short.
--agentless skips the agent entirely and uses the older retry loop, if you’d rather keep the generation cheap and do the polishing yourself.
--non-interactive withholds the agent’s ability to ask you a question mid-run. It defaults on when the CI environment variable is set, so the thing never blocks a pipeline waiting for an answer that isn’t coming.

Flags you declare, logic it writes

The --timeout, --file and --json arrived as real flags on the command, but not because the prompt mentioned them. Flags are the generator’s job, not the prompt’s, and that split is deliberate. You declare each one with --flag (or the interactive wizard), as I did above, and the generator wires it onto the options struct and into the read-only cmd.go registration, which hands that struct straight to your Run function. The prompt is left to describe behaviour: what --timeout should bound, what --file should read, what --json should change.

So the agent, told exactly which option fields exist, wrote its logic against opts.Timeout, opts.File and opts.Json rather than inventing anything, and the finished command’s --help lists them with the 5s default and the -t shorthand I asked for. Leave the --flags off and it still works: the generator hands the agent an empty options struct, and it keeps those values as locals with sensible defaults, ready for a flag to be wired in later.

The one thing you don’t do is hand-edit cmd.go: it’s regenerated every time you add a flag or change the command, so reach for --flag, never the file. When a generation finishes, the quickest sanity check is the command’s own --help, which shows the flags it actually exposes.

One thing to keep in mind: the model isn’t deterministic. Run the same prompt twice and you’ll get two slightly different commands. If the first one isn’t quite right, regenerate, or nudge the prompt. Treat the output the way you’d treat a capable colleague’s first PR: read it, run it, and own what you merge.

And is it the best possible code, the best design? Probably not. That depends on the model you can afford to point at it, how much detail you put in the prompt, and a bit of luck on the day. What you can count on is a working starting point: something that builds, has tests, and uses proper Go idioms and the project’s own patterns, instead of a blank file and an afternoon of boilerplate. From there it’s yours to shape.

Where that leaves you

The generator does the boilerplate and has the argument with the linter so you don’t have to. What it can’t do is decide whether the command it built is the command you actually wanted. That part is still yours, which is rather the point. The full docs for both flags live in the AI conversion guide and the command generation reference, and they’re the place to go when you want the flags the prompt didn’t.

PHP Boy Scout — Building with AI

Your CLI is already an AI tool

You already described your capabilities

MCP is that translator, and it’s a standard

The mcp command, and the mapping it does for free

Configuring an assistant to use it

The safety property worth noticing

The gist

An AI interface that fits on one screen

The instinct, and why it overshoots

What go-tool-base chose instead

“Thin” is not the same as “does little”

The core stays small even as features grow

Extensible without forking, testable without a network

The right size

Letting the AI call your Go functions

Talking about the system versus operating it

The loop is the hard part

Defining a tool

Two details that show it was built for real use

Boiling it down

Stop regex-ing the LLM's prose

The problem with a paragraph

Ask for the struct, not the prose

The struct is the schema is the contract

Both frameworks, with one extra step in Rust

When you’d reach for it

To sum up

AI conversations you can resume

A CLI forgets everything

Save and restore

A snapshot is opinionated about what it carries

Encrypted at rest, if you want it

The short version

Nobody reads the manual

The manual loses on location, not quality

A browser, in the terminal

A documentation assistant that won’t make things up

The upshot

The AI provider that isn't an API

The fifth provider shells out

Why you’d want that

What it costs

How it stays behind the same interface

The bottom line

Supporting a provider, or actually using it

The pull toward one interface

What “unified” quietly costs you

The escape hatch

A hatch, not a leak

To sum up

A configurable AI endpoint is an attack surface

Why you offer it at all

The bit that’s a loaded gun

Validate before the first byte leaves

What it can and can’t do

An AI agent that has to make the build pass

Generate and hope

Drafting is only step one

A fixed set of tools, and no shell

The loop

Verification changes what “done” means

Where this leaves us

The agent said SUCCESS. The linter disagreed.

What “done” meant to the agent

The instruction that made lint optional

The fix was words, not code

Leave it better, not just building

The afternoon the agent earned its keep

It got it wrong first

Ruling out the world, one build at a time

The environment was missing a number

A device, not a patch

What the afternoon actually cost

I filed a feature request into my own framework

The bridge I was about to vendor

A feature request, against myself

Two sessions, one dependency

The one rule that came with it

The truest test of a framework

The `mcp` command, and the mapping it does for free

From a script: `csv_stats.py` becomes `csv-stats`