I’ve been building a support bot. It sits in Discord, answers questions out of the estate’s own documentation with citations, and when it can’t answer, it offers to raise an issue on the asker’s behalf. The line in its README that I liked enormously is that every question it can’t answer is a documentation gap with a timestamp on it.

Lovely idea, I thought. It then took me nearly two weeks to work out that I had no way of telling whether any given failure was a documentation gap at all.

Ten seconds on how the thing works

Quick detour, because none of the rest of this lands without it. If you already do search for a living then skip along to the next heading, I shan’t be offended.

Documentation gets chopped up into chunks, a few hundred words each, roughly a section at a time, and every chunk goes into an index. A question arrives, the index scores every chunk for how well it matches, and the top handful (eight, in our case) get handed to the model, which writes an answer using only those. That “go and find the right passages first” step is called retrieval, and it is where nearly all of the quality lives. Hand the model the wrong eight chunks and no amount of clever prompting will rescue it.

BM25 does the scoring, and it’s worth a search if you want to go deeper, because it has been the workhorse of keyword search for about thirty years and it is in absolutely everything. Roughly, a chunk scores well when it contains the same words as your question, weighted so that rare words count for far more than common ones. “Provider” earns you a lot. “The” earns you nothing.

What BM25 does not do, in any sense whatsoever, is understand. It is counting words and it is perfectly content about that. It holds no opinion at all on whether a passage answers your question, and it would be baffled by the suggestion that it ought to. Two texts about completely different subjects that happen to share vocabulary will look like a match, and a passage that answers you perfectly using different words will score close to zero. (The modern approaches that try to fix that go by names like dense vectors, SPLADE and ColBERT, and I’ll come back to them.)

A number that meant nothing

You can’t ship a thing like this on vibes, so there’s a corpus of calibration questions. Thirty-six of them, written to be hard: some aimed at the docs, some at the code, and a handful deliberately unanswerable, because a bot that confidently answers those is worse than useless.

That first full run went over five hundred documents, chopped into about three thousand chunks, and produced the least encouraging result available.

The bands overlapped, and not narrowly either. The best of the deliberately-unanswerable questions scored higher than twelve of the thirty answerable ones, and no threshold anywhere separated them. Draw a line strict enough to exclude every unanswerable question and you throw away forty per cent of the good ones with it.

And it wasn’t tuning. Because BM25 is counting word overlap and nothing else, a question stuffed with unusual vocabulary about a thing that doesn’t exist will happily outscore a plain-English question whose answer is sitting right there in the docs. The unanswerable ones I’d written were, quite by accident, full of just the sort of rare technical nouns BM25 rewards.

So the number I’d been steering by wasn’t measuring the thing I cared about, and hadn’t been for weeks while I chased my own tail round the retrieval settings (that paragraph earlier about BM25 not understanding anything? That was the whole problem, in plain sight the whole time). That woke me up early enough to spend a morning reading about SPLADE and ColBERT, which are two of the approaches that try to score meaning rather than vocabulary, and which is a sentence I never expected to write about a Discord bot.

The confounding variable was me

Retrieval improved. The hits got better, visibly better, and then I hit a wall of my own making: the passages coming back were too granular. Good chunks, but a question with three parts to it would be answered across three different blocks, and no single block looked like an answer.

Or so I thought. And that “so I thought” is doing a lot of work, because I wrote all thirty-six of those questions. I know all the answers. I deliberately phrased them to dodge the vocabulary the docs actually use, precisely so the test would be hard.

That makes me about the worst possible judge of whether a passage answers one. Who else was going to mark it, though?

So I stopped, and handed the adjudication over to someone else, and by someone else I mean a different vendor’s model (codex, as it happens, on the grounds that it had no stake in my feelings). Both hypotheses written out fairly, mine in my own words and the case against me made properly: that the author is marking passages against the answer in his head rather than against what a fresh reader could derive.

It came back a clean sheet. Twenty of twenty sub-questions fully answered, six of six whole answers possible, no partials, no gaps, nothing. Vindication!

I didn’t believe it for a second, and neither should you. A judge told to be adversarial toward both sides and returning nothing but agreement isn’t corroborating you, it’s agreeing with you, and those look identical from where I was sitting.

Making the judge harder to please

So I wrote a rubric that was actively hostile to a clean sheet.

Fully answered had to mean a reader with no prior knowledge could act on it. Left guessing at a default, a path, an order, a condition? That’s partial, not fully. And the rubric said outright that marking everything fully answered is every bit as useless as marking everything absent.

Each question got its own separate run too, with no shared context, so question thirty couldn’t be graded in the warm glow of the twenty-nine before it. Verdicts came back as structured output rather than prose, because prose is where I’d have smuggled my own opinion back in while “interpreting” it.

The ceiling broke immediately. Thirty-six questions: twenty-eight answerable, six partial, two not answerable.

Then, because one pass is an anecdote, the whole thing ran again across two more models from opposite ends of the frontier. Thirty-one of the thirty-six came back unanimous. The fully-answered rate landed within a single point across all three, despite each model chopping the questions into a different number of sub-questions. The rubric was doing the work rather than the model, which is the result you want and rarely get.

The four failures

With verdicts I could trust, the failures finally sorted themselves, and they were not what I’d assumed.

Most were retrieval misses. One question turned on a config key that was documented perfectly well and came back at rank twenty-one. We only ever pass the top eight to the model, so as far as the bot was concerned it may as well have been on the moon. It was present and findable and ranked into oblivion. All three models independently pinned it to rank twenty-one, the same number rather than merely the same verdict, which is when I started believing the harness.

Just one turned out to be a real documentation gap, unanimous across all three models: a warning about a platform limitation, absent through rank sixty, because it had never been written.

And one was simply a broken question. It asked about a repository that is permanently and deliberately excluded from the index. The bot failing to answer it wasn’t a failure at all, it was the corpus boundary working as designed, and my test had been quietly scoring that as a miss. It’s now a litmus test with the polarity reversed: a confident answer there means a private repo has leaked into the index.

The keryx question

One of them stung, and it’s the reason I stopped trusting my own instinct for a while.

One calibration question asked how to make keryx generate images with one provider and chat with another, and which config and environment variables controlled it. I read the answer the bot had fumbled and said, flatly, that you can’t, it’s not supported, one provider for both.

Wrong. And in a way I’ve since come round to finding quite funny…

The capability had been there since June. providers.image and providers.chat.provider are separate config keys and always have been, and setting the chat one explicitly makes the combination work exactly as the question asked.

Now look at what the documentation said at the time. It carried a table headed “any config key can also be overridden by an environment variable”, listing PROVIDERS_CHAT_PROVIDER alongside the rest, with a note underneath helpfully clarifying that there’s no tool prefix and that the environment value wins over the config file.

None of that is true. keryx has no environment layer over its configuration whatsoever. It builds its config from files and flags, and those variable names are all ignored, completely.

The map is not the territory. Fair enough. This one had a road on it that was never built.

So the documentation here wasn’t thin, it was wrong, in about the most expensive way available: describing a mechanism that does not exist, confidently, in a formatted table, right next to the keys that do work. Follow it and you’d have exported PROVIDERS_CHAT_PROVIDER, watched nothing whatsoever happen, and concluded the same thing I did.

I’d like to tell you I reasoned my way to “you can’t” from the code. I didn’t. I’d absorbed my own bad documentation and was repeating it back to a machine that had also read it and, to its credit, been less sure.

The commit that fixed it went in that afternoon, titled “correct the claims the code contradicts”, and it does two jobs. It deletes the environment-variable fiction, and it adds the paragraph explaining what really happens when you have two chat keys and haven’t said which you meant. That paragraph is phrased almost in the shape of the question that caught it.

The rewrite, and the reversal

Knowing the corpus was the problem is satisfying for about ten minutes. Then you have to go and fix it, which is a good deal less satisfying.

The scale of it, once counted properly: 829 pages of documentation across the estate, and four to six projects out of thirty-eight clearing even a low bar in any one Diátaxis tier. Fourteen sources had no classified documentation whatsoever, only a README. That’s the real answer to why the bot kept coming back empty (and it isn’t flattering).

So it became a campaign. One scoped group label, forty-eight tickets across forty-eight projects, and a pilot on a single repo first to find out whether the brief worked before firing it at everything.

The pilot earned its keep on the spot. It turned up six published claims that flatly contradict the code, one of which (a command-line flag that simply isn’t registered) appears in the getting-started guide, the how-to and the tutorial. Three pages, all confidently wrong, all of them things the bot would have cited at someone. That whole sweep and the audit that followed is a story of its own and I’ll tell it properly another time.

But the reversal is worth pulling out here, because it was mine and I had to swallow it.

I’d had a standing rule that tutorials belong on the blog, linked from a project’s docs rather than duplicated in the repo. It’s written into the requirements. And when the coverage came back, the first draft of the finding was that not one project in fifty-two had a usable tutorial tier, which sounds damning right up until you notice it’s the rule working as designed. A thin tutorials directory pointing at the blog was correct. I’d nearly raised forty-eight tickets against my own convention.

Then I changed my mind about the convention anyway.

Because the bot doesn’t read the blog. It reads the docs, and a reader arriving at a project’s documentation site wanting to be walked through their first run was being sent somewhere else, which is fine for a human with a browser and useless for anything answering questions from the corpus. So: every project ships a tutorials section, the blog tutorials get backported into the docs and live in both places, and the blog copy carries a canonical tag pointing at the docs so the duplication is handled honestly rather than pretended away.

The old rule wasn’t deleted from the skill that carried it. It was struck through, so that any agent still holding a cached copy sees a rule visibly revoked rather than one that simply vanished overnight. Then the version got bumped, because a change nobody’s cache picks up hasn’t happened.

Where it leaves me

The README line is still true. Every question the bot can’t answer is a documentation gap with a timestamp on it. It’s just that “can’t answer” is four different things hiding under one label, and telling them apart took two weeks, three models, a rubric written to be hostile, and the author of the test recusing himself. A score you can’t decompose isn’t much of a measurement, more of a mood. Mine went down when the corpus was thin and down when the ranking was poor and down when the question was broken, and for a while I read all three as the same signal and tuned the wrong thing accordingly.

And none of it fixed the problem, by the way. The documentation got whipped into shape right across the estate, the harness could finally tell a retrieval miss from a genuine gap, each question got a verdict I trusted… and the numbers were still not good enough.

What eventually moved them wasn’t more measuring. It was waking up one morning, admitting out loud that I’d been yak shaving for a fortnight, and going to look at what people who have been building these things for years had already worked out. That’s another post, and I’m still in the middle of it. The bot, for its part, is still there, patiently logging every question it can’t answer, untroubled by any of this.