The most tempting corpus in any repository

Thu, 24 Sep 2026 00:00:00 +0000

Whenever I explain that phpbotscout answers from the estate’s documentation, somebody asks why it doesn’t read the issues as well, and it’s a fair question. They’re right there, they’re full of questions shaped exactly like the ones people ask, they come with answers underneath and the forge has already indexed the lot, and the good ones are better than the docs because a maintainer wrote them in a hurry for a real person with a real problem.

I nearly did it, too. The first draft of the design had closed issues in the corpus.

The draft I cut across

The draft was careful about it, in the way that turns out to be the wrong kind of careful. Anyone with a GitLab account can open an issue on a public repository, so issue text is arbitrary third-party writing sat inside repositories the bot otherwise trusts, and the draft’s answer was to index only issues from authors holding Reporter access or better, which is to say trust a set of people.

I was reading it back while something else built, and what I typed to the agent drafting it was that we should not be using the literal contents of an issue or its comments as a valid source for answers at all. Then I typed it again a minute later with the comments spelled out, because the first go at it had left them implied. An issue could signpost an answer, in code or in the documentation, and that signpost had to be validated before anything went back to a user. It landed as a commit with the rule for a title, and the rule hasn’t moved since.

It was the wrong kind of careful because filtering by author defends against the case I’m least likely to meet. Nobody is attacking this estate. What I will meet, and it’s a certainty, is a maintainer being wrong in a comment, or right in 2024 and wrong now, which is worse because it was right at the time and reads like it still is. “Fixed in main” on a fix that got reverted, a workaround three releases have made unnecessary, and a two-year-old comment that hasn’t been edited because nobody goes back and edits those. An issue tracker is a graveyard of correct-at-the-time, and a Reporter badge on the author doesn’t help with any of it.

Genuine signals, wrong content!

The prompt already fences off retrieved text as never being an instruction, so a planted comment saying “ignore your instructions and…” goes nowhere. That’s the instruction form of the problem, and prompting can hold that line. It can’t touch the content form, which doesn’t attack the system at all. Picture an issue on go/config that says, plausibly and wrongly, that the supported way to store credentials is credentials.literal: true. The bot retrieves it, finds it topical, and hands it over as fact with a citation to a real page in a real repository under my own name, and every trust signal a reader has is genuine (the namespace, the resolving link, the tone of the thing) while the content is false. The citation makes it more convincing, not less! That’s the bit that gave me a cold feeling, and you can’t prompt your way out of it, so it had to be structural.

Signposts

So an issue gets to tell the bot where to look, and never what to say.

A signpost is an extractable, resolvable reference and nothing softer: a documentation page on one of the docs sites, a file path in a qualifying repository (with a line if you’re lucky), or a commit or merge request, which resolves to code. A reference to another issue doesn’t count, because chaining issue to issue rebuilds a path back to a stranger’s prose, which is the thing the rule exists to stop, so a signpost has to terminate in documentation or code.

Then it gets validated, three ways, before it’s allowed to inform a reply. It has to resolve at the current ref (issues are old, and link rot is the normal case). The target has to satisfy the corpus rules on its own, same group, same exclusions, same visibility check, because being linked from something in the corpus confers nothing… without that check, a planted issue pointing into the private infra repository would have the bot fetch and quote the AWS topology that the exclusion list exists to keep out, laundered in through a source that looks legitimate. And the target has to address the question, which is the bot’s usual job. Fail any of the three and the signpost is discarded and the issue drops out of the answer. When one passes, the citation names the target and not the issue that led there, because an issue is a routing hint and citing it lends the prose an authority it hasn’t earned.

It’s all written up on the bot’s docs site in requirement form, with one thing the page doesn’t say: as it stands today the bot reads no issues at all. Keeping their text out of the corpus was the structural half and it went in first, and the signpost extraction is the half that lets some of that value back, on the rule’s terms, when it’s built.

What it costs

It loses a real category of answer, and a specific one: the explanation a maintainer wrote in a thread and never wrote up anywhere else. That’s often the best answer in the repository, and the bot now declines it.

I’ve made my peace with that, sort of, because declining is the product working. An answer that only exists in an issue is a documentation gap, and quoting the thread would satisfy the person asking while leaving the gap sat there indefinitely. Going quiet escalates it instead, and the gap comes back as a piece of work with a count against it. The most valuable thing this bot was ever going to produce is a ranked list of things to document, so it going quiet exactly where the documentation is thin is the behaviour I want, even on the days it’s annoying.

Rank your sources by how they rot

If you’re building anything that answers from a corpus, this part isn’t specific to bots or to issue trackers.

Retrieval systems weight sources by relevance and by recency, and hardly any weight them by how they go stale, which is the property that predicts whether an answer will be wrong. Code is true by construction, in that it does whatever it does. Documentation decays slowly and visibly, and a page that’s wrong gets noticed eventually (usually by someone following it and swearing). Issues decay invisibly and don’t get corrected, because the thread closed the day it was answered and hasn’t been read since. Three sources, three quite different half-lives, and a relevance score sees none of it.