I woke up one morning having decided, somewhere in the night, that I’d wasted a fortnight.

Nothing had gone wrong, exactly. Each day on its own had been productive, I’d fixed real problems, and at the end of two weeks the bot still didn’t answer well enough. That was the morning I finally admitted to myself why.

What the two weeks had actually been

The previous post covers the measuring, so I won’t retell it. Short version: phpbotscout answers questions about the estate from its documentation, it was getting them wrong, and working out why turned into a project of its own, a score that couldn’t be decomposed, a rubric I had to hand to other models because I was the wrong judge, and documentation whipped into shape right across the estate.

All of that was real work and none of it was wasted. The docs are better. The harness can tell a retrieval miss from a real gap now, which it couldn’t before.

And the numbers… were still not good enough.

That’s the position I woke up in. Two weeks of tinkering, every change defensible on its own, and the numbers nowhere near where they needed to be. What I’d been doing was making what I’d built slightly better, over and over, on the assumption that I’d built the right thing in the first place.

Sleeping on it

The first thing I typed when I sat down that morning was:

I want us to step back a little. Having had a night to sleep on this I have come to the conclusion that we are yak shaving right now.

Then the reason, which is the part I’d not let myself think for two weeks: this is not new. Chatbots have been doing this sort of thing for years, long before anyone had a large language model to point at it, and the arrival of LLMs took those approaches up a notch rather than replacing them. There is prior art. There are people who solved the adjacent problem properly and wrote it down.

So I stopped optimising and went to read.

That sounds obvious written down. It didn’t feel obvious at the time, since most days of those two weeks had a defensible task in it, and “go and read what other people did” rarely wins against a concrete task with a number attached. Optimising feels like progress in a way reading just doesn’t, right up until you notice the number has stopped moving.

Reading first, then spikes

The failure mode waiting on the other side of this is obvious enough: you read a load of interesting things, get excited, and start fiddling again with better vocabulary (which is the same two weeks again, just with longer words).

So the reading went into a wiki report before anything got built. Then a defined sequence of spikes, each one testing an assumption so it could be eliminated rather than argued about: contextual chunking as one, a set of query-building variants including a multi-vector approach as another, and so on down the list.

The discipline that mattered was carrying two candidates forward rather than one. The pull toward picking a winner early is strong, because a shortlist of one feels like progress and a shortlist of two feels like indecision (it did to me), and it isn’t. Locking in before the evidence supports it is how you spend the next two weeks optimising the wrong thing, which I’d just spent two weeks doing.

Recusing myself, one level up

I’ve written before about handing judgement to another model because I was too close to the corpus to mark it fairly.

This was the same instinct pointed somewhere else. I sent a second model off with the full context and asked it, specifically, for approaches I hadn’t considered or had become blind to. I didn’t want it telling me which of mine was best, since it was my shortlist I didn’t trust in the first place.

The first version of that move is not trusting your own judgement; this one is not trusting your own survey of the options, which I suspect is harder to notice, since a survey feels complete from the inside by definition.

Admitting it out loud

The first admission came out of one of the spikes, and it’s the most useful thing I typed into that session in the whole two weeks:

I have strong evidence my implementation is wrong and no reference to check it against.

Knowing something’s broken and having nothing to compare it against is an awkward place to be, and the temptation is to keep tweaking until the number looks better (measuring your way to a local maximum, basically). What it actually needed was an outside reference, and that’s what it turned into.

The second is smaller. Halfway through reading its report back, I had to put my hand up:

Can you summarise your finding again, that went over my head a little. What did we learn?

Plenty gets said about the person who nods along to the machine. Less about stopping it and making it explain itself in words you actually follow, which costs nothing except admitting, in the moment, that you didn’t follow it the first time.

What it cost

Twenty dollars topped up on an API account. Contextual chunking rerouted through a subscription quota instead of paying per token, and then watching forty percent of that quota disappear in minutes once it started running!

Research on this stuff isn’t free and it doesn’t scale the way a compile does. That shaped the spike programme rather more than it should have: part of the reason each spike had to eliminate an assumption cleanly is that I couldn’t afford to run them twice.

Where it’s got to

The numbers haven’t moved yet, and I’d be fibbing if I said these two weeks had been building up to a big one.

What changed is that I know what I’m choosing between now, and I know it from what people who came before had already worked out rather than from my own guesses refined fourteen times. The two candidates are still two candidates.

The yak, for once, is thoroughly shaved.