I paid a model to go and find the evidence for an opinion I’d already formed. It came back with the primary source that killed it.

I’d been carrying an argument around for a few weeks: that Google had given up chasing the frontier coding crown and gone and won a more interesting race instead. I liked it. I’d said it out loud to people (which is the dangerous bit). So I sent a research agent off to gather the supporting material, the way you do when you’re about ninety percent sure and you just want the citations.

What I asked for, and what I got

The brief was narrow. Find me statements from Google, or from people close to it, that say or imply they’ve stopped competing on coding.

What came back opened with a note saying that no executive at Google, Alphabet or DeepMind has publicly conceded anything of the sort. Then it handed me an internal memo from the chief executive telling the AI team to accelerate and stay focused on the frontier, reporting that the delayed model everyone had read as a strategic retreat was actually pulled for lagging at coding, and a share price that had dropped four percent the day that news broke. It found four credible analysts making a version of my argument, which was awkward for a different reason: a previous search had assured me there weren’t any.

And then it did something that actually stopped me. A characterisation I’d been repeating as established fact, that the team was in the middle of an identity crisis, traced back to a single piece by a single financial reporter, syndicated across the aggregators without anyone independently checking it. I’d read it in three places and counted that as three sources. It was one.

So that post is dead, and it deserved to be. I’d gone looking for confirmation and been handed a correction, which is about the best outcome available and feels nothing like it at the time.

Marking its own homework wrong

The prompt I use for this kind of work makes the agent grade every single item it hands back. Confirmed means you fetched it and read it. Reported means you saw it mentioned somewhere. Each line carries a grade and a link.

Six of the thirteen distinct links in that report were domain roots, not articles, just bloomberg.com/, semafor.com/, stratechery.com/, roughly as useful as citing the library. Then six separate findings, all graded confirmed, all pointing at one index page that doesn’t resolve, and one analyst who turns up three times under three different job titles.

So the report was marking its own homework wrong, out loud, in a way I could catch with one grep. The grades say one thing, the links say another, and they’re both right there on one page for anyone who cares to look.

I stopped being cross about the argument round about here.

I’d seen this shape before

I wasn’t surprised, mostly because I’d watched this model do it to me three weeks earlier, under conditions tight enough to count as an experiment.

I’d been pulling apart the research corpus behind Scout, my tabletop campaign tool, and I wanted a fresh set of eyes on it. So I stripped every prior AI review physically out of the checkout, gave three different models real file access rather than a summary, handed them an identical prompt, and kept them blind to one another. Two were frontier coding models. The third was Gemini, reached through a different harness, and I added it late (a bit on a whim, if I’m honest), on the reasoning that it had a very different pool of knowledge to draw on and might turn up something the other two wouldn’t. It might also be rubbish, and I wasn’t going to know until I’d run it…

It produced the best finding of the round, and by a distance.

As far as I could see, every piece of adoption analysis in that corpus treated the Dungeon Master as the buyer, and fairly enough, because the DM is the one who installs the thing and pays for it. Sitting somewhere else in that corpus was a separate finding: recording can’t start until each person at the table has agreed to it.

Gemini put the two together. The DM is the buyer, and the table holds a veto!

So if four players are keen and the fifth doesn’t want an AI recording their voice, the DM isn’t going to throw their mate out of the game. They’ll uninstall Scout.

I hadn’t modelled that and neither had the other two, and it came out of welding together two documents that had been sat side by side in a folder for weeks.

In that review it also invented a refusal rate out of nothing to get to a tidy adoption percentage, read straight past a correction in the report it was reading and restated a null result as a positive one, and used a price figure the handover notes had already flagged as wrong by a factor of five to ten.

So that’s the best finding of the round and three fabrications, all in one document and handed over in one breath. When I wrote the round up at the time I got it down to eight words: breadth was its contribution, depth was where it erred.

A leaderboard answers a question I’m not asking

We rank these things on one axis, which model is best, and you get a number, there’s an ordering, and it updates every few months.

That ranking is a purchasing signal, and it’s a decent one. If you’re picking a single model to put behind a product, or deciding where the subscription goes, the leaderboard is telling you something real.

I’m not purchasing, though. I’ve got four of these things open on a normal working day, the subscriptions are paid either way, and a fair chunk of my time goes on setting them against each other. What I’m actually doing all day is handing work out.

And for that, “which is best” doesn’t help me much. It flattens a shape into a number, and the shape is most of what I need.

Swinging the ship

A ship’s magnetic compass doesn’t point at magnetic north. It can’t. It’s sat inside a few thousand tonnes of iron, and the hull drags the needle off true by an amount that changes depending on which way you’re heading.

You don’t bin the compass. You swing the ship instead: turn her slowly all the way round, sighting on something whose bearing you already know, and write down how far the needle lies on each heading. Then you mount that list on a card next to the binnacle.

From then on the compass is more use than a perfect one that’s never been checked, because you know how far it lies and you take it off every time you read it.

That’s what those two runs were, though it wasn’t what either of them was for. I built the Scout review to review Scout, and I commissioned the sweep to win an argument. The card fell out of them sideways, and I suspect that’s why I trust it… nothing in either run was set up to make it come true.

What the card actually changed

Two prompts, and they pull in opposite directions now.

The depth models get asked for a verdict, under a hard budget. One shell command, then write the report. I want a judgement and I want it defended.

The breadth model is forbidden a verdict. Candidates and evidence with sources, not conclusions, and grade every item: confirmed if you fetched and read it, reported if you only saw it mentioned.

That grading rule is most of the card, and I was a good while working out why it does so much. Accurate it isn’t, and I’ve stopped expecting it to be. What it is is auditable, and I’ll take that, because accuracy I can’t check is really just how the document made me feel.

A confirmed grade next to a bare domain root is a failure that puts its own hand up. Six of them put their hands up in about four seconds.

I’ve come at this from the other end before, when a suppression note in a scanner sounded like a reason and turned out to be vouching for something nobody had checked. Same disease, I think, and trusting it harder hasn’t cured either. What’s helped is making it show its working each time, so I can go and pull on it.

The panel post, from the other end

I’ve put three models on one job and had them vote before, and that piece leans on the observation that they fail in different directions. But it uses the difference for redundancy. Same job, three answers, majority carries, and the individual shapes get averaged away. I said there that the roster was disposable, that swapping who sits on the panel is a config change and not a rewrite.

That’s true of a vote and it isn’t true here, and I’ve been chewing on that for a while now. A vote can afford a disposable roster because it cancels the shapes out. Stop voting and start assigning, and who’s on the roster becomes most of the design. Nothing’s averaging the errors out any more… I’m choosing which one I’m prepared to live with and steering round it.

So the two posts disagree, and I think both of them are right about what they’re each doing. I’d rather say that than pretend I’d worked it all out in one go.

What’s written on the card

For the record, Gemini comes last on every coding benchmark I actually care about, and I wouldn’t hand it a refactor. That’s what’s written on its card, right next to the bit about finding what the other two missed.

What I would say is that I’ve mostly stopped asking how good these things are, because I can’t do very much with the answer. What I can use is knowing how they go wrong, and whether they go wrong consistently enough for me to write it down. There are models I rate a great deal higher than this one that I still couldn’t write a card for…