The fortnight I spent yak shaving
Two weeks tuning a retrieval system, every day productive and the numbers still not good enough... so I finally went and read what other people had already built.
Tags
12 pages
Llm
The model that wouldn't be forced
My structured output worked by forcing Claude to call a tool that didn't exist. Opus 5.5 stopped playing along, and the fix was to stop doing the trick.
My best finding came from the model I trust least
I sent a research agent off to back up something I'd been saying out loud for weeks. It handed me the evidence against it instead, and it wasn't the first time that model had done it to me.
Nothing in the room can take it personally
Reviewers in a group agree more than they should, and I don't think it's anybody's fault. So I've started building review panels out of things that have never been in a room.
Three models vote on what happened
I ran one job past three AI providers and had them vote. The voting was the easy part. Deciding what counted as agreement was not.
The bot couldn't answer it. Neither could I.
I read the question my support bot had fluffed, gave the correct answer off the top of my head, and was wrong. The docs had fooled the person who wrote them.
The most tempting corpus in any repository
The issue tracker is full of answers, which is exactly why my support bot isn't allowed to quote it. An issue can point at the docs, but it never gets to be the answer.
A client you can still use when you configured it wrong
Go says a constructor either works or it doesn't. That contract stops making sense the moment a client has a dozen optional parts, so I broke it on purpose.
The barkeep I hadn't invented yet
A soft launch for Scout, an AI co-dungeon-master for long tabletop campaigns, and why it will never speak to the players.
Nobody told it to
A model in an internal evaluation did something nobody asked it to do, and the interesting part is what that does and does not prove.
