Pretty much every company I’ve worked for in the last decade has told me, usually in my first week and often on a slide, that it has a blameless culture. I’m all for what that was meant to mean. I just don’t believe the word, and the longer I’ve spent around incident reviews the less I believe it.

I can’t think of a single incident I’ve been near that had no blame in it anywhere.

What it says on the tin

Think about where the blame actually goes when something breaks. Sometimes it’s me, sometimes it’s my team, sometimes it’s somebody else’s team, and sometimes it’s a provider who had a very bad day and took half the internet down with them. And when nobody can find anyone at all, the incident report reaches for its last resort and calls it an act of god… which, coming from a confirmed atheist, I find absolutely hilarious. The one time the industry admits it can’t find a culprit, it hands the blame to a mythical omnipotent deity!

There’s always something, somewhere in the chain. Blame exists. What “blameless” really is, if you strip the branding off, is a cheat code. It’s a softer way of saying “you done fucked up”, and, much more usefully, it’s a way of stopping an arrogant employer from knee-jerking that straight into “you’re fired”. It turns the conversation into “what can we learn from this, and let’s not do it again”.

That’s a good thing to want! The intention is right. I just think the label has been doing damage for a while now.

What it was meant to say

If you go back to where the term went mainstream, it’s John Allspaw’s post on Etsy’s engineering blog back in 2012, and it’s well worth a read if you’ve only ever met the word on a slide. The idea is that an engineer whose actions contributed to an incident can give a full and detailed account of what they did, what they saw and what they assumed, “without fear of punishment or retribution”. Because if they think they’re going to get a kicking, they’ll leave out the bits you most need to hear.

And then, in the same post, he says engineers are “not at all ‘off the hook’”. They’re very much on the hook for making things safer afterwards. Google’s SRE book puts it as focusing on the contributing causes “without indicting any individual or team”, on the grounds that “you can’t ‘fix’ people, but you can fix systems and processes”.

Now, I’d push back on that one a bit, because you absolutely can fix people, in the sense that line means it. (I mean educate them, before anyone reaches for a pitchfork. The other sort of fixing is a very different blog post, and one I’ve no intention of writing.) Someone who got it wrong because they didn’t know something can be taught it, and that’s most of what mentoring is. Fix the system, by all means. But a review that only ever fixes the system leaves everybody in it knowing what they knew before, and learning was supposed to be the reason for having one in the first place.

So the promise was that it’s safe to tell the truth about what you did, which is a long way from nobody having done anything… and somewhere between that blog post and the slide in my induction, one turned into the other.

Softened on the way up

My real gripe is with the companies that wave the word about as an excuse. An excuse not to assign fault to anyone, person or group, and then, because nothing was anybody’s fault, an excuse not to learn anything either.

The thing is, you can’t do that if you do the root cause analysis properly. Do it properly and the causes turn up very fast, and they’re rarely where anyone would like them to be. So far too many companies don’t do it properly, or they do it, and then frame what it found to suit whoever’s reading it.

At one large corporate I worked for, I wrote a number of root cause write-ups for incidents (and some proper cock-ups). Being me, they were very fine-grained. I dug right down to the gritty truth of where each problem had actually started, and there were plenty of them, and a good many of those causes sat a long way above the engineers, in decisions made higher up about time, people and priorities.

Then I sent them up the chain for review. And at every layer of bureaucracy they passed through on the way to the C-suite, they got edited. A sentence softened here, a cause reworded there, a recommendation turned into an observation… until by the time a write-up reached the top it was little more than a shrug. Management didn’t take any accountability for the parts that were theirs. The blame stayed where it was shallowest, with the engineers, and the report still had “blameless” written on it.

I’ve seen what happens when there’s no safety at all, too. Years earlier I watched a chief exec go looking for someone to pin a budget overrun on, and land on my project manager, who was on a beach with his family at the time and hadn’t set any of the conditions that caused it. And in my experience the person who made the conditions very rarely ends up being the one wearing them.

So I’m not arguing for going back to that. A blameless review that’s done properly is the defence against that chief exec. The trouble is that a blameless review done badly protects the same people he was protecting, only more politely.

Keep the safety, change the tin

What I’d actually like is for us to keep everything Allspaw was asking for and stop calling it blameless. It’s safe to tell the truth here. Nobody’s getting fired for owning up to what they did. And we will find the causes, all of them, including the ones that lead back to whoever set the deadline, cut the estimate or left a team three people short. Then we’ll write them down where they are, not where they’re comfortable.

I wouldn’t call that a witch-hunt, just being accurate. And in my experience it only works if the most senior person in the room goes first, because if they don’t, everyone below them works out very quickly that blame flows downhill and the write-up is for show.

So my own answer to “whose fault was it?” tends to have the same name in it. Whatever the context, my open source work or a paid job, if I was involved then a share of the responsibility is mine (it’s nearly always shared, and pretending otherwise is just the shrug again).

These days that’s easier to see than it’s ever been. Most of what goes wrong in the estate goes wrong in a session I was supervising, on code I approved, and I made a rule for those sessions a while back that a human is accountable for every line that gets merged. So when one of them ships a bug, the chain has exactly one human link in it. There’s no layer above me to soften the write-up and no team below me to leave it with, and a model makes a rubbish act of god… so it’s me, and these days it says so on the tin.