[Reading time: 1 minute 32 seconds]
“For every complex problem there is an answer that is clear, simple, and wrong.” – H.L. Mencken
And that’s the problem with root causes.
Any reasonably complex system will have multiple safeguards against failure.
Yet all systems will end up failing sooner or later.
Of course, any self-respecting engineering organisation will perform a post-mortem once this happens: inspect the failure, try to find out how it came to pass, and try to find ways this failure won’t happen again.
This is a very good idea. What is a terrible idea, however, is to whittle it down to a single point of failure.
Failures of complex systems are always failures of the system as a whole, the one thing that brought the system down was only the straw that broke the camel’s back.
And… humans are always part of these systems. Crucial parts, in fact.
So any analysis into the causes of a particular failure will touch on human aspects: the humans that devised the system, that built it, that operated it. Someone will have missed something somewhere. That’s just how it is.
This leads to something difficult but important: you need to find a way to acknowledge human involvement without blaming anyone.
The best way to do that is to remember that humans always have a dual role: they are both producers of and defenders against failure.
But they can only work as defenders if they are able to see the system clearly enough.
This is the crucial task at the heart of creating robust systems: creating a culture which enables looking for, creating, and using feedback loops. And acting on what you find, without fear.
Can people in your organisation act without fear, especially if they worry (or know) they messed up?