When failure happens we are often quick to just fix the issue and move on. After all, failure is inevitable. If people didn’t fail at things, all of us would be out of a job. I’m a firm believer that there is no substitute for experience. But that doesn’t mean experience is the only way to learn about a problem so you can avoid it.
Some organizations are accustomed to firing people when they make egregious mistakes. They don’t look at the contributing factors. They believe that whatever system is currently in place, works – you just need the right people in the seats. Perhaps that’s true if you are whittling whistles, but not so in IT. When you are hiring highly qualified professionals with a broad skill set you can’t just dismiss them when they make a mistake. If you do, you’re potentially letting years of experience and institutional knowledge walk out the door. In fact, you are likely making things worse with their absence.
In order to reduce the likely hood of a mistake happening in the future, you need to have a multifaceted view of the problem. You need management and the people doing the day-to-day work together, contributing towards a solution. One way to do that, is by establishing a just culture and analyzing a problem through blameless postmortems. I don’t want to rewrite an already excellent article on the subject, go read about blameless postmortems at etsy.
When an outage or other problem occurs, first – fix it. Stop the data leak, get the systems back up, get people back and working, whatever the case may be. Then, while the events are fresh in everyone’s mind, bring the team together and work towards a systemic solution. Failure provides an opportunity to call attention to the people, process, and technology you need to succeed. Perhaps static code analysis could have prevented the problem, or increased redundancy, increased training, or a pre-launch checklist, or shifting security left in the software development life cycle.
You won’t get these fancy tools or process changes by sulking in the corner and complaining about what should have happened. Bring the team together, respect each other, listen to what happened. Move forward with one voice – calling for the same changes to improve the organization.
I have created a presentation template that you can follow to introduce just culture and blameless postmortems in your organization.
Copy it, customize it for your team, follow the notes.
Now go forth and seize the opportunity of failure!