Handling when things go wrong

Many organizations and professionals pride themselves around when things go "right". This seems to often not be the case. Projects often seem to go over budget, over time, off the rails, and generally fall apart. Handling when things go wrong is never really talked about.

Some Do's

• Don't worry about blame. This is the real world. Things are going to go wrong. There will be production issues. There will be budget issues. Wasting time seeking blame is ineffectual and increasingly people don't care. At least until after the dust settles.
• Keep a clear head. Okay, you've decided to be part of the solution. Now ensure that you have a plan. What went wrong? What can be done to fix it? Taking time now to properly assess the situation will save time down the road.
• Notify stakeholders. Be the start of the communication chain. Something went wrong. Your stakeholders will want to know, moreover they'll want to know that you are handling it. Letting them know there is a problem will hedge off all sorts of uncomfortable questions later.
• Ensure this doesn't happen again. Write a unit test. Enact a preemptive procedure. Add a redundant power supply. Whatever you need to do, ensure that this thing that just happened won't happen again. This will give everyone above you a good feeling.

Some dont's

• Don't fix the the issue "real quick". Great you've determined you can fix the problem in five minutes. That means you can send a quick email that there is a problem, it's being fixed, and more information will be forthcoming. If you just "fix it" hoping no one will notice; everyone will notice. And everyone will notice you noticing. It will be very uncomfortable.
• Don't pass the buck. You found the problem, don't give it to someone else to solve and walk away. If you have to give it to someone else at least take the time to ensure a proper hand-off and follow up.
• Don't over complicate the explanation. Your boss, your stakeholders, they actually don't "care" what happened. They want to know the effect on business. You're in the trenches at your particular level in the org chart. The person above you has their own trenches to mind. Be prepared to give a detailed accounting of what happened, but also be prepared to give a good executive summary.

Remember, things go wrong. Power cords get kicked out and later replaced with locking power cords. Switches go down and get replaced with redundant switches. Sites go down under heavy load. There are performance issues. There are data loss issues. Your site has been hacked. The building collapsed when hit by a small meteor. People won't remember all the times things went right. They'll remember when something broke. Then they'll remember who worked under pressure to fix it.