authorize dot net, a leading credit card gateway provider, experienced an operational disaster over the weekend. Smoke and fire caused automatic sprinklers to engage, bringing down their entire data center.
Customers were frantic, trying to:
1) contact authorize dot net
2) implement alternative gateway providers
3) assure THEIR customers
4) minimize their loss of sales
There are many reports of organizations losing thousands of dollars in revenues, receiving angry emails from their customers ("why doesn't YOUR site accept payment like a REAL site?"), etc.
The IT community was in an uproar...frantically zigging and zagging. That any kind of assurance...even any kind of INFORMATION...was not forthcoming from authorize dot net, simply added fuel to the fire (no pun intended). Anger that authorize dot net was (seemingly) not staffed 24/7/365, permeated through forums.
However, voices of reason also rang strong and true throughout the onslaught. Disasters happen, said those voices. It is *how* authorize dot net ultimately reacts and responds that will tell the story of can they be trusted so that we continue to use them.
I've been in the shoes of authorize dot net. Disastrous events happen, and we must take action, and the action we take must provide the best solution(s) to overcome the problem so that we preserve as much as we can (integrity, customers, data).
If authorize dot net had a disaster recovery plan (and one would have to assume they did), why was it not implemented when disaster struck? They did respond...it took, near as I can tell, a minimum of 5 hours for them to begin switching operation to their Plan B location and begin booting up new servers.
It behooves us to examine the culpability of the other side: the authorize dot net customer side. (And I am one of those).
I mentioned the IT community was in an uproar. Most of the messages I read were from scared-to-death people. Their sites could not process transactions and they were losing money.
I have to wonder: What were they thinking?
Obviously THEY had no disaster recovery plan. If they had, they would have simply implemented it (as did those with voices of reason), and the failure would have been minimized for them. Their culpability in the matter was just as deep as authorize dot net's. That they panicked - and lost customers - shows that customer service is not their #1 priority as they had no back-up plan.
How long has it been since you've looked at your vulnerabilities, and made a plan to gap the span should the bridge collapse? Whether you're a huge financial concern like authorize dot net, or whether you are a two-person operation, contingency planning is critical.
I got lucky this weekend because their failure had no impact on my org. I had no back-up plan for merchant services. In fact, I'd never even thought about having a back-up plan for merchant services. Shame on me, because I certainly know better.
I'm using the gift of this story as an opportunity to review all processes next week and ask "What if....?", and then I'm going to make sure I have a plan for every gap I surface.
Want to join me? | 
