Monday, October 12, 2009

DR when it matters

Check this Oct 9th 1:36pm scary message from Omgeo, a financial data service provider:
Due to a fire in a transformer in Boston, our Boston Data Center has been shut down. Additionally, the switch which enables us to migrate direct connect customers from the primary to backup connections has been damaged in the fire.

All leased line client connections via Thomson Reuters Network are currently down. They are invoking disaster recovery procedures but according to early reports it does not appear that they will be back up and running today. Therefore trades cannot currently be processed through Omgeo services. Please plan accordingly and process these transactions outside of Omgeo.

Our technical teams are working diligently to resolve the matter as soon as possible. We sincerely apologize for any inconvenience this may cause. We will continue to keep you informed as more information becomes available.

This follow up came at 9:54pm:
As communicated earlier, we wanted to provide an update on our progress in restoring network access to leased line connections via the Thomson Reuters' network. We have made progress in connecting clients to our back-up data center. In addition, power has been fully restored to our Boston Campus by our local utility provider. We now anticipate that all leased line network connectivity will be re-established during the course of the evening, Eastern Standard Time (EST).
Other then my sympathy to the Omgeo guys who worked hard Friday and over the weekend and even more for their sales department that will have to deal with some angry customers I wonder how can it be that one fire shutdown such a service for so many hours?
As a small shop we try to cover any aspect and make sure we're functional at any scenario. So how come these much bigger shops get themselves to a position that a full site is down and all services are shutdown? How can a switch that direct traffic to a backup location be damaged by the same fire???
My lesson is trust no one and think even harder. You always think you covered all aspects but there is always a new angel to DR. I'm sure those Omgeo guys will rebuild their DR plan...
I know I'm going to spend to coming week looking over my DR plan. Again.

No comments:

Post a Comment