Last night, just before 3AM most of our floor lost power.
The first to take the hit where PCs as they do not have any power backups.
My servers and routers kept going for additional 90-110 minutes based on the UPS they connected to.
I have a bunch of APC UPS systems both from the 2200 & 3000 series.
An hour and a half later, just few minutes after most UPS died, the power came back. For most servers it was too late...
When I received that phone call I had to connect to my ASA and I was happy when it worked. Poking around I verified all the critical routers and switches are up and the Domain Controllers for all 3 domains are running.
One server didn't want to be cooperative, it did reply to ping but wouldn't allow and RPC based connection to the Windows environment. I had to reboot it but I was home in my pajamas and 45 precious minutes away.
iLO saved me! lucky me, this one server had a working, connected & configured iLO.
In few minutes I had this server restart itself and this time it did. what a relief!
In less then an hour I was able to start all the services, run all the programs and test all the network components. 1 hour - that is the time it would take me get dressed and commute to the office...
The aftermath:
Coming to the office I found that some PCs and few server where up all night. The meaning is that it was not some general building outage but a more specific one. I called the building electrician and he found out that the meter box, which is as old as our building got too hot and caused this power outage. We're going to replace all 3 meter boxes tonight after hours.
Preparing to such a major shutdown is very important. Few things to remember:
This is the time to check all your procedures and backups.
This is when you use the contact list you've collected all those years, calling all your vendors so they know your network will be down.
Make sure the users are aware of the shutdown.
Call your phone system support and make sure they know about it, you might need them.
No comments:
Post a Comment