Yesterday, just after 3pm our Internet connection had some major problems.
While browsing and most IM clients died, inbound mail still flowed and MSN Messenger was live at all times.
Testing the network I started with the immediate suspect for any Internet failure - the Proxy server. Seconds later, when I realized that Proxy was up and even when I bypass it I can't browse, I started testing my firewall and DMZ environment.
The way we're configured for Internet is a primary dual T1 connection going to Verizon's MFR router and a secondary T1 connection, both connecting to a DMZ switch which is also where my firewall hook in.
Using the trace command I was able to get all the way to the Verizon router beyond the locally installed MFR router. That indicated a Verizon side problem which is good on one side - My part of the network is working, but really frustrating on the other side - the side that need Verizon's help which is always a long process.
While on the phone with Verizon's support I was told it's a general outage and as we all know by now, it was impossible to get an explanation and\or a time frame. Luckily the problem was solved in 30 minutes and we didn't suffer significant damage.
While troubleshooting this situation I came across a cool useful website Just-traceroute that provide a platform to trace an IP from 4 different locations (USA, France, Singapore & Netherlands). This is handy to isolate many of the possible options. It also provide a built-in "send" button which you can use to email the output to yourself or any admin you're working with.
Response: Vendors Pushing Stretched Layer-2
8 hours ago