Speed Test

I wanted to test my Internet speed at the office. I asked Verizon to provide the data but since they have time and more then 3 month later I'm still waiting, I started playing with some online tools.
I picked speakeasy's speed test as my testing tool and found some puzzling results...

I used my PC to test different browsers (FireFox 3.5.4 & IE6) using different path (firewall as default gateway & ISA 2006 proxy server with GFI WebMonitor 2009).
I have 2 T1s coming from Verizon on a multilink router. I'm not going to start calculate what I should expect, why and how. If you're into this data use a Bandwidth calculator.

First let's see how IE6 did with Proxy enabled

This is theoretically the slowest path since it's not only routed through another server before hitting the firewall but it is also processed by WebMonitor.

Disabling the proxy settings and running the same test on IE6 resulted in the following

While the upload speed was significantly faster I was surprised by the download result. Just can't be!

I had to test FireFox and compare the results

And once more just to make sure I'm getting valid numbers

At this point I realized that this test is worthless. Other then the local factors like other users and services that download & upload, the measurement is flexible and depend on so many factors that I just can't trust it.

BGP router down

We have a Global Crossing line. It is a dual router BGP setup with HSRP between the local routers in my office and BGP fall back on the Global Crossing infrastructure.
This morning, few minutes before the opening bell the primary circuit died. As a result the connection failed over to the backup line, which is what you'd expect. The problem started when the primary line started to bounce. Whenever it came back the connections bounced for a second and the users had to reconnect. Then it failed again and they started to get irritated. They are totally right.
As I called the Global Crossing support I found a very efficient service that was listening to my problem and had the will to help (sound obvious but usually this is not the case with big vendors). They started by checking the logs and found that it is bouncing every few minutes. I asked if they can change the HSRP priority to be higher then the primary router, they had no problem doing the configuration change. Problem resolved!
Now this is how HSRP priority work: the primary router get a higher priority and both get the preempt command which allow change of active state if there is a higher priority router online. By changing the priority on the backup router and changing the preempt to manual we ensured that even when the circuit is fixed and stable online it will not become the primary active line unless we manually change it back. This ensure that users will not get kicked off when the line is fixed or when the telco work on the circuit and bounce it constantly.
The circuit was fixed few hours later and after hours we switched back. It is nice to work with good cooperative service for a change!

Cisco IOS 15.0 - released

Cisco released IOS 15.0. This is the next major release after 12.4. It’s been over 4 years since Cisco has delivered a major release of IOS code - 12.4 was released in May of 2005.
Looking at the list of new features there are few interesting items but none of them appeal to the average SMB.
the one feature we might find useful at the SMB level is Cisco DHCP FORCERENEW which enhances security by providing entity authentication and message authentication.
Before you can consider this new software you have to verify that your hardware is compatible and go over the new features to make sure you really need it.
I guess we won't see this new version anywhere near SMB shops because the cost (both for software and training) doesn't make sense (specially these days) while the added value is limited. But it is still good to be aware and as time pass more and more knowledge will spread around the Internet. Use this knowledge to get familiar with the new IOS because at some point it will find it's way to SMB and you want to be ready for that day.

Just Traceroute

Yesterday, just after 3pm our Internet connection had some major problems.
While browsing and most IM clients died, inbound mail still flowed and MSN Messenger was live at all times.
Testing the network I started with the immediate suspect for any Internet failure - the Proxy server. Seconds later, when I realized that Proxy was up and even when I bypass it I can't browse, I started testing my firewall and DMZ environment.
The way we're configured for Internet is a primary dual T1 connection going to Verizon's MFR router and a secondary T1 connection, both connecting to a DMZ switch which is also where my firewall hook in.
Using the trace command I was able to get all the way to the Verizon router beyond the locally installed MFR router. That indicated a Verizon side problem which is good on one side - My part of the network is working, but really frustrating on the other side - the side that need Verizon's help which is always a long process.
While on the phone with Verizon's support I was told it's a general outage and as we all know by now, it was impossible to get an explanation and\or a time frame. Luckily the problem was solved in 30 minutes and we didn't suffer significant damage.
While troubleshooting this situation I came across a cool useful website Just-traceroute that provide a platform to trace an IP from 4 different locations (USA, France, Singapore & Netherlands). This is handy to isolate many of the possible options. It also provide a built-in "send" button which you can use to email the output to yourself or any admin you're working with.

IM– do we have a choice? - update

follow up an old post, Pidgin released an exciting new version.
among many changes there is one huge addition:
voice support with GTalk and voice and video support with the GMail web client

one more reason to switch!

My Tractor Story (aka ASA)

This is my personal Tractor Story…

Remember how my firewall died?
The next morning around 10am Fedex arrived with the new box.
I opened the box and wondered if it is the hardware or the power cable. It was the hardware. Not even 2 years old piece of hardware just lost it.
I copied the config (finally all those tedious Cisco backups pay back!) and it looked fine. Easy. Maybe too easy.
I wanted to use this opportunity to upgrade my software version from 7.2(3) to 8.2(1) and also upgrade the ASDM (which is the GUI console for PIX\ASA). I usually avoid major changes at these situations but how often you get your production firewall offline for such a job?
Reading all about the upgrade it looked like a straight forward upgrade where none of the configs will be affected. To make sure it is as easy as it looks I called TAC and they told me the same thing: copy the software, reload, good to go. So I did.
Reloading the software was easy, quick and worked just fine. Would it be a short happy afternoon?
Afterhours arrived and it’s time for switching. When I finished all adjustments on the network (remove the backup firewall, change back the D\G and cabling) I started testing with one of my out of network colleagues. She was happy about everything but the VPN.
I was looking and searching and found few missing lines. Copying those lines prompt errors. So WTF is wrong? Took me a while to figure it out (how could I not remember? Because I only did it once when I bought the ASA and never again): the Activation Key was missing hence all the Security Plus features didn’t work and the related config failed to load. Lost few hours for the reminder.
At that point I had to reload all the missing configuration that now loaded like a charm. Line by line I got everything back to work and finished this task successfully.
The only thing I still do not understand is the way Cisco handle this. You guys do it everyday with thousands of clients. When you send me a new box you know I’ll have to reload the Activation Key but even more important, you know that the key I received with the original box will not work. Why don’t you automatically create a new key and email it so when the box is here I’ll be able to use it???

Cisco ASA just died

My firewall lost power around 1:30PM, an out of the blue surprise...
Cisco already shipped a new firewall and lucky me, I have my ISA proxy in a working condition just in case. So the case is here!!!
15 minutes later and we're up again via ISA
Stop. Pray. Breathe.

Browser of choice

The world used to split between those who used Internet Explorer and the rest of us
Nowadays the browsers world is different and if you still insist we can split it to old fashion users (mostly on IE) and the rest of us

In my company the older users and those who fear the computer use IE6. Why IE6? Well, when I tried upgrading to IE7 we had to spend so much time adjusting both to the security features and the new menu structure that for most users going back to IE6 was the only step.
The adventurers use Firefox and they all love it. It is faster and easier to maintain. Since they are more flexible by nature they learn how to configure it with add-ons and make it even better.
Google Chrome, Opera and other browsers are not being used (I wouldn’t block them but no one ever asked for any of them)

Since all PCs come with IE any changes must go through me. I have to install it if the user want Firefox (or any other browser) or upgrade if they want IE7\8.
Whenever IE cause problems I use the opportunity to introduce Firefox to the user in question and usually they make it their new browser of choice.
Though I encourage users to move off IE (let’s face it – it is slower!) I do have one big problem: centralized browser management.
My environment uses a proxy server and with IE life is simple: Group Policy pushes all the settings and I can easily change or update parameters.
When it comes to other browsers or in my case Firefox Group Policy doesn’t work. Yes, there is an adm file out there. Been there, tried that. it is not working!
Even the basic task of pointing all Firefox browsers to the proxy server is a nightmare.
The way I handle it is creating a local fully configured Firefox on my desk and with Chris Ilias’ help I create a configuration file. Then all I have to do is distribute it to all clients (require firefox.exe to be off)
Though it is not too complicated it is a repeated tedious process that I go through with every proxy change.

Now in my shop where I control every aspect of the network it is doable but how does Mozilla (and Google, Opera & anyone else who build a browser) expect corporate to use their browsers? They should invest less in making it shiny and spend some time on adm files or other Active Directory based solutions

Quality of life by Threshold and Remote Control

I was at a party on Saturday night and while talking to a friend who is also in IT I realized that threshold is a very important part of our lives. This friend is part of a 4 guys weekly on call duty and he just finished his round. He is exhausted! His company uses very low thresholds for alerts on any subject, which make his life miserable and got me to appreciate the fact I am the one in charge for this decision on my network…

One of the most important aspects of a OneManITShop life is his (or her, don’t start with me on this) ability to have some personal life out of the office (some will just settle for being out of the office…)
You already finished for the day and ready to leave but WHAT IF?
This little IF can make the difference between spending every minute scary and close to your subway pass and having a relaxed evening or weekend anywhere and comfortably.

One side of this issue is the Threshold:
How to determine the correct mark for alerts?
What can wait for tomorrow or must be taken care of right now?

Rule of thumb
“If it doesn’t affect production it can wait”
and
“If users do not notice it they can’t complain about it”

Whenever I install a new service I pause at the notification configuration stage and try to determine what are the implications on productivity and recovery capabilities.
I try to simulate the different scenarios, also Goggling for similar issues and see how it affected other companies.
If it is a problem that will cause damage (like hardware getting too hot) it is a right here right now situation and alerts should go crazy on me
When it affect users, even if the recovery can be easily applied next morning, I’ll set the alerts to notify me and make sure I connect remotely and fix the problem. If there is one thing I hate is walking into the office and have users hunting me. I better get the alert and figure out the solution the night before.

The other side of this is my ability to receive the alerts and connect to the network from any location.
So I have my home PC and a laptop, both connect to the network via Cisco VPN Client and get anywhere in the network. I use 3 tools for remote connection as I found over the years that having just one always end up badly.
First and most commonly used is Symantec’s PCAnywhere. I’m not crazy about it but when it’s working it is fast to connect and doing the job. Do not rely on PCAnywhere solely – it has high tendency for faulting, sessions die and hosts break very often.
As a backup remote control tool I always keep a copy of DameWare. It has amazing remote control capabilities and the huge plus is the remote installation so you do not have to pre-configured anything on any server or PC. Just push it and connect. It can also pull details like running services & processes, events and hardware data. Make your life easier.

These are great solutions and with RDP they cover almost every option and allow full control but one case is missing. What if you’re at the movies or walking in Manhattan and do not have your laptop?

My iPhone 3G have the answer for this situation. I connect to the network via the built in Cisco VPN client and can access anything as if I’m on a laptop. RDP app can do exactly the same job as would any RDP session from a laptop. Amazing!
Another iPhone app saved me when I had to make a change to my Cisco ASA, using SSH I could do it all off the phone and save myself the trip to the office.

IM– do we have a choice?

IM is part of our life, part of our business.
Like any other technology, it can make us more efficient or less productive. It’s all up to how we use it and what we’re using it for.
My company has branches around the globe. For us IM is a money saver and a bridge over time zones and languages. Many people find it easier to communicate via IM where your accent is not an obstacle and so is your co-worked whose accent you can’t understand. And who’s the native English speaker is not the issue here.
The other side of the same story is all the friends and relatives that contact you once you’re logged in. yes, it is nice to chat with mom while working and your wife just want to ask something but not only you spend the time, it is taking your focus off real work. There is one more aspect, even worse: co-workers using IM to chat so their cube neighbors won’t hear them on the phone all day. This way 2 employees are not working…

While those issues are important, they are not for us to worry about. These are management policies and the way of life is that most organizations, specially the small to medium where we, OneManITShop guys work, will keep IM open for all users.

So what is our take on this?

We control the installation which let us have a say about the type of programs we allow.
One example would be AIM users. There are few versions for AOL. AIM stream ads and has the potential to cause problems. AIMPro is much better and it’s more stable. http://www.trillian-messenger.net/gives some extra features. And there are more options.
Microsoft’s MSN messenger has the same issues where old versions do not stream ads but also block many features.
Another aspect is regulation. Many companies must monitor and archive all chats and that requires a 3rd party program that support the IM clients we use. One more headache… more on this aspect, make sure you block web access to IM clients if your 3rd party tool doesn’t capture it.

So what should we use?
My policy is, especially when users run more then one IM account is to use multi-vendor clients. Personally I use Pidgin and it’s awesome. we also use POD - a MessageLabs tool that also capture and archive (for some $$$)
You get all major IM accounts in one place, one window with tabs and most important – no ads. Surprisingly is has a smaller footprint on the desktop which add to it’s clear advantage.
I always keep the latest updates on my file server but make sure my firewall block those updates that each client initiate – it consume bandwidth and make users update on their own, which usually make me work harder.
One more advantage is achieved when the compliance officer check his monitoring systems - he can see one program per user, not 2 or 3 which make it easier and faster and when his work is more efficient it’s an extra benefit for the company.

What is it all about?

I'm working as a system administrator for 8 years and covered many topics.
My day to day life is very interesting but sometimes lonely, I do not have a team mate to share my problems, frustrations and victories.
Our Wall St. company managed to survive the financial crisis pretty well but at the same time the money we spend on technology (or anything else) is monitored closer then ever and as the IT Manager (yep, that is the official title) I'm expected to find creative solutions that save money. you might find them interesting and even useful.
I'll use this platform to share my thoughts, problems and solutions, cool tools and gadgets that I use (and my wife doesn't seem to care about) and help other people with their life.
I hope this will become a place for all the lonely admins out there and obviously all team members are more then welcome to read, comment and share their life.