ApplicationXtender and Active Directory

I manage an ApplicationXtender server which is used to manage documents for compliance and accounting purposes. We used to work with a local user database but few years back we switched to the AD synced mode which uses the local logged user. This way I win twice: The password is more secure since it is changed by the domain policy AND the user is not prompt with login screen for ApplicationXtender. A rare win win.

My password policy require the users to change their domain password every 3 month. 14 days before due date they start getting these daily reminders.
When a user forget to change it (or just ignore it on purpose) and the 14 days pass all connectivity to any network resources is lost. When the password is changed work is not interrupted and is not affected. Well, this morning I found this to have one exception...

When you change the password for the user while he is logged in ApplicationXtender will not connect. Testing the Data Source connectivity is successful and yet the client hang. Logging off and then logging back with the new password fixes the problem. I agree that it's weird but hey, I didn't write the program...

Cisco security

I was at a new site for a consulting job and found so many security holes that I had the need to write down few basics for those who didn't know or already forgotten.

When you configure a new router you should decide on a security method that will keep it as secured as possible. Keep in mind that doing nothing is NOT a method.
Working for a small to mid-size shop you're the only one handling the routers, maybe 2-3 more people need access for specific tasks. You have to make sure no one else - internal or external get on the device and make any changes.

One of the most important actions these days is removing all Telnet access and switch to SSH. It is not always possible with old equipment but if you have any of the supported boxes please use it. It is a major security improvement.

Next thing to think about is your local user list. Passwords are kept on the router and show in Show Running-config. Most admins think that using type 7 encryption is good enough. Check this online tool and think again. It is able to decrypt Cisco's encrypted "type 7" passwords!
Now you think this is impressive, check this in the router IOS decryption option...

Another instant easy to implement option is AAA. Use another server for authentication to keep passwords off the router. RADIUS servers can sync with Microsoft's Active Directory and use the same password policy you apply for users on the domain, to the router. That will also make your password management an easier task. Windows 2003 & 2008 can use as RADIUS server using Internet Authentication Service.

These are basic easy tools that cost nothing and require few minutes. You should think of them as a MUST and go implement them yesterday. You do not have to know too much, these do not require being an expert so what are you waiting for?

Phones get static and drop calls

We're using an AVAYA S8400 server going out to 3 lines:
2 PRIs for local and long distance calls
T1 for International calls

2 weeks ago users started to complain about static on the line on local and long distance calls. At first when it was one or two users I thought it might be the phone though you expect a failed hardware to fail at all times and the static was random.
Then more and more users complained and some even added a new complain about drop calls.

In order to isolate the problem I have routed 2 heavily used area codes to the T1. Since no one reported problems on International calls it was a great way to find if the problem is local, equipment or infrastructure on my end or is it PRI\Verizon related issue.

Verizon which run both PRIs tested them after hours and reported back that both lines are clear.
At the same time no one complained about static at the 2 isolated area codes.

As the weekend passed we came back Monday morning to a clean line. No static. No drop calls. I changed nothing and yet the problem seem to be resolved...

Before I close this case I had to switch the 2 area codes back to the PRIs and make sure they are still clear. 4 days later the lines are clean.
To be on the safe side I asked Verizon to test the local equipment (that would be the demarc in my server room). Last night when the technician finished testing he confirmed it is clear.

While no one can explain it there is one last possible option, something we'll never know but can always blame: The building lobby is under renovations.
We used to have our office downtown where the infrastructure is old and fail whenever it's raining. Whenever we saw a Verizon truck within few blocks we anticipated phone or Internet problems. Unfortunately we where right most of the time. Taking this experience under consideration it is more then reasonable to think this option is possible. But as I said, we will never have an real answer.

Lotus Notes Expired ID file

Security is the reason Domino Server require all ID files to recertify once every 2 years (that is the default, can be changed manually).
When the expiration date get close Domino is kind enough to notify the user and there is your problem...
Typically there are 3 groups of users:
Some users will actually read the message that ask them to forward it to an administrator (a one click action).
Most of them will call you and ask why they received the message.
The problem is with the other group, those who ignore it. This group cause problems since they will show up one morning (as one of my dearest users did this morning) and will be locked out of Notes with this error on screen

Server Error: Your certificate has expired

When you have such a user you have to use the Administrator console using this procedure: How to manually recertify an expired ID.
Now don't get me wrong. It is not that complicated and I'm not complaining but it does involve an extra step: Physically access the users client to import the new recertified ID file. While in a small shop it is not that bad, in a larger environment it is a huge pain.
How to avoid it? Educate your users, explain about this certification and hope they'll remember next time and hope they leave before the renewal date because most chances are they will not remember.

HP Printers backorder - closer

A while back I told my HP Printers backorder story. In short, I've been waiting for my HP 2035 since August because it was back ordered.
The good news is that I received a notification email that the printer has been shipped and will be here Monday.
Hooray

BGP router down

We have a Global Crossing line. It is a dual router BGP setup with HSRP between the local routers in my office and BGP fall back on the Global Crossing infrastructure.
This morning, few minutes before the opening bell the primary circuit died. As a result the connection failed over to the backup line, which is what you'd expect. The problem started when the primary line started to bounce. Whenever it came back the connections bounced for a second and the users had to reconnect. Then it failed again and they started to get irritated. They are totally right.
As I called the Global Crossing support I found a very efficient service that was listening to my problem and had the will to help (sound obvious but usually this is not the case with big vendors). They started by checking the logs and found that it is bouncing every few minutes. I asked if they can change the HSRP priority to be higher then the primary router, they had no problem doing the configuration change. Problem resolved!
Now this is how HSRP priority work: the primary router get a higher priority and both get the preempt command which allow change of active state if there is a higher priority router online. By changing the priority on the backup router and changing the preempt to manual we ensured that even when the circuit is fixed and stable online it will not become the primary active line unless we manually change it back. This ensure that users will not get kicked off when the line is fixed or when the telco work on the circuit and bounce it constantly.
The circuit was fixed few hours later and after hours we switched back. It is nice to work with good cooperative service for a change!

Lotus Notes send on behalf of

Today, just as I got ready to leave for the day my CEO popped from nowhere and said "I need your help". Thank you very much!
He walked into his office and asked me to look at his Lotus Notes. One of our sale guys sent an invite to few clients where the CEO was Cc'd but the attachment only showed a pdf icon but had no real file attached to the mail.
Since it was late afternoon in the US and the sales guy leave in London he asked for a solution right here right now and did not want to wait for the next day. My task was simple: Resend those emails from the sales guy's mailbox as if he sent them. Make sure the attachment is there for real.
The reason it the attachment was missing to begin with was the mobile he used to forward it didn't support attachments. Go figure...
Problem with Notes is that you can send on behalf directly from your inbox but to make it look as if the other guy sent it himself, you must have a client configured for him, using his ID file. Since this guy never work in my office I have no such PC.
One way to do it was reconfiguring my own PC to use his ID. I can do it but do not like this path since there is always something with Notes when switching IDs.
Instead, I used the Web Access (I was lucky to have his password). Using Copy\Paste I got the text and addresses to the browser and attached the file too. One important thing to remember is sending with the Send and Save option which save a copy of that mail to his mailbox

DR when it matters

Check this Oct 9th 1:36pm scary message from Omgeo, a financial data service provider:

Due to a fire in a transformer in Boston, our Boston Data Center has been shut down. Additionally, the switch which enables us to migrate direct connect customers from the primary to backup connections has been damaged in the fire.

All leased line client connections via Thomson Reuters Network are currently down. They are invoking disaster recovery procedures but according to early reports it does not appear that they will be back up and running today. Therefore trades cannot currently be processed through Omgeo services. Please plan accordingly and process these transactions outside of Omgeo.

Our technical teams are working diligently to resolve the matter as soon as possible. We sincerely apologize for any inconvenience this may cause. We will continue to keep you informed as more information becomes available.

This follow up came at 9:54pm:

As communicated earlier, we wanted to provide an update on our progress in restoring network access to leased line connections via the Thomson Reuters' network. We have made progress in connecting clients to our back-up data center. In addition, power has been fully restored to our Boston Campus by our local utility provider. We now anticipate that all leased line network connectivity will be re-established during the course of the evening, Eastern Standard Time (EST).

Other then my sympathy to the Omgeo guys who worked hard Friday and over the weekend and even more for their sales department that will have to deal with some angry customers I wonder how can it be that one fire shutdown such a service for so many hours?
As a small shop we try to cover any aspect and make sure we're functional at any scenario. So how come these much bigger shops get themselves to a position that a full site is down and all services are shutdown? How can a switch that direct traffic to a backup location be damaged by the same fire???
My lesson is trust no one and think even harder. You always think you covered all aspects but there is always a new angel to DR. I'm sure those Omgeo guys will rebuild their DR plan...
I know I'm going to spend to coming week looking over my DR plan. Again.

Yahoo! Mail screen resolution

A user called in with a problem: she can't open her Yahoo! Mail account. She does log in and see the main template, it show "3 unread messages" at the screen title but the messages are not loading.
When I stepped to her desk I noticed that her iexplorer.exe process uses 99% of the CPU. I killed it and asked her to try again. Same problem, same solution. I tried a different approach and used my test account on Yahoo! (I never use it but this is why I keep it). Unlike her, I was able to get a message error (is it because my mailbox is empty?) which provided me with the solution as you can see in the picture.
When I changed her screen settings to 1024 x 768 she had no problems loading her mail. Now it is up to her to decide if she want her 15" monitor to show large fonts with the existing 800 x 600 which she like with no personal mail or change it to the smaller font with Yahoo! Mail working...

Cisco IOS 15.0 - released

Cisco released IOS 15.0. This is the next major release after 12.4. It’s been over 4 years since Cisco has delivered a major release of IOS code - 12.4 was released in May of 2005.
Looking at the list of new features there are few interesting items but none of them appeal to the average SMB.
the one feature we might find useful at the SMB level is Cisco DHCP FORCERENEW which enhances security by providing entity authentication and message authentication.
Before you can consider this new software you have to verify that your hardware is compatible and go over the new features to make sure you really need it.
I guess we won't see this new version anywhere near SMB shops because the cost (both for software and training) doesn't make sense (specially these days) while the added value is limited. But it is still good to be aware and as time pass more and more knowledge will spread around the Internet. Use this knowledge to get familiar with the new IOS because at some point it will find it's way to SMB and you want to be ready for that day.

Just Traceroute

Yesterday, just after 3pm our Internet connection had some major problems.
While browsing and most IM clients died, inbound mail still flowed and MSN Messenger was live at all times.
Testing the network I started with the immediate suspect for any Internet failure - the Proxy server. Seconds later, when I realized that Proxy was up and even when I bypass it I can't browse, I started testing my firewall and DMZ environment.
The way we're configured for Internet is a primary dual T1 connection going to Verizon's MFR router and a secondary T1 connection, both connecting to a DMZ switch which is also where my firewall hook in.
Using the trace command I was able to get all the way to the Verizon router beyond the locally installed MFR router. That indicated a Verizon side problem which is good on one side - My part of the network is working, but really frustrating on the other side - the side that need Verizon's help which is always a long process.
While on the phone with Verizon's support I was told it's a general outage and as we all know by now, it was impossible to get an explanation and\or a time frame. Luckily the problem was solved in 30 minutes and we didn't suffer significant damage.
While troubleshooting this situation I came across a cool useful website Just-traceroute that provide a platform to trace an IP from 4 different locations (USA, France, Singapore & Netherlands). This is handy to isolate many of the possible options. It also provide a built-in "send" button which you can use to email the output to yourself or any admin you're working with.

Group Policy Preferences in Windows 2008

Today I want to go over the new Group Policy Preferences that come with 2008 server.
I recently added a 2008 server in my shop and the new options are exciting!
I think it is a major improvement that worth the upgrade. The fact that you only need one server to use it make it even more appealing.
Check this great video to get a first look.
The management screen is split to Policies - where most of the old options stay and Preferences with the new options.
The next option level is sorted to 2:
Windows Settings - contain everything that required scripting in the past
Control Panel Settings - contain everything that was changeable via... Control Panel

I find the most exciting feature to be Scheduled Tasks. Yes, it is not fancy or new but it is something useful that wasn't there and actually change my life as an administrator. I can configure tasks for PCs or Servers anywhere, anytime without physically getting there. That is a major time saver.
Other features that got better and I find useful include:
Devices - allow you to enable\disable devices
Folder Options - changing the settings of a specific folder
Power Options - much better control on the power options, this make green look greener
Services - allow you to change Service configuration

Another major change is Item-level targeting, a concept that apply changes only under specified set of conditions. With Item-level targeting you can compare Registry keys, use a range of IP addresses, rely on the local PC language or domain name and few other targeting options.

Requirements:
As I said before, you do not have to upgrade all your servers, 1 2008 server would be enough.
To manage from Windows 2003 server or Vista (and Windows 7) Remote Server Administration Tools (RSAT) is required
Client side require Group Policy preferences client-side extension (CSE)

This is another step to make our life easier and make tasks easier. The domain environment is much more manageable under this new set of tools.
I just wonder if it is becoming so easy to manage the domain what would happen to us, does our expertise still count?

Free version of GFI WebMonito for ISA Server

Using ISA Server as a firewall years back and as my Proxy server for the last few years I learned to appreciate GFI's WebMonitor application.
Now it has a FREE version!
If you never tried it before because you did not have the budget or just because you never heard of it, this is the perfect opportunity.
The monitoring provide data that most small shops do not have about WHO and HOW MUCH. If you do not have any bandwidth usage data it will upgrade your capabilities and since it's free it's a win-win.
I'm using the paid version with all 3 scan engines and few rules that block downloads and specified file types. When I've installed the trial on my ISA server 2000 (yeah, many years ago) I had no budget and had to fight for the money. I wish they had this version back then...