Thursday, November 02, 2006

Exchange 2003 in real trouble

Hi,

Just wanted to share an experience I had the other day at a customer's network (a small company with around 15 guys in the sales department and a few more in administration etc.).

The local IT responsible called me and asked me if I had an idea why their Internet connection was extremely slow that Monday - it had been so since around 10:40 the same morning... Hmm, I had no idea to begin with, so we had to do some troubleshooting over the phone.

First we did some ICMP echo requests against some remote servers and I could only agree, things didn't look good - so the question was: Is this because of the ISP or some local problem?

Well, the ISP had no problems in the area they said (but it seems they never have any problems - officially). So I set up some monitoring in their Sonicwall firewall and we quickly spotted heavy action from their Exchange 2003 server. He tried to restart the server and after half an hour it actually restarted - it was under heavy load! After the reboot the network performance went bad again, so I asked him to go into Exchange System Manager (ESM) and take a look at the queues...

The queue situation turned out to be pretty ugly and it looked like the server was an open relay, hit by some SPAM/hacker attack or whatever - even though I was pretty sure everything was setup correctly and in a secure manner when I installed the server. So I grabbed my jacket and went to the customer's location.

When I arrived I could see almost 1000 queues left in retry state, each with a single message of around 1 MB. The server was responding extremely slowly and every mouse click was like an eternity. When I took a closer look at the emails it seemed as though they were all sent from an internal user, let's call him Mr. Spammer! I feared some malware had entered his computer - even though I knew he was not a local admin, they had antivirus & -spyware running, XP built-in firewall activated etc.

I went to the users seat an unplugged his computer, looked in his Event Log, running processes/services and Outlook sent items - and there it was... At 10.40 this guy actually produced an email for a high number of external receivers, probably above one thousand... That should be OK, but the thing is, he attached a 1 MB picture - this changed the story completely. Exchange simply couldn't handle that amount of traffic.

So now I was sure this was not an attack or anything - I was pretty relieved I can tell you - it was now "just" a local user who acted like a SPAMMER... An internal attack you could say. Well, I never saw that one coming!

Now I had one problem - the queues didn't go away. Restart of SMTP service didn't work, the "Default SMTP Virtual Server" queue (and pickup) directory was empty so I was left with the manual approach of deleting every item in each queue.... Or using the AQADMCLI tool.

This tool is an admins friend in situations like mine. I downloaded the tool from another server, placed it on a floppy, copied to the (offline) Exchange server and executed the file. This gives you a Command Prompt, and all I had to do was to type:

setserver SERVERNAME
delmsg flags=all

This displayed a lot of entries about queues being flushed - or messages being deleted - and then all I had to do was to type in quit. Problem solved!

So todays blog is just a small tribute to AQADMCLI and it's authors - I hope this info will come in handy someday in your admin-life :)

You can read more about the tool here.

6 Comments:

Anonymous said...

Hello Jakob,

Very nice blogspot man, how can u find time to post topics? :)

Anyway this is a very valuable topic to share. I face a similar case couple months ago; only the reasons was different, actually server was an open relay, and hit by some SPAM/hacker attack. I leave it for the security product vendor people. But what if there is no such product, what is the right step to make??

Any advices?

Regards,

New Friend

Jakob H. Heidelberg said...

Hi!
Thanx - time is short these days, but I like to blog whenever I can :)

You ask me "what is the right step to make" when there is no security (I guess this covers scan engines of all kinds) product on the server and you are then hit... An answer to that question is very hard to give - mainly because it's different from case to case (depending on the attack). In most cases what I would do is to reinstall the server from scratch and then recover the mailboxes.

If a computer has been hit by a virus, trojan, rootkit, spyware, malware etc. there is only on thing to do if you ask me: isolate, analyze problem and then probably reinstall! I never trust such a compromised machine again - even though it might be an expensive faulttolerant cluster server worth thousands of Dollars :-)

But, but, but - if the problem was "only" Open Relay, then it is enough just to close down the relay functionality - it's not really a 'security issue' in itself, it's more like a (bad) configuration/setup (that could bring 'security issues' along the way: Denial of Service etc.). After making sure the machine is not an Open Relay you could use the following links to confirm the setup is OK:

http://www.abuse.net/relay.html
http://www.spamhelp.org/shopenrelay
http://www.mxtoolbox.com

Microsoft on Open Relay:
http://support.microsoft.com/kb/304897

Osama Al Khalili said...

Hello Master,

Many Thanx for you man for deep analysis.
I'll follow Microsoft link for more satisfaction ;)

By the way; I've found out your blogspot through your articles on www.windowsecurity.com, very precise piece of art :D

One more thing! VISTA version I have is no more working :S its a Beta version.. Can u give me a link or something to get the latest available version?

I'll be very grateful.


Regards,

Your Student

Jakob H. Heidelberg said...

Hi Osama!
Thanx for the kind words.

Regarding the Vista BETA I'm sorry but I can't help you. Guess you need MSDN or Technet to get an RTM version these days.

rmooney01 said...

Jacob,
Just wanted to say thank you! I have had this exact problem today and it has taken me hours to find your blog, but it appears that this little utility works great. When the directions on using it are prestented correctly as you have them.

I thank you, and my company thanks you!!!!!1

rmooney01 said...

Jakob,

This is exactly the situation I've had today. A person sent out a 5mb file to 800 recipients yesterday and has brought our office to its knees. I just used this utility and restarted our exchange server and from what I saw it was doing it's job. I had found that same utility on another site, but thier direction on using it was poor. You have done admins a great service.

I thank you, and my company thanks you.

Rob