In this case a spammer had successfully exploited a vulnerability in the server code which allowed the sender to use the client's machine for sending hundreds of thousands of spam. The tremendous load on the server caused by the outgoing mail, replies of undeliverable mail, and the ensuing hate mail crippled the other services provided by the machine.
Solution -- This was a rather simple disaster to recover from once we were finally able to access the machine. At the time, this client was running SunOS and the process table overflowed, creating a denial of service where we couldn't create a new login process as Root. Once we regained access to the machine, the fix was to update the vulnerable version of Sendmail, ensure that the configuration didn't allow relaying from anything but the local network.
A very important lesson learned here was the extreme importance in separating your services amongst multiple servers. The web site shouldn't have been down because of a problem with the e-mail server. This client decided that it was much cheaper to invest in a couple new servers than to have his source of income shut down unnecessarily. The third phase of this solution was to check the system for other vulnerabilities to e-mail attacks. We found a CGI program on the site that was named FormMail. This is a file that is exploited by spammers regularly.
For information and assistance on these or any other computer related issues, you can contact us by: E-Mail, or by calling 775/741-8278.