We've seen more than a two fold increase in spam which has been backing up legitimate mail. Our spam filtering increases the delivery times significantly but as so much mail is queued right now, it seems the best thig would be to turn it off for a bit while we work on combatting the delivery of spam at the soure(s).

If you find a huge amount of spam as compared to normal, that will be due to our having temporarily turning off spam filtering.

We're still filtering for viruses and general known spam servers but our system is not inspecting the contents of the messages for spam.

When the queue gets to a more managable size, we'll turn things back on.

-tom

We're working on a permanent fix for the hardware problems we saw on Monday. With that, we had a bit of trouble earlier which caused 20 minutes of downtime and we'll likely see one more of those today as we get our final pieces in place.

UPDATE: 12:44PDT: We're having to reboot now which should give us 10 minutes or so of downtime.

UPDATE 1pm PDT: Things are not going as smoothly as expected. We're having problems at the moment and are working to get them resolved. More updates coming soon...

UPDATE 1:09pm PDT: We're back up and running so we can handle things but we still have work to do. Again, more updates when available.

UPDATE: 1:41pm PDT: Another quick reboot required. Should be back in 10 minutes...

UPDATE 1:45pm PDT: Back up again. Still not out of the woods but trying to minimize downtime.

UPDATE 2:01pm PDT: We're doing the hard work now. With our previous testing and current reading, this should go smoothly but I must stress the term "should" because of how things have gone so far today. I'll update again here as things progress.

UPDATE 3:09pm PDT: Things have been going smoothly (as we had hoped!) so far but we will need to do a quick reboot after that -- appx 20 minutes from now and lasting less than 10 mintes. We'll have a final step afterwards which I'll write about once we pass this next point. Again, things are up and running now and have been for a while but our mail is a bit backed up due to the previous outages.

UPDATE 3:22pn PDT: We're gonna do a quick reboot with a new configuration. Should be less than 10 minutes outage.

UPATE 3:39pm PDT: The reboot is not going so well with the new partition. We have some work to do here. Will update again shortly...

UPDATE 5:29pm PDT: Things having been running smoothly for a while but we've had to queue mail for a bit as we get our final pieces in place. While you can send/receive mail, most new mail is not being immediately processed. We have (what I hope is) a final reboot coming up shortly. More details to follow...

UPDATE 6:01pm PDT: We have one more reboot to handle which should take 15 minutes if all goes well. Will report as things progress.

UPDATE 6:10pm PDT: Oversight on our part makes us need to copy a file over and then reboot again. We'r looking at another 15 minutes.

6:56pm PDT: We should be good to go! I'm watching closely but I believe we're out of the woods... finally! Thanks to everyone for their patience.

-tom

We've been having a problem with the primary mail server for the last hour. We're working on it now but do not yet have an estimate for when it will be back up. I'll post here again as soon as I get an update.

UPDATE 8:40am: We're having a significant hardware failure and are working around the problem now.

UPDATE 10:20am: We've worked around the problem and mail seems to be flowing. While you'll be able to login for POP3 and IMAP, the mail that was sent to the system during the outage has been spooled on a backup server and will take several hours to filter down to your mailboxes. More info before too long...

-tom