UPDATE: the system seems to be up and running normally now. The downtime seems to have lasted for just over 2 hours. We are still investigating the core cause of the outage and once it is identified we will take measures to avoid the situation in the future. My apologies for the outage and the delay in getting this site updated in a more timely manner.

We've been having a problem with one of our main webservers which also affects email for many customers. We are still working on the problem and hope to have it back up and running shortly. The affected server is "ring.vpop.net".

It's been quite a day today and I know extremely frustrating for many of you. I have some details about what has been going on today, what we can expect this evening and why it has taken so long to update this page.

Last things first: We purposely host our System Status site on a different location so if something goes wrong with our core network we can still provide you with updates. Unfortunately, in addition to our own problems, the service which hosts our System Status page was performing maintenance so we were unable to update this page.

Early this morning our main spam filtering system went down. Due to the amount of spam we are filtering these days mail can get backed up quite quickly. While no mail was lost, much of it has been delayed and for several hours. As a way to increase delivery speed we have temporarily turned off spam filtering, until the mail queue empties. We do have backup spam filtering capabilities but unfortunately they can no longer handle the sheer volume of junk that comes in, in a timely manner, so we'll need to remedy this situation.

Web site statistics are also affected at this time. Again, while no data was lost, we'll need to reprocess the logs from last night and then remount that partition so your stats are again viewable in the usual location.

We are still working on some of the problems and I will post an update when things are back and fully functional.

Once again, I appreciate your patience and apologize for the frustration you have had throughout the day.

Tom McDonald

There has been a vulnerability discovered which needs to be addressed immediately on all of our servers and this requires a reboot. Each machine can take up to 20 minutes to complete the reboot process. Some machines will be handled this afternoon while the primary systems will be handled at 10pm PDT tonight.

If you notice problems today, please give appx. 20 minutes and test again, just in case you are trying to access things while a system is being rebooted.

As always, thanks for your patience!

Tom McDonald, President,
VPOP Technologies, Inc.

Today we had an outage which seems to have lasted appx. 20 minutes. This affected a core server for many web sites hosted by VPOP. The problem was related to human error (menaning, me) while managing the system. Affected system: ring.vpop.net

The outage was related to serving web sites and providing email access to many customers.

My apologies for the delay and downtime.:(

Thanks,
Tom McDonald, President

We'll be upgrading our spam filtering system today which should lower "false positives" and catch a higher percentage of spam before you ever see it. There may be a few minutes here and there where delivery of mail will be delayed but it should not be noticeable to most. We'll be watching the results closely after the upgrade is complete. Please don't hesitate to contact us if you feel delays are occurring for longer than just a couple of minutes today.

This morning we are seeing delays with email of several hours. The DNS server stopped responding which caused the mail server to not know where to deliver its mail, resulting in the mail being spooled. We have restarted the DNS server and mail is flowing through but there is a fair sized backup of mail in the spool which will take a few hours to catch up. We saw this same behavior last Friday evening as well. To prevent this type of failure in the future we have put a process in place to verify that the DNS server is responding properly and if not, it will automatically be restarted.

My apologies for the mail delay again.
-tom

We will be rebooting one of our main web servers today (11:00 a.m. PDT) to take advantage of an additional processor in the computer. I expect that system to be down for appx. 15 minutes, from shutdown to fully active web serving. Email will also be delayed during this time period. The additional processor will significantly increase the speed and reliability of the system.

UPDATE: Everything appears to have gone smoothly. We got started just a few minutes later than expected and had down time of appx 15 minutes. Thanks all for your patience!

We'll be moving our primary email server to a faster machine today. This will cause POP3 and IMAP service to be unavailable for appx 15 30 minutes. All incoming mail will be queued for the few minutes that we are making the final move.

The new mail system will provide faster processing and more reliable logins. The webmail interface will also see a nice improvement in speed and reliability.

UPDATE: The move is complete. It looks to have taken closer to 40 minutes. We have a few corner cases to deal with but for the general mail account, everything is in good shape. Thanks for your patience during the move. And please do let us know if you see any problems.

We will need to reboot the system shortly to take care of a kernel bug. This affects you if your site is hosted on ring.vpop.net. The anticipated downtime is less than 30 minutes.

UPDATE: The system maintenance has been completed. We were down from appx 12:21PDT through 12:38PDT. Thanks again for your patience!

We saw a problem with our primary email handler early yesterday morning which continued through the late morning. Mail was not lost but rather, it was spooled for delivery once the mail system came back up. New email messages have been flowing fine since yesterday afternoon but those messages in the mail queue from the outage are trickling out at a terribly slow pace. We are working on getting that process to speed up and hope to have all queued mail delivered shortly.

During this process we have temporarily turned off spam filtering. Once everything is caught up we will turn the spam filtering system back on and we should be in good shape.

UPDATE: The mail queue has emptied and we turned the spam filters back on. We are monitoring things closely and right now, things are looking very good. Thanks for your patience!

There is a new virus out there which is being sent to domain owners across the internet.  It may claim to be "From:" your domain admin or even from "staff@vpop.net" (or some other authentic looking address).  The message will contain a ZIP or EXE attachment and request that you install it for protection.  Be assured that VPOP will never send you a message with a file attached that you should install -- Never.  You should be extremely careful when opening any attachment but be especially careful when you receive one which you did not expect.

We are currently blocking most (if not all) instances of this type of message due to our spam and virus filtering but it is possible that something will make it through the defense system.  Again, please be cautious when opening any email and even more so when opening an attachment of any sort.