One of our primary webserver's is taking an extrmely long time to respond and is, for the most part, down. This means that your site may take a long time to display or might not display at all. We are investigating the problem now.

UPDATE 11:10am: We have the system back up and running. Still investigating the cause but the problem seems to be gone for now.

-tom

We've seen more than a two fold increase in spam which has been backing up legitimate mail. Our spam filtering increases the delivery times significantly but as so much mail is queued right now, it seems the best thig would be to turn it off for a bit while we work on combatting the delivery of spam at the soure(s).

If you find a huge amount of spam as compared to normal, that will be due to our having temporarily turning off spam filtering.

We're still filtering for viruses and general known spam servers but our system is not inspecting the contents of the messages for spam.

When the queue gets to a more managable size, we'll turn things back on.

-tom

We're working on a permanent fix for the hardware problems we saw on Monday. With that, we had a bit of trouble earlier which caused 20 minutes of downtime and we'll likely see one more of those today as we get our final pieces in place.

UPDATE: 12:44PDT: We're having to reboot now which should give us 10 minutes or so of downtime.

UPDATE 1pm PDT: Things are not going as smoothly as expected. We're having problems at the moment and are working to get them resolved. More updates coming soon...

UPDATE 1:09pm PDT: We're back up and running so we can handle things but we still have work to do. Again, more updates when available.

UPDATE: 1:41pm PDT: Another quick reboot required. Should be back in 10 minutes...

UPDATE 1:45pm PDT: Back up again. Still not out of the woods but trying to minimize downtime.

UPDATE 2:01pm PDT: We're doing the hard work now. With our previous testing and current reading, this should go smoothly but I must stress the term "should" because of how things have gone so far today. I'll update again here as things progress.

UPDATE 3:09pm PDT: Things have been going smoothly (as we had hoped!) so far but we will need to do a quick reboot after that -- appx 20 minutes from now and lasting less than 10 mintes. We'll have a final step afterwards which I'll write about once we pass this next point. Again, things are up and running now and have been for a while but our mail is a bit backed up due to the previous outages.

UPDATE 3:22pn PDT: We're gonna do a quick reboot with a new configuration. Should be less than 10 minutes outage.

UPATE 3:39pm PDT: The reboot is not going so well with the new partition. We have some work to do here. Will update again shortly...

UPDATE 5:29pm PDT: Things having been running smoothly for a while but we've had to queue mail for a bit as we get our final pieces in place. While you can send/receive mail, most new mail is not being immediately processed. We have (what I hope is) a final reboot coming up shortly. More details to follow...

UPDATE 6:01pm PDT: We have one more reboot to handle which should take 15 minutes if all goes well. Will report as things progress.

UPDATE 6:10pm PDT: Oversight on our part makes us need to copy a file over and then reboot again. We'r looking at another 15 minutes.

6:56pm PDT: We should be good to go! I'm watching closely but I believe we're out of the woods... finally! Thanks to everyone for their patience.

-tom

We've been having a problem with the primary mail server for the last hour. We're working on it now but do not yet have an estimate for when it will be back up. I'll post here again as soon as I get an update.

UPDATE 8:40am: We're having a significant hardware failure and are working around the problem now.

UPDATE 10:20am: We've worked around the problem and mail seems to be flowing. While you'll be able to login for POP3 and IMAP, the mail that was sent to the system during the outage has been spooled on a backup server and will take several hours to filter down to your mailboxes. More info before too long...

-tom

It seems that the network connected to VPOP's system is completely down. This means that no mail or web traffic is making it in or out. We are working with our upstream provider now to get the problem resolved and will post an update here as soon as we have one.
-------------------

UPDATE [9:24am] Things are back up. Although the outage was less than 15 minutes it was a complete outage. I'm still trying to find out the details from our upstream but for now, we're up.

-tom

Over the last week we've seen intermittent but severe slowdowns with web site delivery. The problem is with the router of our upstream provider (the company that sells us our internet connection). We have been working steadily to get the issue corrected but response has been slow and no permanent solution has been found.

We are working on multiple solutions now at as great a pace as is possible. As we get more information about our specific course of action I will post the details here.

-tom

A primary webserver is offline (ring.vpop.net) and is being worked on. I hope to have it back up in the next 15 minutes or so.

[UPDATE 3:10am] We're back up though it took a bit longer than expected.:(

Thanks,
-tom

The same webserver is having issues again. We've been working on the problem but have yet to get it resolved. I'll post here again as information is available. My apologies for the delay in getting this noticed posted late. :(
-tom

[UPDATE:] The system is back up for now but it appears that we are having an intermittent hardware problem. We should be replacing the hardware late this morning. I post to this area when I have a firm time.

[UPDATE:] The system is still having problems and needs to be taken down again. More updates shortly...

[UPDATE:] We're up and running for the moment but will have to take the system down again in the late morning to replace the hardware.

[UPDATE 11:48am:] We will be doing a quick reboot at noon to put us in a better position to replace our hardware and gather additional information. We should see appx. 12 minutes of downtime during this procedure.

[UPDATE 2pm:] The previous reboot ended up taking appx. 4 minutes. We're in much better shape but still have some testing to do which will require some downtime for reboots. We plan on initiating these late tonight so as to minimize the general impact on things. We should all wake up to a more permanent solution to the problems we've seen today.

Once again, my apologies for the outage and extra work these outages must cause you as a result.
-tom

A primary webserver stopped responding at appx. 11:35 this evening and we were forced to reboot it. It is on its way back up though the file system checks will take 20 minutes or so to complete before it is fully operational. More details shortly.

[UPDATE 12:15] The server is up and running normally now. We're investigating the issue further as I type this message.

One of our main webservers has received more traffic than it can handle and is not responding well right now. We are working on the problem and will update here as soon as we have more information.

[UPDATE:] Things seem to be back and running normally.

-tom

With last night's reboot we found some inconsistencies with Perl modules and are upgrading both the system installation of Perl as well as the supporting libraries. This means there will be some intermittent CGI failures if you are using Perl and your needed libraries are still in the process of rebuilding.

[Update: 6:29pm PDT] The rebuild process has completed and things are looking to be in decent shape. We still have a bit of clean-up but by-and-large things are better off.

-tom

We've had to reboot one of the primary webservers which will cause approximately 30 minutes of downtime. More news to be posted shortly.

Our mail server move will commence in just a few minutes. The rate and amount of spam hitting our system has gone beyond what the old system could handle and this move to a new server should alleviate any pain.

You may notice up to 30 minutes of inability to authenticate via POP3/IMAP. We will be spooling mail during the outage.

UPDATE: The maill server seems to be up and functioning now. There may be some lag time due to DNS issues where webmail is accessible but POP3 and IMAP service should be up and running. We saw just under 20 minutes of downtime.

-tom

Our spam filtering system seems to be having a problem which is causing mail to be delayed far longer than is acceptable. In order to get mail out of the queue and delivered properly we have disabled the spam filtering system. We will reenable it shortly after the queue empties but for this morning you will likely see far more spam than usual.

-tom

For those of you who were notified of your server being moved, we will begin the process in appx. 5 minutes or, 12:15am (PST) Nov. 12, 2005. Web and Email access will not be functional during this time though mail will be queued for delivery until the servers are back up and running.

Please check this page often as I'will be posting updates as they become available.

UPDATE: Web serving is up as of appx. 2:15am.

UPDATE 2: Mail should be working as well though it will take several hours for all of the queued mail to be delivered.

Thanks,
-tom

Email delivery has been delayed by several hours because our spam filtering process had problems. Email that should have been delivered quickly was deferred because it could not be processed properly by the spam filtering system.

The spam filtering system has been repaired and mail is starting to get delivered but it will be several hours before the deferred queue is emptied.

UPDATE: the system seems to be up and running normally now. The downtime seems to have lasted for just over 2 hours. We are still investigating the core cause of the outage and once it is identified we will take measures to avoid the situation in the future. My apologies for the outage and the delay in getting this site updated in a more timely manner.

We've been having a problem with one of our main webservers which also affects email for many customers. We are still working on the problem and hope to have it back up and running shortly. The affected server is "ring.vpop.net".

It's been quite a day today and I know extremely frustrating for many of you. I have some details about what has been going on today, what we can expect this evening and why it has taken so long to update this page.

Last things first: We purposely host our System Status site on a different location so if something goes wrong with our core network we can still provide you with updates. Unfortunately, in addition to our own problems, the service which hosts our System Status page was performing maintenance so we were unable to update this page.

Early this morning our main spam filtering system went down. Due to the amount of spam we are filtering these days mail can get backed up quite quickly. While no mail was lost, much of it has been delayed and for several hours. As a way to increase delivery speed we have temporarily turned off spam filtering, until the mail queue empties. We do have backup spam filtering capabilities but unfortunately they can no longer handle the sheer volume of junk that comes in, in a timely manner, so we'll need to remedy this situation.

Web site statistics are also affected at this time. Again, while no data was lost, we'll need to reprocess the logs from last night and then remount that partition so your stats are again viewable in the usual location.

We are still working on some of the problems and I will post an update when things are back and fully functional.

Once again, I appreciate your patience and apologize for the frustration you have had throughout the day.

Tom McDonald

There has been a vulnerability discovered which needs to be addressed immediately on all of our servers and this requires a reboot. Each machine can take up to 20 minutes to complete the reboot process. Some machines will be handled this afternoon while the primary systems will be handled at 10pm PDT tonight.

If you notice problems today, please give appx. 20 minutes and test again, just in case you are trying to access things while a system is being rebooted.

As always, thanks for your patience!

Tom McDonald, President,
VPOP Technologies, Inc.

Today we had an outage which seems to have lasted appx. 20 minutes. This affected a core server for many web sites hosted by VPOP. The problem was related to human error (menaning, me) while managing the system. Affected system: ring.vpop.net

The outage was related to serving web sites and providing email access to many customers.

My apologies for the delay and downtime.:(

Thanks,
Tom McDonald, President

We'll be upgrading our spam filtering system today which should lower "false positives" and catch a higher percentage of spam before you ever see it. There may be a few minutes here and there where delivery of mail will be delayed but it should not be noticeable to most. We'll be watching the results closely after the upgrade is complete. Please don't hesitate to contact us if you feel delays are occurring for longer than just a couple of minutes today.

This morning we are seeing delays with email of several hours. The DNS server stopped responding which caused the mail server to not know where to deliver its mail, resulting in the mail being spooled. We have restarted the DNS server and mail is flowing through but there is a fair sized backup of mail in the spool which will take a few hours to catch up. We saw this same behavior last Friday evening as well. To prevent this type of failure in the future we have put a process in place to verify that the DNS server is responding properly and if not, it will automatically be restarted.

My apologies for the mail delay again.
-tom

We will be rebooting one of our main web servers today (11:00 a.m. PDT) to take advantage of an additional processor in the computer. I expect that system to be down for appx. 15 minutes, from shutdown to fully active web serving. Email will also be delayed during this time period. The additional processor will significantly increase the speed and reliability of the system.

UPDATE: Everything appears to have gone smoothly. We got started just a few minutes later than expected and had down time of appx 15 minutes. Thanks all for your patience!

We'll be moving our primary email server to a faster machine today. This will cause POP3 and IMAP service to be unavailable for appx 15 30 minutes. All incoming mail will be queued for the few minutes that we are making the final move.

The new mail system will provide faster processing and more reliable logins. The webmail interface will also see a nice improvement in speed and reliability.

UPDATE: The move is complete. It looks to have taken closer to 40 minutes. We have a few corner cases to deal with but for the general mail account, everything is in good shape. Thanks for your patience during the move. And please do let us know if you see any problems.

We will need to reboot the system shortly to take care of a kernel bug. This affects you if your site is hosted on ring.vpop.net. The anticipated downtime is less than 30 minutes.

UPDATE: The system maintenance has been completed. We were down from appx 12:21PDT through 12:38PDT. Thanks again for your patience!

We saw a problem with our primary email handler early yesterday morning which continued through the late morning. Mail was not lost but rather, it was spooled for delivery once the mail system came back up. New email messages have been flowing fine since yesterday afternoon but those messages in the mail queue from the outage are trickling out at a terribly slow pace. We are working on getting that process to speed up and hope to have all queued mail delivered shortly.

During this process we have temporarily turned off spam filtering. Once everything is caught up we will turn the spam filtering system back on and we should be in good shape.

UPDATE: The mail queue has emptied and we turned the spam filters back on. We are monitoring things closely and right now, things are looking very good. Thanks for your patience!

There is a new virus out there which is being sent to domain owners across the internet.  It may claim to be "From:" your domain admin or even from "staff@vpop.net" (or some other authentic looking address).  The message will contain a ZIP or EXE attachment and request that you install it for protection.  Be assured that VPOP will never send you a message with a file attached that you should install -- Never.  You should be extremely careful when opening any attachment but be especially careful when you receive one which you did not expect.

We are currently blocking most (if not all) instances of this type of message due to our spam and virus filtering but it is possible that something will make it through the defense system.  Again, please be cautious when opening any email and even more so when opening an attachment of any sort.