We've been seeing problems with our spam filtering off and on for the last couple of weeks. Today the mail queue has grown large enough to where it may take several hours to catch up. We've decided to turn off the filtering as we work on the problem so that mail can get delivered quickly versus being hung up in our filtering process.

You will see a marked increase in the volume of spam that makes it to your inbox for a couple of hours today as we continue to work on things.

Once the queue empties we'll turn things back on and continue to work on the filtering problem.

-tom

We are upgrading one of our primary webservers tonight which affects a few hundred customers. If your site is on ring.vpop.net your site will be affected off and on for a few hours tonight. With the holiday weekend and this being a Sunday night it seemed the best time to pull things off. I'll update here as things progress and email for us will be working throughout so if you have questions or issues, drop us a line. I'll try and field all calls during the process as well. If I miss you, please let me know what time is appropriate for a call back and I'll do my best to get to you then.

[UPDATE 1:20am]: The upgrade has gone well and most services are up and running. We have some tweaking to do but web sites are being served.

[Update 8:10am]: We've needed to recompile the webserver software so while the system is up it is not currently serving content.

[Update 9:35am]: We're still working on the webserver software update but are running into issues so we still are not serving web site content. Will update again shortly.

[Update 11am]: Things seem to be back up and running. There was a very subtle change in the way our webserver works with PHP between versions and it took a lot longer to find than it did to fix.

We're still rebuilding some modules but sites are up and running at this point.

-tom

After the reboot yesterday we apparently caused a problem for some customers which borke their ability to check email using POP3 clients such as Outlook or Thunderbird. The problem would have been encountered if your POP3 Incoming mail server was set to "mail.yourname.com" as opposed to "mail.vpop.net".

The issue has been resolved and I apologize to everyone affected. I'm catching up on returning the phone calls and emails now.

-tom

[Update 17:15 PST] Things look to be in good shape. Thanks for your patience!
-tom

[Update 17:00 PST] Rebooting now.

[Update 16:25 PST] We'll be rebooting in ~15 minutes.

[Original] We need to reboot one of our mail servers due to a security issue that has been announced for its operating system. Not exactly sure of the time but will update here when we know specifically. Guessing 1.5 hours from now but need to wait for various "build" processes to complete.

-tom

There appears to be a network outage at our Downtown LA, Ca. server facility. We are working on the problem with our upstream provider and will post an update here as soon as we know more.

The implication of this outage is that virtually all services are unavailable, including access to email, web sites, etc.

Update: The problem has been solved. Our building managers inadvertently cut power to the "switch" which connects us to the outside world. After some investigation they were able to find the problem and fix the situation. Unfortunately we were offline for appx. 1 hour and 20 minutes.

My apologies to all affected.:(

-tom

We are aware that there is a problem with our main webserver and we are working to get it up and running as soon as possible. We will post updates as soon as we know more

[UPDATE 4:30] System is back up and running.

We updated MySQL and only saw a couple of seconds of downtime as the server restarted. We should be in good shape but definitely let me know if you see any funkiness with your database driven web site.

-tom

Hello All,

We're upgrading MySQL today which should result in appx. 60 seconds (or less) of downtime for services which rely on that DB. This means, if you have a database driven web site, you'll see a few errors during that time frame.

Will post again when the process has been completed and tested.

-tom

A main webserver (ring.vpop.net) is having a problem right now and we are investigating. Will post here with updates.

UPDATE: 12:32: Things are back to normal. There were a few processes that were consuming too many resources and it caused the webserver (Apache) to lag to the point where it was not serving pages. Looks like we were down for appx. 8 minutes.

Apologies for the outage.:(
-tom

One of our primary webserver's is taking an extrmely long time to respond and is, for the most part, down. This means that your site may take a long time to display or might not display at all. We are investigating the problem now.

UPDATE 11:10am: We have the system back up and running. Still investigating the cause but the problem seems to be gone for now.

-tom

We've seen more than a two fold increase in spam which has been backing up legitimate mail. Our spam filtering increases the delivery times significantly but as so much mail is queued right now, it seems the best thig would be to turn it off for a bit while we work on combatting the delivery of spam at the soure(s).

If you find a huge amount of spam as compared to normal, that will be due to our having temporarily turning off spam filtering.

We're still filtering for viruses and general known spam servers but our system is not inspecting the contents of the messages for spam.

When the queue gets to a more managable size, we'll turn things back on.

-tom

We're working on a permanent fix for the hardware problems we saw on Monday. With that, we had a bit of trouble earlier which caused 20 minutes of downtime and we'll likely see one more of those today as we get our final pieces in place.

UPDATE: 12:44PDT: We're having to reboot now which should give us 10 minutes or so of downtime.

UPDATE 1pm PDT: Things are not going as smoothly as expected. We're having problems at the moment and are working to get them resolved. More updates coming soon...

UPDATE 1:09pm PDT: We're back up and running so we can handle things but we still have work to do. Again, more updates when available.

UPDATE: 1:41pm PDT: Another quick reboot required. Should be back in 10 minutes...

UPDATE 1:45pm PDT: Back up again. Still not out of the woods but trying to minimize downtime.

UPDATE 2:01pm PDT: We're doing the hard work now. With our previous testing and current reading, this should go smoothly but I must stress the term "should" because of how things have gone so far today. I'll update again here as things progress.

UPDATE 3:09pm PDT: Things have been going smoothly (as we had hoped!) so far but we will need to do a quick reboot after that -- appx 20 minutes from now and lasting less than 10 mintes. We'll have a final step afterwards which I'll write about once we pass this next point. Again, things are up and running now and have been for a while but our mail is a bit backed up due to the previous outages.

UPDATE 3:22pn PDT: We're gonna do a quick reboot with a new configuration. Should be less than 10 minutes outage.

UPATE 3:39pm PDT: The reboot is not going so well with the new partition. We have some work to do here. Will update again shortly...

UPDATE 5:29pm PDT: Things having been running smoothly for a while but we've had to queue mail for a bit as we get our final pieces in place. While you can send/receive mail, most new mail is not being immediately processed. We have (what I hope is) a final reboot coming up shortly. More details to follow...

UPDATE 6:01pm PDT: We have one more reboot to handle which should take 15 minutes if all goes well. Will report as things progress.

UPDATE 6:10pm PDT: Oversight on our part makes us need to copy a file over and then reboot again. We'r looking at another 15 minutes.

6:56pm PDT: We should be good to go! I'm watching closely but I believe we're out of the woods... finally! Thanks to everyone for their patience.

-tom

We've been having a problem with the primary mail server for the last hour. We're working on it now but do not yet have an estimate for when it will be back up. I'll post here again as soon as I get an update.

UPDATE 8:40am: We're having a significant hardware failure and are working around the problem now.

UPDATE 10:20am: We've worked around the problem and mail seems to be flowing. While you'll be able to login for POP3 and IMAP, the mail that was sent to the system during the outage has been spooled on a backup server and will take several hours to filter down to your mailboxes. More info before too long...

-tom

It seems that the network connected to VPOP's system is completely down. This means that no mail or web traffic is making it in or out. We are working with our upstream provider now to get the problem resolved and will post an update here as soon as we have one.
-------------------

UPDATE [9:24am] Things are back up. Although the outage was less than 15 minutes it was a complete outage. I'm still trying to find out the details from our upstream but for now, we're up.

-tom

Over the last week we've seen intermittent but severe slowdowns with web site delivery. The problem is with the router of our upstream provider (the company that sells us our internet connection). We have been working steadily to get the issue corrected but response has been slow and no permanent solution has been found.

We are working on multiple solutions now at as great a pace as is possible. As we get more information about our specific course of action I will post the details here.

-tom

A primary webserver is offline (ring.vpop.net) and is being worked on. I hope to have it back up in the next 15 minutes or so.

[UPDATE 3:10am] We're back up though it took a bit longer than expected.:(

Thanks,
-tom

The same webserver is having issues again. We've been working on the problem but have yet to get it resolved. I'll post here again as information is available. My apologies for the delay in getting this noticed posted late. :(
-tom

[UPDATE:] The system is back up for now but it appears that we are having an intermittent hardware problem. We should be replacing the hardware late this morning. I post to this area when I have a firm time.

[UPDATE:] The system is still having problems and needs to be taken down again. More updates shortly...

[UPDATE:] We're up and running for the moment but will have to take the system down again in the late morning to replace the hardware.

[UPDATE 11:48am:] We will be doing a quick reboot at noon to put us in a better position to replace our hardware and gather additional information. We should see appx. 12 minutes of downtime during this procedure.

[UPDATE 2pm:] The previous reboot ended up taking appx. 4 minutes. We're in much better shape but still have some testing to do which will require some downtime for reboots. We plan on initiating these late tonight so as to minimize the general impact on things. We should all wake up to a more permanent solution to the problems we've seen today.

Once again, my apologies for the outage and extra work these outages must cause you as a result.
-tom

A primary webserver stopped responding at appx. 11:35 this evening and we were forced to reboot it. It is on its way back up though the file system checks will take 20 minutes or so to complete before it is fully operational. More details shortly.

[UPDATE 12:15] The server is up and running normally now. We're investigating the issue further as I type this message.

One of our main webservers has received more traffic than it can handle and is not responding well right now. We are working on the problem and will update here as soon as we have more information.

[UPDATE:] Things seem to be back and running normally.

-tom

With last night's reboot we found some inconsistencies with Perl modules and are upgrading both the system installation of Perl as well as the supporting libraries. This means there will be some intermittent CGI failures if you are using Perl and your needed libraries are still in the process of rebuilding.

[Update: 6:29pm PDT] The rebuild process has completed and things are looking to be in decent shape. We still have a bit of clean-up but by-and-large things are better off.

-tom

We've had to reboot one of the primary webservers which will cause approximately 30 minutes of downtime. More news to be posted shortly.

Our mail server move will commence in just a few minutes. The rate and amount of spam hitting our system has gone beyond what the old system could handle and this move to a new server should alleviate any pain.

You may notice up to 30 minutes of inability to authenticate via POP3/IMAP. We will be spooling mail during the outage.

UPDATE: The maill server seems to be up and functioning now. There may be some lag time due to DNS issues where webmail is accessible but POP3 and IMAP service should be up and running. We saw just under 20 minutes of downtime.

-tom

Our spam filtering system seems to be having a problem which is causing mail to be delayed far longer than is acceptable. In order to get mail out of the queue and delivered properly we have disabled the spam filtering system. We will reenable it shortly after the queue empties but for this morning you will likely see far more spam than usual.

-tom

For those of you who were notified of your server being moved, we will begin the process in appx. 5 minutes or, 12:15am (PST) Nov. 12, 2005. Web and Email access will not be functional during this time though mail will be queued for delivery until the servers are back up and running.

Please check this page often as I'will be posting updates as they become available.

UPDATE: Web serving is up as of appx. 2:15am.

UPDATE 2: Mail should be working as well though it will take several hours for all of the queued mail to be delivered.

Thanks,
-tom

Email delivery has been delayed by several hours because our spam filtering process had problems. Email that should have been delivered quickly was deferred because it could not be processed properly by the spam filtering system.

The spam filtering system has been repaired and mail is starting to get delivered but it will be several hours before the deferred queue is emptied.

UPDATE: the system seems to be up and running normally now. The downtime seems to have lasted for just over 2 hours. We are still investigating the core cause of the outage and once it is identified we will take measures to avoid the situation in the future. My apologies for the outage and the delay in getting this site updated in a more timely manner.

We've been having a problem with one of our main webservers which also affects email for many customers. We are still working on the problem and hope to have it back up and running shortly. The affected server is "ring.vpop.net".

It's been quite a day today and I know extremely frustrating for many of you. I have some details about what has been going on today, what we can expect this evening and why it has taken so long to update this page.

Last things first: We purposely host our System Status site on a different location so if something goes wrong with our core network we can still provide you with updates. Unfortunately, in addition to our own problems, the service which hosts our System Status page was performing maintenance so we were unable to update this page.

Early this morning our main spam filtering system went down. Due to the amount of spam we are filtering these days mail can get backed up quite quickly. While no mail was lost, much of it has been delayed and for several hours. As a way to increase delivery speed we have temporarily turned off spam filtering, until the mail queue empties. We do have backup spam filtering capabilities but unfortunately they can no longer handle the sheer volume of junk that comes in, in a timely manner, so we'll need to remedy this situation.

Web site statistics are also affected at this time. Again, while no data was lost, we'll need to reprocess the logs from last night and then remount that partition so your stats are again viewable in the usual location.

We are still working on some of the problems and I will post an update when things are back and fully functional.

Once again, I appreciate your patience and apologize for the frustration you have had throughout the day.

Tom McDonald

There has been a vulnerability discovered which needs to be addressed immediately on all of our servers and this requires a reboot. Each machine can take up to 20 minutes to complete the reboot process. Some machines will be handled this afternoon while the primary systems will be handled at 10pm PDT tonight.

If you notice problems today, please give appx. 20 minutes and test again, just in case you are trying to access things while a system is being rebooted.

As always, thanks for your patience!

Tom McDonald, President,
VPOP Technologies, Inc.

Today we had an outage which seems to have lasted appx. 20 minutes. This affected a core server for many web sites hosted by VPOP. The problem was related to human error (menaning, me) while managing the system. Affected system: ring.vpop.net

The outage was related to serving web sites and providing email access to many customers.

My apologies for the delay and downtime.:(

Thanks,
Tom McDonald, President

We'll be upgrading our spam filtering system today which should lower "false positives" and catch a higher percentage of spam before you ever see it. There may be a few minutes here and there where delivery of mail will be delayed but it should not be noticeable to most. We'll be watching the results closely after the upgrade is complete. Please don't hesitate to contact us if you feel delays are occurring for longer than just a couple of minutes today.

This morning we are seeing delays with email of several hours. The DNS server stopped responding which caused the mail server to not know where to deliver its mail, resulting in the mail being spooled. We have restarted the DNS server and mail is flowing through but there is a fair sized backup of mail in the spool which will take a few hours to catch up. We saw this same behavior last Friday evening as well. To prevent this type of failure in the future we have put a process in place to verify that the DNS server is responding properly and if not, it will automatically be restarted.

My apologies for the mail delay again.
-tom