We're moving the main hosting server to a nice new machine which will cause a bit of downtime as we copy the last version of changed files on the system.

We are also changing IP#s which should be transparent to all as we'll update the DNS for the domains for which we provide this service and have a tunnel from the old machine to the new one while we work with the few people who handle their own DNS or have it hosted elsewhere. Will update as things progress.

Email should not be affected during this period.

Thanks,
-tom

UPDATE: We've completed the move and things seem to be working normally.

We're seeing an issue with our spam filtering taking way too long to process messages which has caused quite a bit of mail to be backed up. We're working on the problem now and will post here when we have a better idea for how long it will take to empty the queue of messages.

UPDATE 9:49am PST: Mail continues to be handled painfully slow which means most messages are being queued for delivery. We're narrowing the problem but no ETA for a fix yet.

UPDATE 10:05am PST: We've identified a significant contributor to the slowness and fixed the situation. Mail is flowing through much more quickly now but but there is quite a bit in the queue that needs to be processed which could take an hour or more to complete. Will watch the the rate at which the queue empties for a bit and post an estimated completion time.

UPDATE 10:57am PST: Queue is emptying at a reasonable pace while also handling new inbound messages. I expect another hour or so for things to be completely caught up but will post here again as I get a better view on things.

UPDATE 11:38am PST: All queued mail has been delivered and things are functioning properly.

Thanks,
-tom

We have most folks moved over to the newer authentication scheme and will be doing the actual hardware switch tonight. It's exciting because we'll be giving ourselves tons of headroom as compared to the current system and with that extra processing power we'll be trying out more aggressive spam filtering techniques to reduce the noise in your mailboxes as well as increase your access speeds.

We'll be accepting mail throughout the hardware switch but in order to keep things synchronized properly we'll turn off the ability to check mail which should last for appx. 90 minutes. Hoping for sooner than that but we're giving ourselves a bit of a buffer. This will also affect the ability to add/remove/modify email accounts.

If we run into any notable problems we'll post here but with our prep work I'm fairly confident things will go smoothly.

Thanks!
-tom

UPDATE: We ended up with about 35 minutes of downtime for checking mail. All systems appear to be functioning properly and queued mail has been delivered. Thank you as always for your patience and certainly let us know if you have any problems: admin@vpop.net

We'll be switching our mail system over to a new server in the next day or so and in the prep work we noticed a few accounts that still authenticate using an older username scheme. While I'll be notifying the individual customers I thought I'd post here too, just in case spam filters or other issues cause someone to miss the notification.

The change is quite simple. If your username in your mail program is not your full email address, you'll want to change it to be so. For example, if my username was listed as "thomas", I'd want to change it to "thomas@vpop.net".

We have links to tutorials for some of the more popular mail programs at the bottom of this page: http://www.vpop.net/?c=help8.html&l=help_left.html&m=help_menu.html

Thanks!
-tom

There have been a couple of security patches issued for our systems which need to be applied ASAP due to some nasty threats that exist. They are kernel level issues and require a reboot to take effect. Each reboot should take ~5 minutes.

Thanks. And apologies for the short notice.

UPDATE: All machines rebooted and things look good across the board. Took about 5 minutes of downtime for each server.

-tom

We're seeing another problem with the webserver and we are having to reboot it. This is vastly different from "restarting" the webserver software. It can be 30 minutes before the entire system comes back. I'll post here shortly with an upate.

[Update 3:09] Server is back up. Still working but wanted to get this out.

-tom

We're in the middle of upgrading our webserver which is resulting in a bit of downtime. We're working on keeping it under an additional 15 minutes and if things are not complete, we'll revert. More updates ASAP.

[Update 3:34pm] We seem to be up and running. Watching for any "out of the way" errors or that sort of thing.

-tom

We're seeing a larger than normal amount of spam needing to be processed which is slowing mail delivery. We're investigating further to get things sped back up to normal delivery speeds.

[UPDATE 6:20am PDT] We've refreshed our rules which identify spam much earlier in the delivery process and mail seems to be flowing much closer to normal levels now. We still have a bit of mail in the queue which will filter down over the next hour or so.

-tom

Our primary webserver, which handles the bulk of our customers, is offline as we reboot unexpectedly. When performing this sort of operation it can take up to 40 minutes to recover.

We're fielding calls as rapidly as possible but I wanted to let folks know that we are tackling the issue. Email and other services are unaffected.

My apologies for the outage and I'll post here again when things look to be stable.

Thanks,
-tom
[Update: 4:29pm] Things appear to be back to normal. Our apologies again for the outage. :(

We've been seeing problems with our spam filtering off and on for the last couple of weeks. Today the mail queue has grown large enough to where it may take several hours to catch up. We've decided to turn off the filtering as we work on the problem so that mail can get delivered quickly versus being hung up in our filtering process.

You will see a marked increase in the volume of spam that makes it to your inbox for a couple of hours today as we continue to work on things.

Once the queue empties we'll turn things back on and continue to work on the filtering problem.

-tom

We are upgrading one of our primary webservers tonight which affects a few hundred customers. If your site is on ring.vpop.net your site will be affected off and on for a few hours tonight. With the holiday weekend and this being a Sunday night it seemed the best time to pull things off. I'll update here as things progress and email for us will be working throughout so if you have questions or issues, drop us a line. I'll try and field all calls during the process as well. If I miss you, please let me know what time is appropriate for a call back and I'll do my best to get to you then.

[UPDATE 1:20am]: The upgrade has gone well and most services are up and running. We have some tweaking to do but web sites are being served.

[Update 8:10am]: We've needed to recompile the webserver software so while the system is up it is not currently serving content.

[Update 9:35am]: We're still working on the webserver software update but are running into issues so we still are not serving web site content. Will update again shortly.

[Update 11am]: Things seem to be back up and running. There was a very subtle change in the way our webserver works with PHP between versions and it took a lot longer to find than it did to fix.

We're still rebuilding some modules but sites are up and running at this point.

-tom

After the reboot yesterday we apparently caused a problem for some customers which borke their ability to check email using POP3 clients such as Outlook or Thunderbird. The problem would have been encountered if your POP3 Incoming mail server was set to "mail.yourname.com" as opposed to "mail.vpop.net".

The issue has been resolved and I apologize to everyone affected. I'm catching up on returning the phone calls and emails now.

-tom

[Update 17:15 PST] Things look to be in good shape. Thanks for your patience!
-tom

[Update 17:00 PST] Rebooting now.

[Update 16:25 PST] We'll be rebooting in ~15 minutes.

[Original] We need to reboot one of our mail servers due to a security issue that has been announced for its operating system. Not exactly sure of the time but will update here when we know specifically. Guessing 1.5 hours from now but need to wait for various "build" processes to complete.

-tom

There appears to be a network outage at our Downtown LA, Ca. server facility. We are working on the problem with our upstream provider and will post an update here as soon as we know more.

The implication of this outage is that virtually all services are unavailable, including access to email, web sites, etc.

Update: The problem has been solved. Our building managers inadvertently cut power to the "switch" which connects us to the outside world. After some investigation they were able to find the problem and fix the situation. Unfortunately we were offline for appx. 1 hour and 20 minutes.

My apologies to all affected.:(

-tom

We are aware that there is a problem with our main webserver and we are working to get it up and running as soon as possible. We will post updates as soon as we know more

[UPDATE 4:30] System is back up and running.

We updated MySQL and only saw a couple of seconds of downtime as the server restarted. We should be in good shape but definitely let me know if you see any funkiness with your database driven web site.

-tom

Hello All,

We're upgrading MySQL today which should result in appx. 60 seconds (or less) of downtime for services which rely on that DB. This means, if you have a database driven web site, you'll see a few errors during that time frame.

Will post again when the process has been completed and tested.

-tom

A main webserver (ring.vpop.net) is having a problem right now and we are investigating. Will post here with updates.

UPDATE: 12:32: Things are back to normal. There were a few processes that were consuming too many resources and it caused the webserver (Apache) to lag to the point where it was not serving pages. Looks like we were down for appx. 8 minutes.

Apologies for the outage.:(
-tom

One of our primary webserver's is taking an extrmely long time to respond and is, for the most part, down. This means that your site may take a long time to display or might not display at all. We are investigating the problem now.

UPDATE 11:10am: We have the system back up and running. Still investigating the cause but the problem seems to be gone for now.

-tom

We've seen more than a two fold increase in spam which has been backing up legitimate mail. Our spam filtering increases the delivery times significantly but as so much mail is queued right now, it seems the best thig would be to turn it off for a bit while we work on combatting the delivery of spam at the soure(s).

If you find a huge amount of spam as compared to normal, that will be due to our having temporarily turning off spam filtering.

We're still filtering for viruses and general known spam servers but our system is not inspecting the contents of the messages for spam.

When the queue gets to a more managable size, we'll turn things back on.

-tom

We're working on a permanent fix for the hardware problems we saw on Monday. With that, we had a bit of trouble earlier which caused 20 minutes of downtime and we'll likely see one more of those today as we get our final pieces in place.

UPDATE: 12:44PDT: We're having to reboot now which should give us 10 minutes or so of downtime.

UPDATE 1pm PDT: Things are not going as smoothly as expected. We're having problems at the moment and are working to get them resolved. More updates coming soon...

UPDATE 1:09pm PDT: We're back up and running so we can handle things but we still have work to do. Again, more updates when available.

UPDATE: 1:41pm PDT: Another quick reboot required. Should be back in 10 minutes...

UPDATE 1:45pm PDT: Back up again. Still not out of the woods but trying to minimize downtime.

UPDATE 2:01pm PDT: We're doing the hard work now. With our previous testing and current reading, this should go smoothly but I must stress the term "should" because of how things have gone so far today. I'll update again here as things progress.

UPDATE 3:09pm PDT: Things have been going smoothly (as we had hoped!) so far but we will need to do a quick reboot after that -- appx 20 minutes from now and lasting less than 10 mintes. We'll have a final step afterwards which I'll write about once we pass this next point. Again, things are up and running now and have been for a while but our mail is a bit backed up due to the previous outages.

UPDATE 3:22pn PDT: We're gonna do a quick reboot with a new configuration. Should be less than 10 minutes outage.

UPATE 3:39pm PDT: The reboot is not going so well with the new partition. We have some work to do here. Will update again shortly...

UPDATE 5:29pm PDT: Things having been running smoothly for a while but we've had to queue mail for a bit as we get our final pieces in place. While you can send/receive mail, most new mail is not being immediately processed. We have (what I hope is) a final reboot coming up shortly. More details to follow...

UPDATE 6:01pm PDT: We have one more reboot to handle which should take 15 minutes if all goes well. Will report as things progress.

UPDATE 6:10pm PDT: Oversight on our part makes us need to copy a file over and then reboot again. We'r looking at another 15 minutes.

6:56pm PDT: We should be good to go! I'm watching closely but I believe we're out of the woods... finally! Thanks to everyone for their patience.

-tom

We've been having a problem with the primary mail server for the last hour. We're working on it now but do not yet have an estimate for when it will be back up. I'll post here again as soon as I get an update.

UPDATE 8:40am: We're having a significant hardware failure and are working around the problem now.

UPDATE 10:20am: We've worked around the problem and mail seems to be flowing. While you'll be able to login for POP3 and IMAP, the mail that was sent to the system during the outage has been spooled on a backup server and will take several hours to filter down to your mailboxes. More info before too long...

-tom

We need to update the kernel for the primary system which will require a reboot. We're replacing the OpenSSL library which is linked against many systems on the main server. In order to do this we need to rebuild the library and reboot the system. It is technically possible to do this without a reboot for completeness we'll be taking the extra couple of minutes to ensure that we've locked down any holes that might exist.

Total down time should be appx. 15 minutes. My apologies for the outage.

-tom

UPDATE: As expected, the outage was just under 15 minutes. Thanks for your patience!

It seems that the network connected to VPOP's system is completely down. This means that no mail or web traffic is making it in or out. We are working with our upstream provider now to get the problem resolved and will post an update here as soon as we have one.
-------------------

UPDATE [9:24am] Things are back up. Although the outage was less than 15 minutes it was a complete outage. I'm still trying to find out the details from our upstream but for now, we're up.

-tom

Over the last week we've seen intermittent but severe slowdowns with web site delivery. The problem is with the router of our upstream provider (the company that sells us our internet connection). We have been working steadily to get the issue corrected but response has been slow and no permanent solution has been found.

We are working on multiple solutions now at as great a pace as is possible. As we get more information about our specific course of action I will post the details here.

-tom

A primary webserver is offline (ring.vpop.net) and is being worked on. I hope to have it back up in the next 15 minutes or so.

[UPDATE 3:10am] We're back up though it took a bit longer than expected.:(

Thanks,
-tom

The same webserver is having issues again. We've been working on the problem but have yet to get it resolved. I'll post here again as information is available. My apologies for the delay in getting this noticed posted late. :(
-tom

[UPDATE:] The system is back up for now but it appears that we are having an intermittent hardware problem. We should be replacing the hardware late this morning. I post to this area when I have a firm time.

[UPDATE:] The system is still having problems and needs to be taken down again. More updates shortly...

[UPDATE:] We're up and running for the moment but will have to take the system down again in the late morning to replace the hardware.

[UPDATE 11:48am:] We will be doing a quick reboot at noon to put us in a better position to replace our hardware and gather additional information. We should see appx. 12 minutes of downtime during this procedure.

[UPDATE 2pm:] The previous reboot ended up taking appx. 4 minutes. We're in much better shape but still have some testing to do which will require some downtime for reboots. We plan on initiating these late tonight so as to minimize the general impact on things. We should all wake up to a more permanent solution to the problems we've seen today.

Once again, my apologies for the outage and extra work these outages must cause you as a result.
-tom

A primary webserver stopped responding at appx. 11:35 this evening and we were forced to reboot it. It is on its way back up though the file system checks will take 20 minutes or so to complete before it is fully operational. More details shortly.

[UPDATE 12:15] The server is up and running normally now. We're investigating the issue further as I type this message.

One of our main webservers has received more traffic than it can handle and is not responding well right now. We are working on the problem and will update here as soon as we have more information.

[UPDATE:] Things seem to be back and running normally.

-tom

With last night's reboot we found some inconsistencies with Perl modules and are upgrading both the system installation of Perl as well as the supporting libraries. This means there will be some intermittent CGI failures if you are using Perl and your needed libraries are still in the process of rebuilding.

[Update: 6:29pm PDT] The rebuild process has completed and things are looking to be in decent shape. We still have a bit of clean-up but by-and-large things are better off.

-tom

We've had to reboot one of the primary webservers which will cause approximately 30 minutes of downtime. More news to be posted shortly.

Our mail server move will commence in just a few minutes. The rate and amount of spam hitting our system has gone beyond what the old system could handle and this move to a new server should alleviate any pain.

You may notice up to 30 minutes of inability to authenticate via POP3/IMAP. We will be spooling mail during the outage.

UPDATE: The maill server seems to be up and functioning now. There may be some lag time due to DNS issues where webmail is accessible but POP3 and IMAP service should be up and running. We saw just under 20 minutes of downtime.

-tom