VPOP Updates

Excess Spam Getting Through Today

Tom — Wed, 23 Apr 2008 12:47:43 -0700

We've been seeing problems with our spam filtering off and on for the last couple of weeks. Today the mail queue has grown large enough to where it may take several hours to catch up. We've decided to turn off the filtering as we work on the problem so that mail can get delivered quickly versus being hung up in our filtering process.

You will see a marked increase in the volume of spam that makes it to your inbox for a couple of hours today as we continue to work on things.

Once the queue empties we'll turn things back on and continue to work on the filtering problem.

-tom

Upgrading One Of Our Primary Webservers

Tom — Sun, 17 Feb 2008 22:20:50 -0800

We are upgrading one of our primary webservers tonight which affects a few hundred customers. If your site is on ring.vpop.net your site will be affected off and on for a few hours tonight. With the holiday weekend and this being a Sunday night it seemed the best time to pull things off. I'll update here as things progress and email for us will be working throughout so if you have questions or issues, drop us a line. I'll try and field all calls during the process as well. If I miss you, please let me know what time is appropriate for a call back and I'll do my best to get to you then.

[UPDATE 1:20am]: The upgrade has gone well and most services are up and running. We have some tweaking to do but web sites are being served.

[Update 8:10am]: We've needed to recompile the webserver software so while the system is up it is not currently serving content.

[Update 9:35am]: We're still working on the webserver software update but are running into issues so we still are not serving web site content. Will update again shortly.

[Update 11am]: Things seem to be back up and running. There was a very subtle change in the way our webserver works with PHP between versions and it took a lot longer to find than it did to fix.

We're still rebuilding some modules but sites are up and running at this point.

-tom

Mail Problems Due To Yesterday's Reboot

Tom — Wed, 16 Jan 2008 08:14:18 -0800

After the reboot yesterday we apparently caused a problem for some customers which borke their ability to check email using POP3 clients such as Outlook or Thunderbird. The problem would have been encountered if your POP3 Incoming mail server was set to "mail.yourname.com" as opposed to "mail.vpop.net".

The issue has been resolved and I apologize to everyone affected. I'm catching up on returning the phone calls and emails now.

-tom

Rebooting Mail Server Briefly

Tom — Tue, 15 Jan 2008 11:42:55 -0800

[Update 17:15 PST] Things look to be in good shape. Thanks for your patience!
-tom

[Update 17:00 PST] Rebooting now.

[Update 16:25 PST] We'll be rebooting in ~15 minutes.

[Original] We need to reboot one of our mail servers due to a security issue that has been announced for its operating system. Not exactly sure of the time but will update here when we know specifically. Guessing 1.5 hours from now but need to wait for various "build" processes to complete.

-tom

Network Outage -- Updated

Tom — Sat, 16 Jun 2007 11:41:07 -0700

There appears to be a network outage at our Downtown LA, Ca. server facility. We are working on the problem with our upstream provider and will post an update here as soon as we know more.

The implication of this outage is that virtually all services are unavailable, including access to email, web sites, etc.

Update: The problem has been solved. Our building managers inadvertently cut power to the "switch" which connects us to the outside world. After some investigation they were able to find the problem and fix the situation. Unfortunately we were offline for appx. 1 hour and 20 minutes.

My apologies to all affected.:(

-tom

Webserver problem

Tom — Thu, 26 Apr 2007 15:49:47 -0700

We are aware that there is a problem with our main webserver and we are working to get it up and running as soon as possible. We will post updates as soon as we know more

[UPDATE 4:30] System is back up and running.

MySQL Successfully Updated

Tom — Wed, 21 Mar 2007 13:02:18 -0700

We updated MySQL and only saw a couple of seconds of downtime as the server restarted. We should be in good shape but definitely let me know if you see any funkiness with your database driven web site.

-tom

Upgrading MySQL Today

Tom — Wed, 21 Mar 2007 10:57:28 -0700

Hello All,

We're upgrading MySQL today which should result in appx. 60 seconds (or less) of downtime for services which rely on that DB. This means, if you have a database driven web site, you'll see a few errors during that time frame.

Will post again when the process has been completed and tested.

-tom

Webserver Having Problems

Tom — Mon, 19 Mar 2007 12:24:50 -0700

A main webserver (ring.vpop.net) is having a problem right now and we are investigating. Will post here with updates.

UPDATE: 12:32: Things are back to normal. There were a few processes that were consuming too many resources and it caused the webserver (Apache) to lag to the point where it was not serving pages. Looks like we were down for appx. 8 minutes.

Apologies for the outage.:(
-tom

Webserver Down

Tom — Fri, 05 Jan 2007 11:02:43 -0800

One of our primary webserver's is taking an extrmely long time to respond and is, for the most part, down. This means that your site may take a long time to display or might not display at all. We are investigating the problem now.

UPDATE 11:10am: We have the system back up and running. Still investigating the cause but the problem seems to be gone for now.

-tom

Spam Problems - Spam Filtering Turned Off

Tom — Wed, 11 Oct 2006 17:36:26 -0700

We've seen more than a two fold increase in spam which has been backing up legitimate mail. Our spam filtering increases the delivery times significantly but as so much mail is queued right now, it seems the best thig would be to turn it off for a bit while we work on combatting the delivery of spam at the soure(s).

If you find a huge amount of spam as compared to normal, that will be due to our having temporarily turning off spam filtering.

We're still filtering for viruses and general known spam servers but our system is not inspecting the contents of the messages for spam.

When the queue gets to a more managable size, we'll turn things back on.

-tom

Mail Server Issues

Tom — Wed, 04 Oct 2006 12:44:11 -0700

We're working on a permanent fix for the hardware problems we saw on Monday. With that, we had a bit of trouble earlier which caused 20 minutes of downtime and we'll likely see one more of those today as we get our final pieces in place.

UPDATE: 12:44PDT: We're having to reboot now which should give us 10 minutes or so of downtime.

UPDATE 1pm PDT: Things are not going as smoothly as expected. We're having problems at the moment and are working to get them resolved. More updates coming soon...

UPDATE 1:09pm PDT: We're back up and running so we can handle things but we still have work to do. Again, more updates when available.

UPDATE: 1:41pm PDT: Another quick reboot required. Should be back in 10 minutes...

UPDATE 1:45pm PDT: Back up again. Still not out of the woods but trying to minimize downtime.

UPDATE 2:01pm PDT: We're doing the hard work now. With our previous testing and current reading, this should go smoothly but I must stress the term "should" because of how things have gone so far today. I'll update again here as things progress.

UPDATE 3:09pm PDT: Things have been going smoothly (as we had hoped!) so far but we will need to do a quick reboot after that -- appx 20 minutes from now and lasting less than 10 mintes. We'll have a final step afterwards which I'll write about once we pass this next point. Again, things are up and running now and have been for a while but our mail is a bit backed up due to the previous outages.

UPDATE 3:22pn PDT: We're gonna do a quick reboot with a new configuration. Should be less than 10 minutes outage.

UPATE 3:39pm PDT: The reboot is not going so well with the new partition. We have some work to do here. Will update again shortly...

UPDATE 5:29pm PDT: Things having been running smoothly for a while but we've had to queue mail for a bit as we get our final pieces in place. While you can send/receive mail, most new mail is not being immediately processed. We have (what I hope is) a final reboot coming up shortly. More details to follow...

UPDATE 6:01pm PDT: We have one more reboot to handle which should take 15 minutes if all goes well. Will report as things progress.

UPDATE 6:10pm PDT: Oversight on our part makes us need to copy a file over and then reboot again. We'r looking at another 15 minutes.

6:56pm PDT: We should be good to go! I'm watching closely but I believe we're out of the woods... finally! Thanks to everyone for their patience.

-tom

Mail Server Having Problems

Tom — Mon, 02 Oct 2006 07:14:38 -0700

We've been having a problem with the primary mail server for the last hour. We're working on it now but do not yet have an estimate for when it will be back up. I'll post here again as soon as I get an update.

UPDATE 8:40am: We're having a significant hardware failure and are working around the problem now.

UPDATE 10:20am: We've worked around the problem and mail seems to be flowing. While you'll be able to login for POP3 and IMAP, the mail that was sent to the system during the outage has been spooled on a backup server and will take several hours to filter down to your mailboxes. More info before too long...

-tom

Emergency Update/reboot

Tom — Thu, 28 Sep 2006 17:28:51 -0700

We need to update the kernel for the primary system which will require a reboot. We're replacing the OpenSSL library which is linked against many systems on the main server. In order to do this we need to rebuild the library and reboot the system. It is technically possible to do this without a reboot for completeness we'll be taking the extra couple of minutes to ensure that we've locked down any holes that might exist.

Total down time should be appx. 15 minutes. My apologies for the outage.

-tom

UPDATE: As expected, the outage was just under 15 minutes. Thanks for your patience!

Network Outage -- Updated

Tom — Thu, 28 Sep 2006 09:11:48 -0700

It seems that the network connected to VPOP's system is completely down. This means that no mail or web traffic is making it in or out. We are working with our upstream provider now to get the problem resolved and will post an update here as soon as we have one.
-------------------

UPDATE [9:24am] Things are back up. Although the outage was less than 15 minutes it was a complete outage. I'm still trying to find out the details from our upstream but for now, we're up.

-tom

Connectivity Problems -- Slow Web Site Delivery

Tom — Tue, 12 Sep 2006 08:55:12 -0700

Over the last week we've seen intermittent but severe slowdowns with web site delivery. The problem is with the router of our upstream provider (the company that sells us our internet connection). We have been working steadily to get the issue corrected but response has been slow and no permanent solution has been found.

We are working on multiple solutions now at as great a pace as is possible. As we get more information about our specific course of action I will post the details here.

-tom

Webserver Offline

Tom — Thu, 31 Aug 2006 02:47:45 -0700

A primary webserver is offline (ring.vpop.net) and is being worked on. I hope to have it back up in the next 15 minutes or so.

[UPDATE 3:10am] We're back up though it took a bit longer than expected.:(

Thanks,
-tom

Webserver problem continues

Tom — Tue, 08 Aug 2006 07:02:58 -0700

The same webserver is having issues again. We've been working on the problem but have yet to get it resolved. I'll post here again as information is available. My apologies for the delay in getting this noticed posted late. :(
-tom

[UPDATE:] ~~The system is back up for now but it appears that we are having an intermittent hardware problem. We should be replacing the hardware late this morning. I post to this area when I have a firm time.~~

[UPDATE:] The system is still having problems and needs to be taken down again. More updates shortly...

[UPDATE:] We're up and running for the moment but will have to take the system down again in the late morning to replace the hardware.

[UPDATE 11:48am:] We will be doing a quick reboot at noon to put us in a better position to replace our hardware and gather additional information. We should see appx. 12 minutes of downtime during this procedure.

[UPDATE 2pm:] The previous reboot ended up taking appx. 4 minutes. We're in much better shape but still have some testing to do which will require some downtime for reboots. We plan on initiating these late tonight so as to minimize the general impact on things. We should all wake up to a more permanent solution to the problems we've seen today.

Once again, my apologies for the outage and extra work these outages must cause you as a result.
-tom

Webserver problem

Tom — Mon, 07 Aug 2006 23:48:49 -0700

A primary webserver stopped responding at appx. 11:35 this evening and we were forced to reboot it. It is on its way back up though the file system checks will take 20 minutes or so to complete before it is fully operational. More details shortly.

[UPDATE 12:15] The server is up and running normally now. We're investigating the issue further as I type this message.

Webserver outage

Tom — Tue, 18 Jul 2006 14:39:50 -0700

One of our main webservers has received more traffic than it can handle and is not responding well right now. We are working on the problem and will update here as soon as we have more information.

[UPDATE:] Things seem to be back and running normally.

-tom

Rebuilding Perl

Tom — Tue, 13 Jun 2006 17:10:27 -0700

With last night's reboot we found some inconsistencies with Perl modules and are upgrading both the system installation of Perl as well as the supporting libraries. This means there will be some intermittent CGI failures if you are using Perl and your needed libraries are still in the process of rebuilding.

[Update: 6:29pm PDT] The rebuild process has completed and things are looking to be in decent shape. We still have a bit of clean-up but by-and-large things are better off.

-tom

Webserver Reboot

Tom — Mon, 12 Jun 2006 21:31:03 -0700

We've had to reboot one of the primary webservers which will cause approximately 30 minutes of downtime. More news to be posted shortly.

Mail Server Move

Tom — Wed, 30 Nov 2005 16:21:16 -0800

Our mail server move will commence in just a few minutes. The rate and amount of spam hitting our system has gone beyond what the old system could handle and this move to a new server should alleviate any pain.

You may notice up to 30 minutes of inability to authenticate via POP3/IMAP. We will be spooling mail during the outage.

UPDATE: The maill server seems to be up and functioning now. There may be some lag time due to DNS issues where webmail is accessible but POP3 and IMAP service should be up and running. We saw just under 20 minutes of downtime.

-tom

Spam Filtering Issue

Tom — Mon, 14 Nov 2005 07:37:31 -0800

Our spam filtering system seems to be having a problem which is causing mail to be delayed far longer than is acceptable. In order to get mail out of the queue and delivered properly we have disabled the spam filtering system. We will reenable it shortly after the queue empties but for this morning you will likely see far more spam than usual.

-tom

Server Move Beginning

Tom — Sat, 12 Nov 2005 00:14:47 -0800

For those of you who were notified of your server being moved, we will begin the process in appx. 5 minutes or, 12:15am (PST) Nov. 12, 2005. Web and Email access will not be functional during this time though mail will be queued for delivery until the servers are back up and running.

Please check this page often as I'will be posting updates as they become available.

UPDATE: Web serving is up as of appx. 2:15am.

UPDATE 2: Mail should be working as well though it will take several hours for all of the queued mail to be delivered.

Thanks,
-tom

Phone System Issues

Tom — Sun, 25 Sep 2005 20:00:18 -0700

We seem to be having problems with our phones which is causing error when folks try to connect. We are working with the local Telephone company to get things worked out but their progress has so far been very slow going. We'll keep this page up to date with news but in the meantime, if you have a problem or question and cannot connect via phone, please do drop us an email here: admin@vpop.net.

Thanks,
-tom

Mail Delays

Tom — Mon, 05 Sep 2005 10:41:11 -0700

Email delivery has been delayed by several hours because our spam filtering process had problems. Email that should have been delivered quickly was deferred because it could not be processed properly by the spam filtering system.

The spam filtering system has been repaired and mail is starting to get delivered but it will be several hours before the deferred queue is emptied.

Web Server Outage

Tom — Fri, 29 Jul 2005 08:31:31 -0700

UPDATE: the system seems to be up and running normally now. The downtime seems to have lasted for just over 2 hours. We are still investigating the core cause of the outage and once it is identified we will take measures to avoid the situation in the future. My apologies for the outage and the delay in getting this site updated in a more timely manner.

We've been having a problem with one of our main webservers which also affects email for many customers. We are still working on the problem and hope to have it back up and running shortly. The affected server is "ring.vpop.net".

Mail Delays and Web Site Stats

Tom — Fri, 10 Jun 2005 17:25:13 -0700

It's been quite a day today and I know extremely frustrating for many of you. I have some details about what has been going on today, what we can expect this evening and why it has taken so long to update this page.

Last things first: We purposely host our System Status site on a different location so if something goes wrong with our core network we can still provide you with updates. Unfortunately, in addition to our own problems, the service which hosts our System Status page was performing maintenance so we were unable to update this page.

Early this morning our main spam filtering system went down. Due to the amount of spam we are filtering these days mail can get backed up quite quickly. While no mail was lost, much of it has been delayed and for several hours. As a way to increase delivery speed we have temporarily turned off spam filtering, until the mail queue empties. We do have backup spam filtering capabilities but unfortunately they can no longer handle the sheer volume of junk that comes in, in a timely manner, so we'll need to remedy this situation.

Web site statistics are also affected at this time. Again, while no data was lost, we'll need to reprocess the logs from last night and then remount that partition so your stats are again viewable in the usual location.

We are still working on some of the problems and I will post an update when things are back and fully functional.

Once again, I appreciate your patience and apologize for the frustration you have had throughout the day.

Tom McDonald

System Maintenance Today

Tom — Tue, 05 Apr 2005 13:35:00 -0700

There has been a vulnerability discovered which needs to be addressed immediately on all of our servers and this requires a reboot. Each machine can take up to 20 minutes to complete the reboot process. Some machines will be handled this afternoon while the primary systems will be handled at 10pm PDT tonight.

If you notice problems today, please give appx. 20 minutes and test again, just in case you are trying to access things while a system is being rebooted.

As always, thanks for your patience!

Tom McDonald, President,
VPOP Technologies, Inc.