First off I’d like to apologize for any inconvenience the recent email issues caused you.

Here is a brief description of what happened.

Some time on Wednesday afternoon (November 5th) the mail delivery component of our email system encountered an issue and went into a non-responsive (defunct) state.

Mail continued to be received and queued for delivery on the server, but wasn’t being delivered into individual mail boxes (or forwarded to other accounts if applicable).

At the same time our IMAP server (one of the methods of retrieving email) started dropping connections. As far as we know the POP3 server (another method of retrieving email) continued to function. Webmail uses the IMAP server, so was also affected.

Unfortunately this issue didn’t come to our attention until the Thursday morning (November 6th). A number of you did attempt to bring this issue to our attention earlier. Unfortunately most of these attempts to notify us ended up as victims of the very problem they were attempting to report.

Once we did become aware of the issue, it was fixed in less than an hour and it then took about another 45 minutes to process the backlog of email (roughly 12000+) that had queued up on the server.

No email was lost, only delivery was delayed.

So what changes are we going to make in light of this incident?

First off, we are going to make our emergency contact phone number (1-866-813-3417) more visible (I myself had trouble finding it on our web site).

Secondly we are going to try add to our monitoring systems some way to detect this type of issue. As far as our current monitoring systems were concerned the service was up and running. They weren’t able to detect that mail delivery, while it was technically running, had gone into a defunct state. We’ll be looking at changing this.

Thank you to all of you who took the time (to attempt) to notify us. Roughly 95% of all issues we encounter are dealt with before we receive any notification from clients (and 90% of those are dealt with automatically by the monitoring system before I even notice). But roughly 5% of all issues we encounter are brought to our attention by clients. Your feedback is crucial, so please, continue to report any issues you encounter as soon as you encounter them.

Thank you!

Christopher S. Hyne
Owner, BluePhyre Technologies