Email Collection unavailable (Ref 33459) - Further Information

Email

Posted on: Friday 7 July 2006, 19:01

Over the last few days customers have been impacted by timeouts and subsequent difficulties while collecting their email. We recognise the frustration this has caused, and completely understand that email is a vital service for many of our customers, who expect the system to work correctly at all times. We feel the same.

At this time, some mail collection issues are still apparent, although we believe to a significantly lesser degree than earlier in the week. Once we have resolved the underlying cause here we will go into more detail about the events that lead to the problem, and the longer term investment plans for our email platform and general network infrastructure. That post will be made later next week. The current status of the email platform and action we are taking immediately are documented below.

The issue revolves around our new Sun Storagetek solution, to which we have been migrating customers over the last week. As we reached the end of this process, it became clear that there was a slowdown in mail collection times and timeouts began to occur in customers' email applications. We eased this by partially reversing the move of customers to the new storage. This was not an issue that was highlighted by the load testing we conducted before deploying the storage solution.

Sun are investigating this matter as a top priority problem and the issue has been highly escalated within their organisation. Sun engineers are working alongside our own team and both parties will remain on-site until we are confident the problem has been resolved. This work is expected to continue throughout the weekend, and at 8PM today Sun are planning to deploy a set of patches, which include a possible fix, to the back-up half of the storage platform. This will allow for testing and if successful the upgraded back-up storage platform will be made primary, and the same action will be taken on the current live side (Which will then become the back-up).

We believe that these patches will provide a solution, however as a precaution we have scheduled engineers to remain on-site for the weekend to continue monitoring and working on this. We are also very conscious that due to the normally lower load on the platform at a weekend, there is a possibility that the problem could appear to be resolved, but then recur at some point during the week when demand for email is much higher.

At the same time, our engineers are also currently working to resolve a number of cases where customers who have usernames beginning with the letter A have found that some or all of their mail cannot currently be downloaded. This is being corrected and it is expected the matter will be fully resolved during the weekend.

We would also like to take this opportunity to ask customers to be careful about how they configure their email clients to collect mail. We have identified some customers who make literally hundreds of new mail checks every minute. This adds significant load to the email platform, and while not something for wish we wish to specify a rule here, we believe that setting the interval between email collections to 5 minutes or more should prove adequate for most customers.

We will continue to update service status during the weekend, and will post more background information about this issue later next week, once we are fully confident that the underlying cause of the current issues has been resolved. The PlusNet Usergroup have created a dedicated forum for discussion about this, and any feedback in relation to the email platform is welcome at http://usergroup.plus.net/email_discussion

With Regards,

Ian Wild
Customer Communications Manager

Return to Index