(mt) Media Temple would like to apologize to our (gs) Grid-Service customers for the series of issues relating to the (gs) system in the past few months. As an appreciation for your patience, we have applied 2 months free credit to your account. This credit has been issued automatically and is reflected in the billing area of your AccountCenter. We appreciate your continued business.

Notably, during our scheduled system upgrade on November 30th, the (gs) Grid-Service was offline longer than expected due to a failed upgrade to the storage firmware in Cluster.2. Although our company has a demonstrated 10-year track record of successful system maintenance actions, this past weekend’s event was an unfortunate exception. The majority of all scheduled items were completed and upgraded according to plan. However, one of BlueArc’s Titan disk systems, which provides a portion of the storage to our (gs) Grid, did not upgrade successfully nor did it roll back correctly when errors were discovered. Consequently this portion of the system maintenance missed its allotted time window by 7 hours. All other facets of the system maintenance were completed ahead of schedule and we encourage you to review the original scheduled maintenance announcement for additional reference: http://weblog.mediatemple.net/weblog/2007/11/21/electrical-systems-maintenance-notice-nov-30th/

The situation with the storage upgrade is particularly frustrating because the vendor supplied update was intended to fix issues - not create new ones. Even after the prolonged upgrade, the system is still unfortunately exhibiting some problems. We are waiting on a full analysis from the vendor regarding the reasons for the failed upgrade and continued instability. When these findings are available they will be communicated immediately.

It is well known that a vast majority of the performance and stability issues that have affected (gs) Grid-Service since its launch relate to storage issues. Consequently (mt) Media Temple engineers, along with senior management, have been working on a redesign to the storage architecture along with several other radically improved features in the platform. Most notably a new storage solution is been developed internally, with substantially reduced commercial vendor dependence and an architecture that will bring a high level of reliability back into our systems. This will result in a longer term solution that will be named the (cs) Cluster-Server, currently scheduled to go into beta in January. The beta testing program will give many customers the opportunity to experience some of the changes and improvements which we have made. Check http://www.mediatemple.net/labs/cs/ for more information when you have a chance.

(mt) Media Temple will continue to work vigorously on the (gs) Grid-Service platform until it is stable again. Our 75+ staffed company is fully committed to making the product as reliable as possible well before any new platforms are released. This evening engineers will be implementing some newly found tuning parameters to the system which are believed to correct some of the performance issues witnessed this morning. While we anxiously await the full root cause analysis from BlueArc concerning the failed upgrade and continued stability problems, we encourage customers to continue watching incident #306 for update to date system status.

Thank you again for your patience.

Best Regards,

Demian Sellfors
CEO
(mt) Media Temple, Inc

This entry was posted on Tuesday, December 4th, 2007 at 9:05 am and is filed under Electrical Systems Maintenance Notice - Nov. 30th. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.