Details:

• Date/Time: The issue started at approximately 8:00AM on Monday, February 1 2010 and was resolved by 11:30PM, Pacific Time on Thursday, February 4.

• Symptoms: Customers located on Storage Segment 01 of (gs) Grid-Service Cluster 05 may have noticed performance degradation and overall latency for web and email services.

• Impact: The impact of this System Incident affected customers located on (gs) Grid-Service Cluster 05, Storage Segment 01. All other Storage Segments remained unaffected.

• Root Cause: While we aren’t completely ruling out disk fragmentation, the most likely cause of this incident was an abnormally high lock count on the storage array. This did not alert our existing monitoring system of an issue, causing a delay in our initial investigations. The high lock count manifested itself into customer-facing symptoms, predominantly web and email latency. As we isolated the problem to Segment 01 only, the large majority of sites on Cluster 05 remained unaffected by these issues.

• Takeaways: (mt) Engineers will continue their tremendous collaboration with our Storage Vendor to gain insight into the ways we can predict and address issues of nature which will set the course for how we handle these matters in the future. In addition, we are also revising our metrics for the “sweet spot” of storage utilization to be even more conservative in our capacity and rebalancing plans. The revised metrics equate to less sites per (gs) Storage Segment, which will provide ample room for dealing with file locking spikes, fragmentation, and disk space utilization issues.  We do acknowledge that these issues could potentially come up from time to time, but our goal is they do not cause a customer-facing symptom; instead, we will do our best to deal with them “behind the scenes”. Adding an additional layer of protection that is inherent in our deployment practices will prove to help us avoid an issue of this nature again in the future. While our communications with our Storage Vendor continue, we still have some additional work to complete to specifically find the root cause, however, we’re confident that your services will be up and running from this point on.

• The odds of a recurrence are minimal.

This now concludes this System Incident. If you feel that you are still experiencing the symptoms outlined in this post, please open a support request from the (mt) AccountCenter.

—————–
Monday, February 8th 2010 @ 10:00 PM - 1:00 AM Pacific Standard Time

AFFECTED RESOURCES:

——————-
Several (dv) Dedicated-Virtual Server and (dpv) Nitro host machines in VA-IDC4. This includes:

nitro329.mediatemple.net
vz475.mediatemple.net
vz476.mediatemple.net
vz477.mediatemple.net
vz478.mediatemple.net
vz479.mediatemple.net
vz480.mediatemple.net

To see which host server your hosting service resides on, please see the “Server Details” section of the (mt) AccountCenter.

PER-CUSTOMER IMPACT:

——————–
Approximately 30 minutes.

- Your hosting service will experience a complete disruption to all resources, including Web, FTP, Email, and Databases.
- Any emails that are inbound during the maintenance period will be queued remotely and delivered to you once the work is complete.
- You will be able to access the (mt) AccountCenter and other (mt) resources not hosted on your (dv) Dedicated-Virtual Server.

MAINTENANCE ACTION:

——————-
Network Engineers have identified a failing network switch that provides an uplink to a portion of our (dv) and (dpv) customers. While all services are up and operational during this time, we have scheduled this work pro-actively to move all attached customer servers to a new switch before a critical failure occurs. We require the entire maintenance window for preparation and execution, however we anticipate any customer-facing downtime, if needed, to be less than 30 minutes.

For more information about our network infrastructure and technology, please see this page.

We would like to remind all of our customers that scheduled maintenance actions are a necessary and vital aspect of our operations that ensures the ongoing reliability of your hosting service. Should you have any questions regarding this scheduled maintenance please open a support request inside the (mt) AccountCenter.

Regards,

(mt) Media Temple, Inc
Hosting Operations

We will now consider the incident closed. This incident has been meticulously documented, and moving forward, will not result in a customer-facing issue again.

Please open a Support Request in your AccountCenter if you have any further questions or concerns regarding this incident.

We will update, and hopefully close, this incident shortly after we confirm the Storage Segment is permanently stable. Thank you for your patience.

To find out which Cluster and Storage Segment your (gs) Grid-Service is hosted on, please see this article: http://kb.mediatemple.net/questions/1569

We will update this page with more information often, until a resolution is reached.

Our very own Chris Lea, architect and driving force behind the Virb infrastructure and startup rock-star consultant passes on his priceless knowledge on how to bulletproof your MySQL backups. Follow Chris’s instructions on Think Vitamin and you will rest easier knowing that you have good, safe backups of your database.

Think Vitamin is the official blog of Carsonified, a long time (mt) partner . Think Vitamin is a member of the Smashing Network and overall awesome resource created by the prolific web professional, Ryan Carson

We will provide you with additional information in our next update, scheduled to be made on or before 2010-02-05 at 5:00PM PST.

• Web performance will be negatively impacted for a period of time after Storage Segment 1 is brought back online. Your sites will be available however you may notice slower page loads or general latency when viewing your site. Performance will gradually increase as new requests are loaded into cache. We anticipate this lasting no longer than 4 hours.

We will continue providing additional information as soon as it becomes available until a full resolution is met. Barring any complications or unforeseen circumstances, our next update will be on 2010-02-05 at 10:00AM PST.

We will update this System Incident when the maintenance is completed. Beyond this, our extensive internal testing based on our Storage Provider’s recommendations will proceed. This might require us to schedule another maintenance at some point for Storage Segment 01, and we will address this in a future updates to this incident if this is required.