(mt) weblog

#1167 (gs) Grid-Service Investigating Possible Exploit

gclement — Thu, 11 Mar 2010 23:56:09 +0000

At this time, some users have experienced errors using the 1-Click Applications pages in the AccountCenter. This is an unexpected result of our ongoing investigation into Incident #1167 and the overall auditing of security within our internal infrastructure.

Symptoms:

Users are receiving an error stating: “Can’t select database”.

Others have experienced an error creating new 1-Click installations or upgrading existing 1-Click Installations.

We are aware of this issue and have corrected it at this time. If you continue to experience this error, please open a support request from your AccountCenter.

#1187 - Scheduled Maintenance for (gs) Cluster.02 Storage Segment 08

Robot-Incident — Thu, 11 Mar 2010 18:14:44 +0000

DATE/TIME WINDOW:

Sunday, March 14th, 2010 @ 10:00 PM - 2:00 AM PDT Monday, March 15th, 2010

AFFECTED RESOURCES:

(gs) Grid-Service Cluster.02 Storage Segment 08

To see which Cluster your (gs) Grid-Service resides on, please see the “Server Details” section of the (mt) AccountCenter. More details can be found in our Knowledge Base.

PER-CUSTOMER IMPACT:

All services will be unavailable for approximately 60 minutes.

MAINTENANCE ACTION:

(mt) Engineers will be upgrading the operating system on this server, improving overall system performance.

Although we require the entire maintenance window for preparation and execution, we only anticipate an impact of 60 minutes per customer. Should you have any questions regarding this scheduled maintenance, please open a Support Request from your AccountCenter.

#1178 Cluster.02 Emergency Maintenance

gclement — Thu, 11 Mar 2010 05:00:58 +0000

This post is a summary of Incident #1178, relating to Cluster.02 Emergency Maintenance.

Details:

Date/Time: The issue started at approximately 8:40AM PST on Monday, March 8 2010 and was resolved by Wednesday, March 10 2:00PM PST.
Symptoms: This incident included noticeably delayed page loads, slow email service, and general service slowness.
Impact: This incident impacted all services within Cluster.02.
Root Cause: The root cause of this incident was legitimate increased load across Cluster.02.
Takeaways: Additional hardware has been ordered to offset the increased load across Cluster.02. We have also upgraded all the processors used for Cluster.02 servers. These have been upgraded from 4-core units to 8-core units.

This now concludes this System Incident. If you feel that you are still experiencing the symptoms outlined in this post, please open a support request from the (mt) AccountCenter.

#1178 All Cluster.02 Services Remain Stable

amanda — Wed, 10 Mar 2010 21:00:23 +0000

Both email and web load remains stable and healthy. Customers should not be experiencing latency in relation to this System Incident.

(mt) Engineers continue to analyze the data gathered during their investigation in order to determine the root cause of the symptoms experienced while this incident was active. When a conclusion has been reached regarding this incident’s root cause, an update will be published.

#1178 Email Latency Addressed

amanda — Wed, 10 Mar 2010 18:50:21 +0000

The efforts made by (mt)Engineers have fortunately reduced email queues and service levels back within normal operating limits. Delays with email delivery should no longer be present. We are taking further steps to address any blocklist issues while we continue our work to investigate the root cause of this incident.

Engineers do not anticipate email latency to return. We will provide information as developments are made. Please check back for an update by 2:00PM for before.

Password change operation completed successfully.

nmiller — Wed, 10 Mar 2010 18:19:31 +0000

As of about 9pm last evening, (mt) Engineers successfully completed the password change operation that we previously outlined. If you would like additional details on what took place and why, please visit this special Q&A article from our KnowledgeBase:

http://kb.mediatemple.net/questions/1807

In the course of changing the appropriate passwords, we also took the liberty of opening a brand new Support Request within each affected account. This was done to make troubleshooting any related issues easier for all involved parties. If you have additional questions or need assistance with any issues that have arisen due to the password update, please respond to that specific Support Request or call us at our toll-free number, 877-578-4000.

Moving forward, (mt) Engineers will continue the process of investigating the root cause of this Incident. As always, we will update this page as more information becomes available, or if we determine that any other actions are necessary.

#1178 Blocklisting Determined to be Unrelated

amanda — Wed, 10 Mar 2010 18:07:40 +0000

(mt) Engineers have now determined that email blocklisting seen by (gs) Grid-Service Cluster.02 customers is not related to this incident. If you are experiencing a blocklisting issue, please submit a new Support Request and it will be addressed individually.

The next update regarding this incident will be made as information is available or no later than 2PM today.

#1178 Possible Impact to Email Services

amanda — Wed, 10 Mar 2010 17:40:51 +0000

We have had reports of email related problems on Cluster.02. Some customers are experiencing email delivery latency, and others have reported that emails have bounced due to blocklisting. We believe these problems may be related to the core problem that this incident stems from. (mt) Engineers are taking action to relieve these problems, as well as investigate the correlation between these new issues and those related to web services seen earlier.

We will provide a status update regarding these matters within the hour.

#1178 Monitoring and Research Efforts Continue

amanda — Wed, 10 Mar 2010 16:10:38 +0000

At this time (mt) Engineers have upgraded the vast majority of storage segments on (gs) Grid-Service Cluster.02 with additional resources. While website load-times have stayed healthy through the night, they seem to show varying degrees of load to web services currently. Our engineers will continue to closely monitor the service levels throughout the day to observe the effects of these upgraded servers during “peak” times.

Barring any unforeseen complications, we will only provide an update to this incident once our investigation of the root cause is complete.

#1183 (gs) Database Issues Resolved

tstein — Wed, 10 Mar 2010 02:59:38 +0000

As of 6:51PM PST, our system engineers have restored database availability for all customers located on Cluster 02. We were able to confirm that the database host machine was online throughout the incident however the machine was not available to our public networks. The network issue was isolated after this was diagnosed and all services are currently online. Please note that no MySQL data was lost or corrupted as a result of this incident; the issue was limited strictly to our network.

Thank you for your patience while we worked to resolve this matter.