State of the Grid 2010
December 30th, 2010 at 6:01 pm
Hi, my name is Matt Jones and I’m on the Product Team here at (mt). I want to close out 2010 by making sure everyone who wants to stay informed about our (gs) Grid-Service has the latest information on the status of the platform. If you’re unfamiliar with the (gs), it is a cluster-based hosting platform that we launched in late 2006. The (gs) is home to over 400,000 domains serving up more than half a billion web hits per day. At launch, the (gs) gave birth to a new way of thinking about on-demand web hosting and was a big departure from traditional shared hosting technology. We set expectations high and in some cases over-delivered but the system itself has not been without its share of problems.
In the past we have communicated a great deal of transparency about the Grid, conveying its success and its shortcomings, but in 2010 we fell short on the frequency of our communications. Users of the Grid deserve to have all the information – whether good, bad, or ugly and I’m committed to delivering the details. Read on to see progress this year with the (gs) and upcoming improvements for 2011.
2010 in Review
Despite the wide adoption of the (gs) platform, as we came into 2010 the stability of web, email, and MySQL were improved, but not up to the (mt) standard. We hear the sentiments on Twitter and blogs, and we surveyed all of our customers this year, and while the vast majority of users remained happy, there were far too many that experienced frustrating downtime, security and service interruption issues. The top three issues our users asked us to focus on were Uptime, Stability and Security. With these as our prime targets, we’ve continued to make improvements to the (gs) platform in 2010. The following is a sample of some of the biggest changes this year:
Security Improvements
2010 brought a drastic increase in malicious attacks on hosting providers – more than we’ve ever seen. While many large web hosts felt similar pain, we found our (gs) platform to be a particularly attractive target due to its size and unique architecture. Many of our customers suffered and, frankly this is just not acceptable to us at (mt). To mitigate and repair the issues, we formed an internal, multi-disciplinary security team who led efforts to drastically improve security on the platform.
These improvements, many of which we’ve talked about before included: Ongoing security scans and greater automated detection of the most common exploits across the entire (gs), enhanced blocking of internet addresses that try to hack the Grid, and several infrastructure and configuration level fixes. We’re happy to mention that though attacks continue, as they do at every hosting company, we’ve seen a 77% drop in affected domains since October. Exploit counts continue to drop and we estimate about 0.7% of (gs) domains are experiencing problems. The best way to keep your websites safe is to follow good practices like keeping your software up-to-date and changing your passwords, but (mt) will be watching your back.
Uptime & Stability
We have always monitored server health, but at this scale traditional ‘best practices’ become unmanageable. We needed to step up and anticipate problems before your sites were affected, so we completely rebuilt our internal monitoring systems to give us the greatest possible visibility into every aspect of the Grid architecture. We’ve built in specific alerts for leading indicators to performance issues – giving us the visibility we need to minimize or prevent downtime from occurring. Today, we’re continuously identifying and fixing issues that you will never notice.
Updated Email Spam Filters
With over half a million mailboxes on the (gs) system, spam email was becoming an issue that not only annoyed our users, but created noticeable performance issues on the (gs). We worked with Cloudmark, the industry’s fastest, most accurate and most scalable spam, phishing and virus protection system, to update all user mailboxes. Filtering is now better than ever, but if you are still experiencing problems please contact support and let us know immediately.
That’s not everything that changed in 2010 but it definitely represents the highs and lows. Now we want to share with you some highlights that are on our roadmap for the Grid for the early half of the year.
2011: Coming Soon
Faster Web and Email
(mt) Engineers have been working hard to optimize responsiveness and reliability for the most important services: Web and Email. By changing our load balancing scheme and email architecture we expect mail to be between 1.5 and 3 times faster. We don’t have numbers on the web response yet, but we will be discussing the geeky details in the user forums. Incremental roll outs will begin in early January and we will share more specific details once they get started.
In addition, we are working to decrease page load times by adding Solid State Drives (SSD) to all storage segments. The SSDs will let us serve more content from cache on disks that respond many times faster than normal hard drives. This has already decreased load times for the newest clusters and we are evaluating the best way to roll this out to all clusters.
Beyond making these performance improvements we’re also actively investigating ways to separate our email and hosting architectures. To make a long story short, we’re convinced this is the way to go to improve the reliability and performance of the Grid and offer a world-class email solution at the same time. Again, we’ll provide many more details as research and discovery phases conclude.
Easier Software Updates
One of the best ways to stay secure on the internet is to keep your software up-to-date.
The Grid’s custom Linux architecture has made updating software without breaking user sites or creating customer downtime very difficult. Many users have asked for newer versions so they can use the latest and greatest applications, and because they are as concerned as we are about the security benefits of up-to-date software. The next cluster will launch with completely updated OS, PHP, and MySQL and we’re continually working to update software across all clusters.
What About (cs)?
In 2009 we announced that we were working on the successor to the (gs)… A product called the (cs) Cluster-Server. This was exciting news to users on the (gs), but its now almost 2011, and the (cs) still hasn’t been rolled out. Understandably, our users are wondering what the hold up is.
Put simply, we chose to dedicate our resources to improving the hosting experience for our existing (gs) customers, numbering in the tens of thousands before we were comfortable working on something new. That said, many of the (gs) improvements you’ve felt in 2010 as well as the upcoming enhancements for 2011 were originally slated to be a part of (cs). We felt the benefits to providing improvements for existing users far outweighed the benefits of creating a brand new system, and overall Grid performance and Stability improved in 2010 because of it.
We haven’t yet delivered on our promises with regards to the (cs). We want to make it quite clear that there isn’t anything more important to us than launching (cs) and continuing to make the Grid a more reliable, secure, and feature-rich service. In the past couple months we have realigned and strengthened our teams and shifted focus to make this happen. We’re in much better shape than ever before to make these promises a reality. We know that actions speak louder than words, so stay tuned. We’re excited for what’s in store for 2011.
You’ll Be Hearing From Us…
There are many other projects in progress that we’re excited to talk about, but it’s a bit too soon to share details. Expect product updates on the weblog on launch days and more detailed conversations in the user forums. Support agents, engineers, and managers are in the forums every day. Don’t be shy!
As we head into 2011 I want to re-emphasize that I’m committed now more than ever to improving the Grid product and to launching (cs). In fact, it’s one of my resolutions for the New Year! (Well, that, and finish my first Ironman Triathlon.)
I would love your feedback on anything mentioned above or questions about anything else that’s on your mind. You can leave a comment here on the blog, find me on Twitter @ehmatt, or join me in the (gs) User Forums.
Thank you for your continued loyalty in (mt) Media Temple.
Happy New Year,
Matt Jones
» 
December 30th, 2010 at 6:12 pm
More dedicated MySQL would be nice, instead of having it spread across multiple sites, which causes it to slow sometimes.
I don’t want to have to pay $40 a month just to have my own MySQL box included. If that was happening then why not just upgrade to the (dv). This is really something that should be fixed. The boxes are a joke with the current pricing honestly.
December 30th, 2010 at 6:22 pm
Thanks (mt) for providing such an excellent hosting environment. Every time I ran into a problem, I felt that you guys went above and beyond. Your play a big part to us and our clients.
December 30th, 2010 at 6:32 pm
Thanks for the feedback Dane – MySQL improvements are going to be essential in 2011.
December 30th, 2010 at 6:47 pm
Thanks for letting us know, Austin. Great experiences are what we aim for!
December 30th, 2010 at 6:49 pm
Good Job MT!
December 30th, 2010 at 7:13 pm
First, thank you for your entire team’s commitment to being a customer service oriented outfit. I’ve been consistently surprised at how easy it is to get someone on the phone to help me when I need it. Thanks for that.
The largest problem I have with mt is that the default configurations for mysql on gridserver aren’t quite capable for a multi-site configuration. And the dedicated virtual servers don’t have quite enough hard drive capacity to handle all of the available plesk licenses. In that regard, things are starting to feel like the old discount hosting situation again. Sure I have tons of available bandwidth and CPU cycles, but I’ll never have a chance of consuming all of it because my sites would break way before I get there.
So, I’m hoping that specifications will improve on all of your offerings in 2011. Need more power.
December 30th, 2010 at 7:23 pm
We need private IPs, as we used to have in the Shared Server platform in the past. I’ve had my outgoing newsletter IP-blocked by some ISP due to bad neighbors in (gs), and dedicated IP was cited by support as one of the future (cs) improvements. Presently, dedicated IPs are only an option with SSL certificates, which make them economically unsound for the average blogger.
Happy 2011 and good luck with the Ironman.
December 30th, 2010 at 9:33 pm
Thanks Eduardo. Updated spam control will help keep our IPs off the block lists so dedicated addresses aren’t needed, let us know if it happens again.
December 31st, 2010 at 3:54 am
Thank you (mt) for an amazing job with the (gs). I really would appreciate a backup solution for the grid, like Snapshot Backups or so.
December 31st, 2010 at 6:04 am
Appreciate your candid State of the GS and the outline of the plans going forward. Honesty sells. Frequent communication sells more. Looking toward a more responsive (gs) system especially MySQL db response.
December 31st, 2010 at 9:25 am
Thanks for the kind words, Lorenz. We agree that with you on backups. Our first priority is to make sure the overall (gs) platform is stable before we add new features but we see backups as one of the top enhancements we need to add in 2011.
Sara
(mt) Product Team
December 31st, 2010 at 9:38 am
Howard, thanks for your continued loyalty. We missed the mark in 2010 and know that we have to step up our communications. Matt and I, along with the entire (mt) team, are committed to keeping you informed.
And, we agree with you, MySQL improvements are a must in 2011.
Thanks again.
Sara
(mt) Product Team
December 31st, 2010 at 10:31 am
2010 had a good chunk of security issues. Nothing different from any hosting company out there though. Was most impressed though with how Media Temple handled the security issues swiftly.
PS Loving the Twitter support.
December 31st, 2010 at 4:52 pm
I really like the (gs) Grid-Service. The workflow process it allows me to have is really efficient.
However the only negative is the speed of DB apps like WordPress. They seem to run much slower on (gs), compared to other shared servers (even after optimisation and CDN etc..).
Anything that improves the speed of the (gs), will be very welcome. I would happily pay more to see this kind of improvement.
January 2nd, 2011 at 6:13 am
Have to agree with AJ and others about the mySQL improvements needed since we now use WordPress for pretty much everything.
A simplified backup routine would be ideal too, since after getting hit several times this summer, I had to restore frequently and I’m not great at scripting backups. Fortunately, Media Temple provided excellent support even when they were struggling to handle the security breaches.
January 2nd, 2011 at 10:54 am
Happy New Year, (mt)
For 2011 I hope we will finally see a reduction in latency of WordPress sites hosted on (mt) GS.
As many have expressed before and continue to express; seeing free hosted WordPress sites load faster than anything hosted on (mt) GS is very frustrating.
Please stop pointing fingers at your user’s code (unoptimized …, theme, …), upgrade/buy a container, … ” WP has different servers, which are optimized for WP “. Fact is the same data and code for small & infrequently visited sites hosted on WP.com runs circles around GS. So, maybe it is time to try something different, like giving those that want to solely use WP a (mt) GS WP account on a ” WP optimized ” GS / CS server. Something has to change to give your small customers with WP a boost.
Looking forward to the 2011 updates & upgrades.
January 3rd, 2011 at 8:14 am
I love Media Temple’s articles, tweets, customer service and pricing. I’m not sure how anyone could complain. I have used the GS service in the past many times and continually point clients to this solution as it is cheap, works and comes with MT customer support.
I have moved away from GS altogether and into the VE/DV arena where I have been extremely happy. I would suggest anyone not happy with the GS just shell out a little extra $$ for the DV.
You can always split it with a friend(s) to reduce cost.
Thanks MT!
January 4th, 2011 at 10:49 am
I really need a backup tool for my MySQL databases. Backuping up files is easy using FTP, but backing up a large database is nearly impossible (PHPMyAdmin dump does NOT do the job…) at this moment.
January 5th, 2011 at 9:11 am
When can we use Mysql 5.0.xx version on Grid GS? I still used with mysql 4.
January 5th, 2011 at 12:54 pm
@Kp, we’re working to roll out MySQL 5.0.xx to clusters 1-3, other clusters have 5.1.26 available.
January 8th, 2011 at 6:26 pm
I second the backup suggestion. And here’s another part of that. Make it include mail. The email accounts on my server are my responsibility, and it’s not really right (in my mind) that I cannot access the files to do a backup of them.
January 10th, 2011 at 1:30 am
MySQL improvements sounds really nice… I have had too many “Error establishing a database connection” in 2010.
January 11th, 2011 at 12:42 pm
I feel that we were mislead by your representative when we signed up for your service. We were told the service would not experience issues due to “clustering.” This has not been the case. I was nervous about this structure to begin with and was assured that there was no need for concern.
Now, for the second time in less than a week (that I know of), we are completely down, and have been for several hours. This is just simply unacceptable! I don’t like to call in because the first line ppl that answer the phone never know anything and can never explain anything. I get much better communication from your techs on twitter. As an ecommerce customer, this is affecting my business! When I get responses though, like “Multitudes of users, using a limited amount of resources is going to have issues every so often.” it is upsetting. If you don’t have enough resources to handle demand, GET MORE RESOURCES! Your rep on Twitter said he/she could not “pinpoint” what the problem is today. Really? Well, then I hope somebody can!
I really have been disappointed in MT this year. The speed is too slow and these outages are more than a little frustrating.
I appreciate the candor of your statement here and hope you will continue to communicate about improvements.
Not a happy camper.
January 19th, 2011 at 4:33 pm
Still waiting on (cs). I know, I know, it’s coming. I’m holding off on going with a (gs) for a third go-round given the issues I had before. Consequently, I will be avoiding (mt) until the (cs) is launched. Soon, please?
-fin
Thales
January 20th, 2011 at 4:08 pm
@ Mary, I apologize for the (gs) not working for you the way you expected. We’re identifying performance bottlenecks (you’ve seen MySQL mentioned here quite a bit) and are addressing them as quickly as we can with extra resources and updated logic as the situations require.
Thanks for your feedback. We’ll definitely continue to communicate and work to improve your experience.
January 31st, 2011 at 1:07 pm
Gimme some speed (mt) speed, I want speed, I have a need for speed, especially regarding database use;-)
I think the plan to separate the email server from the web server would be a nice move, especially when there are people who send bulk email which must also drain resources? Personally I think newsletters should be sent from a dedicated system like Campaign Monitor, but of course, who am I to judge how people should use their mail accounts and do their mass emailing…
February 2nd, 2011 at 1:42 pm
It would be really good if the speed of downloads could be better here in europe. A Year ago, speed was like 200 – 300 kb/s, now it’s 50 – 80 KB/s. I hope we can use 400KB in near future, that would be really nice
February 5th, 2011 at 9:06 pm
Liberen el CS por favor! , please release the CS!
February 12th, 2011 at 12:11 pm
Great job so far MT. I think you deserve a pat on the back to say the least. The only thing that i can think of that would be a good addition is the ability to run mongo db on (gs) or (cs) when it’s finally launched. It might be a bit much to ask for, but we’ve got to hope and dream at the least..
February 23rd, 2011 at 1:16 am
I agree with Daniel Gruber for best performance for Europe clients… or a central (London or other Europe) location…
I also agree with regard to the backup (files and db) and optimization of MySQL.
March 9th, 2011 at 6:56 pm
Thanks guys keep up the great work. The uptime & security improvements are very welcome.
March 9th, 2011 at 9:03 pm
Matt,
I’m glad that MT made the decision to improve GS and hold off CS. I’ve been with MT since your very early days (techs are always surprised to see my account number), and I have all my client sites on MT. I have to say that I’m less and less pleased with the problems already mentioned above:
1. Frequent downtime.
2. MySQL latency issues for all WordPress sites
3. Techs who constantly point to some coding issue as the reason for slowness on the WordPress admin.
4. The “you can fix it if you upgrade to Cloud, Container, VE, DV, xyz, new MT add on product”
I would really like to continue to use MT for hosting, but it won’t be able to tolerate the slowness much longer. I’d like you to share in an email/post what exactly you are doing to address the latency issues with MySQL and WordPress for GS and when they will be fixed on each cluster. Use my sites as benchmarks.
Again, very loyal customer, but the goodwill capital that you’ve built up over the years has been spent in 2010.
March 9th, 2011 at 11:20 pm
I am very happy with my mediatemple experiences. Fast loading times, stable, and haven’t had any security issues (which so far always happened with other hosts). Great value for the money!
March 10th, 2011 at 12:43 am
I also agree with regard to the backup (files, email and db) and optimization of MySQL.
I gotta say for the price (gs) is awesome in power/flexibility/ease of use
I’ve had a few problems with uptime etc of the last couple of years, but i appreciate the honest reporting of the issues and the speed at which things have been resolved.
Thank you and looking forward to your next developments.
March 10th, 2011 at 4:35 am
I appreciate the honesty in this blog post, the biggest issue I had when hosting with them was the site went down more than any other host I had before (used the gs service). It is also why I no longer use this company, but the company does look like they are trying at least to get things better.
March 10th, 2011 at 5:58 am
Frank,
Thanks for the feedback! With respect to MySQL latency, we just had a new database pool hardware spec written, and just last week started some major configuration changes with the help of the good folks at Percona. You should see some major improvements in MySQL soon, and I’ll be working on turning ‘soon’ into a real date.
Matt
March 10th, 2011 at 5:59 am
@ Tim Thanks! We’ll be working to make sure your experience gets better from here.
March 10th, 2011 at 7:12 am
@Matt for the response. My business is staked to MediaTemple so I want to see you succeed. I love all the little extras that MT has provided in the last year (most recently JetPack), but getting the core business services on GS working smoothly is paramount to keeping customers. I’d gladly give up the extras for a flawless and unmatched delivery of the MT core services–web hosting–fast, reliable and stable.
March 10th, 2011 at 7:25 am
Completely agree on MySQL latency being priority #1.
The reality is that a clean WordPress install is almost unusable on GS because of the MySQL slowness. We have only been able to use our GS for dev sites, not production. Pretty soon it will be unusable for dev sites too if there is not substantial change.
GS could be a GREAT product if it wasn’t for the slow MySQL performance, let’s make it happen!
March 10th, 2011 at 7:31 am
A backup system would be much appreciated.
March 10th, 2011 at 7:48 am
I agree with everyone. Mysql performance and optimization for gs is crucial for multisite configuration. I have both (gs) and (dv) services with you and the (dv) small hd space makes it less attractive for the price. Moreover the plesk is an awful product in general and would love to see alternative panel options like webmin. I am paying $109/each for 2 LAMP dedicated boxes with another company which has 4gb ram, 250gb hd space, duo core 2.5 xeon to handle my 11.2mil records queries for survey web app I developed. I know that if MT had such pricing I would not have gone with someone else.
So the moral of the story is I use MT (dv) as my dev server for now at $100/month with 1gb of ram and 20gb hd space and it’s virtual server.
Hope go see better pricing and performance in 2011.
March 10th, 2011 at 9:34 am
Faster email is great, if it works with the email client of choice. While I understand that Thunderbird doesn’t have the market-share of other clients, it would be nice to not have to fight your servers when my staff prefers it. It’s particularly troublesome trying to get it to use the Sent folder on your system when using IMAP.
I bring this up here rather than as a support issue because your support team blew me off with the old “3rd party software” argument and resolving the issue on my side wasted more time than it should have.
March 11th, 2011 at 8:07 am
I think it is funny that the day after I received this in my inbox my GS went down
Nice to see some honesty, though. That I do appreciate.
March 30th, 2011 at 2:21 am
I think you should pour more resources into (cs) and get it out. So you have an perfect upgrade path to (dv). (gs) has reached to a state that i am not even sure if it is worth any more effort. (mt) ‘s (dv) are very good. But your (gs) is making 99% of your bad press.
With (cs) would hopefully allow your customer who are over using their resources limitation to move on to (dv).
I am just annoyed that three months pasted, and not even a beta process for (cs) or even update on it.
March 30th, 2011 at 10:32 am
Just Focus on (cs), (gs) is not worth the effort anymore.
March 30th, 2011 at 1:23 pm
@ Edward, thanks for the recommendation – an easy upgrade path is one of the ‘must haves’ for the new product.
June 6th, 2011 at 2:14 am
You are doing wonderful job and support thro GS.
Our suggestion is you can provide mails back up support and migration of accounts including website, mails and database to another account
July 30th, 2011 at 2:26 pm
I would use your product if it was cheaper – Perhaps provide a smaller package with less resources? (Just some freedback
)