Media Temple Customers

A blog by (mt) for (mt) customers

Archive for the ‘Uncategorized’ Category

System Status Ambiguity

with 10 comments

I keep getting comments on this site and Twitter messages about sites being completely down or in various states of disarray. Some customers complain that Media Temple has not called them on the phone or explained anything.  Sites being down this long is insane and the status page does not really clarify things:

recoverystatus

Media Temple should definitely communicate better with these users. All that is gained from the Status Incident Page is that sites are being moved at a very slow pace. Media Temple should answer these questions to the customers who are still experiencing problems:

What does this mean? 

Why were some sites back online yesterday and others still offline right now?

Why is it taking so long?

What is this thing about the Fiji Agent? Can I perform migration on my own to speed things up?

Was the multiple phone calls I got due to my loud complaints on this blog and Twitter? I don’t know. For me 8 hours of downtime, 20 hrs of Account Center lockout and intermittent email/ftp errors which translated into 1yr of free hosting seemed fair… It’s like beta testing their new architecture… as a web developer free hosting to play with is something I would always agree to even if it’s unstable (which I hope the new Cluster will be)

My personal sites and projects I plan keep on this Media Temple (gs) account. Why would I trust them at this stage? Primarily because an account I have on the newer Gen 2  architecture on Cluster 4 hasn’t experienced any serious downtime and I’m led to believe my migrated Cluster 2 sites will be fine from now on out.

My clients who squawked the loudest I will be moving to a different ISP. I have a responsibility to them and if they would feel safer that way, I will oblige.

For people whose sites are still down and losing Alexa rankings, traffic, ad revenue, and clients- that really sucks. I don’t know what can be done for them. This is my attempt to help, as a small web developer  I understand your frustration. Please leave suggestions here, perhaps your voices will be more unified than on Twitter. Media Temple does seem read all the comments and posts on this blog very quickly.

Written by bnvk

May 6, 2009 at 10:05 pm

Proper Compensation & Migration

with 4 comments

So, after the downtime yesterday, the blocked Account Center today, more downtime this afternoon- Media Temple has started posting their hourly progress of migrating all sites off of the unstable Gen 1 architecture on Cluster 2.

As of  10:51 PM on May 5th they have moved 1,535 sites and have 845 more sites to go which if my calculations are correct that means Cluster 2 is 64% migrated to the stable Gen 2 architecture. I got a personal call from one of their admins who informed me all my sites would be moved at 4:00 AM.

Earlier in the evening fellow web design / (mt) customer Aaron Halford brought to my attention the following Tweet from @MediaTemple 

1yrfree

Although (mt) hasn’t made any mention of this on their website I’m sure they’ll hold good with this offer. I’ve got to say this feels like generous and just compensation. Whatever the reasons for the hardware failure, whatever the reasons for not being migrated sooner- this compensation is testament to customer care and loyalty I’ve yet to see with any large company in any industry. Way to go Media Temple. 

Perhaps there are others who have had dire consequences from the downtime to which even this is not enough, but in my case it is. 

So if you’re a customer on Cluster 2 who has experienced the downtime on May4th/5th send Media Temple a Direct Message on Twitter or fill out a support ticket from your (mt) Account Center. 

Hopefully the new Gen 2 architecture will be tight as a drum and we will all be smooth sailing from here. 

Written by bnvk

May 6, 2009 at 7:06 am

Posted in Uncategorized

Cluster 2 Back, Customer Care, Migration Recap

with 6 comments

So Cluster 2 was back online after 8+ hours of downtime in my case, and still down in some other  peoples cases. My Account Center is still locked out after 19+ hours and some sites of mine have errors that I can’t seem to troubleshoot due to the Account Center Lockout. 

picture-2

I got a phone call from Andrew, a Customer Service V.P. at Media Temple whom I have spoken to before, here are some points that were addressed.

“What percentage of Cluster 2 customers have been migrated to Gen 2 architecture?”

Andrew: “44% of Cluster 2 Customers have been moved.”  (which is the same as it was back on March 6th)

Brennan: “Wow!”

Andrew: “We just finished our Fiji Agent last friday. This will speed up the process.”

“Why was no one moved during the last two months? Or at least why was whiny little bitch customers like me not moved, even after I asked to be moved after the 36 hours of downtime?”

Andrew: “We had faith that the Cluster 2 hardware would be ok.” 

Brennan: “Seriously? You had faith that after 36 hours of downtime [the same hardware setup] would be ok?” 

Andrew: “Well, let me rephrase that. We wanted to make sure we were making the best decisions to migrate people at the best times so there would be minimal amounts of downtime” …and… “we have been putting all of our resources and developers into  the Fiji Agent.”

Brennan: “Wow! Too bad this had to happen.”

Media Temple claimed to be on a “road to better transparency” something about this doesn’t feel transparent to me. I said “Transparency doesn’t mean talking more, it means being honnest”

Andrew: “We haven’t been dishonest about anything.”

While technically that might be true. Media Temple did claim on March 6th that:

Next 30 days, Cluster-to-Cluster migration tools. We have committed significant developer and administrator manpower to the development of Cluster-to-Cluster migration tools.”

I understand. Software development does sometimes take longer than anticipated…

But for some reason I am inclined to say bullshit. If it looks like a bull and smells like shit it’s probably bullshit. I seem to remember around the time of SXSW (March 17th), during the absurdly long period (1.5 weeks) of POP email server IP blacklisting, there was a lot of Tweets about how amazing the Media Temple SXSW party was. In fact they threw The Official Closing Party for the SXSW Interactive Festival. This party was replete with international music groups, an open bar, a photo booth, and a mechanical bull. 

sxsw_bull

Maybe I’m off base here, but I imagine throwing a party of that scale takes time to organize and a substantial budget to pay for. Oddly, I don’t see the correlation between mechanical bulls and fixing unstable web servers but maybe that’s just me.

In hindsight, considering this new downtime debacle, and the facts, I have to ask the question: How much of Media Temple’s resources were devoted to getting frustrated customers to a more stable hosting platform like their Gen 2 architecture? 

Oh, Andrew also offered to move my sites within the hour to Cluster 4 so I could use the Account Center… this however could result in more patches of downtime… I wonder if they understand that downtime during the middle of the day is a bad thing for websites? I said “How about you do that later tonight.”

**The quotes may not be exact as I didn’t have an audio recorder handy. I tried my best. 

Written by bnvk

May 5, 2009 at 6:23 pm

8+ Hours Of Downtime

with 13 comments

After 4+ hours of unexpected server downtime, a couple irate clients, 1 semi angry Tweet I received a call from  customer service at Media Temple. She wanted to know if I had any questions…

My only question was:

“What percentage of Cluster 2 customers have been moved from Gen 1 architecture to Gen 2 architecture”

This customer support lady didn’t know the answer to that question. She put me on hold to try to get the answer. After a few minutes waiting she said “they have no way of knowing exactly.”

That was the wrong answer. Because now I am left feeling that Media Temple either has no clue about what they’re doing or they are not being honest. How can I jump to this conclusion? Well on March 6th 2009, following the 36+ hours of downtime, Media Temple made this blog post: Anatomy of Storage on the GRID

It explained all the old architecture (Gen1 ) and new architecture (Gen 2). 

2gens

This blog post also claimed:

“At time of writing, 37% of Cluster 1 customers, and 44% of Cluster 2 customers have already been migrated.”

So I am forced to really wonder how 2 months later they don’t know at all… I mean 80% or 70% percent would have been undesirable answers but “We have no way of knowing” is kind of absurd. They didn’t project to be done with until June but what I am left to think now?  

In either case I  getting very very distrustful with them as a hosting company. I was one of the customers affected by the 36 hours of downtime staring Feb 28, 2009. I had setup this blog. I got a lot of attention for rallying the twitter/blog-o-sphere. Back then I got a phone call from a V.P. at Media Temple followed by personal phone number for V.P. at Blue Arc (mt hardware supplier). They both sorta blamed the other for the downtime. Fair enough. It still sucked. I didn’t want to go through the headache of moving to a new ISP and I was appeased by some PR people- they gave us all free hosting for a month. But now another HUGE outage is just unacceptable. 

 

 

Written by bnvk

May 5, 2009 at 4:09 am

“Cluster 2 Unavailable” The Aftermath

with 5 comments

mediatemp_down

After 38 hours of downtime on Media Temple’s (gs) Cluster 2 from  12:30 PM Sat Feb 28th – 6:00 AM Mon March 2nd all systems seem to be back online to where they were before the crash. 

At 6:00 PM Mon Mar 2nd Media Temple posted a blog entry: “Incident Resolved. Service credits being issued” which explains some of the technical details as to why we experienced this epic amount of downtime. They also stated that they are extending credit to all (3,000)  Cluster 2 customers totaling approx. $100,000.00 

To explain the customer service side of how (mt) handled the downtime they sent Steve Reynolds a personal email which he posted on his blog “Official Response from Media Temple to Open Letter” Media Temple takes responsibility and aims to do it better next time.

After speaking with Media Temple V.P., Alex Capehart, I felt better about staying on board and giving (mt) a second chance. After speaking with BlueArc V.P., Louis Gray, I felt a tad less safe staying with Media Temple. Why? They both said things under the “You can’t go public with this…” clause at which I will not go public… but there was a fundamental difference between them in regards to their companies responsibility. Fair enough. They  both seem like sincere; nice dudes who both have a good handle on what’s going on within their industry. Rarely in life are two point of views identical when explaining why a misfortune has occurred- especially when it’s two companies stuck in a symbiotic relationship. 

The fact is accidents occur- look at what happened to Ma.gnolia at least in (mt)’s case no data was lost. I think the ugly truth is that customers who can’t yet afford $50+ /mo hosting will probably have to put up with some occasional downtime. For some customers this will be inexcusable and they will be change web hosts. It is unfortunate that the customers who will be most affected by these outages are the customers on the verge of growing into more secure hosting plans- the independent design firms hosting a handful of clients but can’t yet spring for VPS or dedicated.

The company that figures out how to deliver bulletproof, scaleable, and affordable hosting and have great customer service will quickly have a rock solid client base and reputation. What matters most until then for me is the customer service, transparency, etc… that make these outages tolerable. In pursuit of that cause I’ll give Media Temple one more chance.

Written by bnvk

March 3, 2009 at 6:14 am

Posted in Uncategorized

Collection of Media Temple Customer Experiences

leave a comment »

Here are few links to well written informative posts about how others are experiencing this horrid amount of downtime. 

Duane Storey – The Day That Media Temple Stood Still

Steve Reynolds Blog – An Open Letter to Media Temple and Media Temple Outage affects Customers

Social Marketing Contract – Media Temple Provides Religious Sunday Experience

Karelia Software – Weekend Website Outage

And there is always the live Twitter Search

Written by bnvk

March 2, 2009 at 9:34 am

When You Yell Loud Enough…

with 13 comments

…you get heard. I just got off the phone with Alex Capehart who is the V.P. of Business Development from Media Temple. Wow. Write some blog posts. Rally the troops… you get a response. What Alex said made sense to some degree. 

Apparently Bluearc is who we should mostly be upset with for the now 30+ hours of downtime. As Alex put it “[Media Temple] is up to their elbows trying to get this thing sorted out.” We could look at this as a cheap ploy by (mt) to win me over so I can help sway the perception of the blogosphere in their favor. However, the key fact that makes me buy this story is that earlier today (before I setup this blog) I was followed on Twitter by @bluearc and then Direct Messaged by them. I responded. Then I was followed by @louisgray who I quickly learned he was a VP at Bluearc. Here is his DM Tweets.  

picture-4

So why wasn’t Media Temple more transparent with it’s customers from the get go? Firstly, they probably underestimated the scale of the hardware failure. Secondly, politics pure and simple. I told Alex that most of (mt)’s customer base are above average computer users and the status updates they give just aren’t cutting it. “Working with vendor” is ambiguous and leaves customers feeling like they are being lied to. The way Alex responded to this leads me to believe there are many politics at play- politics with Bluearc, politics with other hardware manufactures, and politics within Media Temple itself. Hopefully the CEO and others at (mt) will hear this loud and clear as “Keep us in the loop better and more in depth!” 

Does all of this make our frustrations go away? No. Does it make up for the hours of service we’ve lost? Certainly not! Where do I stand on this? I am not ready to jump ship from Media Temple just yet. I need to see what tomorrow brings- once they’re past the problem. Alex explained to me that Cluster 3 & 4 actually have a different architecture than Cluster 2 and that the hardware is from a different vendor- a more reliable vendor he assured me. I want to build a good relationship with (mt) that can support me as my hosting needs grow, but I also need assurance that my accounts will be stable- and I need some sort of compensation for this considerable loss of service. I’m sure many of you feel the same way. We will see…

Written by bnvk

March 2, 2009 at 7:30 am