Media Temple Customers

A blog by (mt) for (mt) customers

When You Yell Loud Enough…

with 13 comments

…you get heard. I just got off the phone with Alex Capehart who is the V.P. of Business Development from Media Temple. Wow. Write some blog posts. Rally the troops… you get a response. What Alex said made sense to some degree. 

Apparently Bluearc is who we should mostly be upset with for the now 30+ hours of downtime. As Alex put it “[Media Temple] is up to their elbows trying to get this thing sorted out.” We could look at this as a cheap ploy by (mt) to win me over so I can help sway the perception of the blogosphere in their favor. However, the key fact that makes me buy this story is that earlier today (before I setup this blog) I was followed on Twitter by @bluearc and then Direct Messaged by them. I responded. Then I was followed by @louisgray who I quickly learned he was a VP at Bluearc. Here is his DM Tweets.  

picture-4

So why wasn’t Media Temple more transparent with it’s customers from the get go? Firstly, they probably underestimated the scale of the hardware failure. Secondly, politics pure and simple. I told Alex that most of (mt)’s customer base are above average computer users and the status updates they give just aren’t cutting it. “Working with vendor” is ambiguous and leaves customers feeling like they are being lied to. The way Alex responded to this leads me to believe there are many politics at play- politics with Bluearc, politics with other hardware manufactures, and politics within Media Temple itself. Hopefully the CEO and others at (mt) will hear this loud and clear as “Keep us in the loop better and more in depth!” 

Does all of this make our frustrations go away? No. Does it make up for the hours of service we’ve lost? Certainly not! Where do I stand on this? I am not ready to jump ship from Media Temple just yet. I need to see what tomorrow brings- once they’re past the problem. Alex explained to me that Cluster 3 & 4 actually have a different architecture than Cluster 2 and that the hardware is from a different vendor- a more reliable vendor he assured me. I want to build a good relationship with (mt) that can support me as my hosting needs grow, but I also need assurance that my accounts will be stable- and I need some sort of compensation for this considerable loss of service. I’m sure many of you feel the same way. We will see…

Advertisements

Written by bnvk

March 2, 2009 at 7:30 am

13 Responses

Subscribe to comments with RSS.

  1. This is a pretty soft stance you’ve taken here all of a sudden. You sound kinda starstruck after speaking with a PR guy. Are you actually beginning to take MT’s side?

    “Apparently Bluearc is who we should mostly be upset with for the now 30+ hours of downtime.” Uhh… no! I’m a customer of MediaTemple, not Bluearc. (As are you.) This is a MediaTemple issue. MediaTemple, not Bluearc, is fully accountable here; much in the same way that I, not MT, am answerable to my clients.

    Jimbo

    March 2, 2009 at 7:42 am

    • I am in no way taking Media Temples side and saying they are forgiven. I was impressed that one of their V.P.s called me- which is a lot more than many other hosting companies have done for when downtime has occurred. Lunarpages sent out mass emails to their customers. 1&1 has tech support in India… that didn’t even understand the problem… Alex took responsibility and said my criticism was fair and that they’re working their ass of trying to fix the shit- so I am giving them the benefit of the doubt and a chance to redeem themselves.

      I worked at one of the top 5 web hosting companies in the late 90s and saw in office and behind the server politics… yes there are politics with everything especially cutting edge technology. Are any of your clients gonna drop you immediately because of this outage or they are gonna cut you some slack? I for one hope that my clients who have been affected by this outage will give me second chance I’m giving Media Temple. However, if Media Temple doesn’t meet my demands I’m bouncing to a new host ASAP! I would love to hear your voice on this blog too. I don’t want to be the only authority on this cluster fuck of downtime!

      brennannovak

      March 2, 2009 at 8:03 am

  2. I couldn’t agree with Jimbo more.

    It’s not like I can tell my clients ‘check the (mt) blog’. My decision to use (mt) = my fault their service is terrible.

    Thank god they all use Google Apps – 24 hours without e-mail would have been completely devastating.

    Charles

    March 2, 2009 at 8:09 am

  3. I think it’s rather weak of Media Temple to continuously blame their storage vendor. Ultimately Media Temple chose what hardware to use, and they should be taking responsibility for those decisions.

    Duane Storey

    March 2, 2009 at 8:24 am

    • Yes. Passing blame is the ugly part of the human condition. Often times it’s also a very necessary part. Apparently Bluearc makes some of the best shared computing setups for processing 3D and film rendering… and was proving some top notch bench marks 2+ years ago… Their technology had never been applied to ridiculously high traffic web environment where CPU requests happen randomly and are not controlled at a constant rate. This excuse makes sense to my tech brain… Is that a risk that Media Temple took? Sure. Has it in the last year been very unstable? Yes! All this Alex reassured me was why Cluster 3 / 4 are using different technology and different hardware that is made by a very well respected company known in the internet industry for such.

      brennannovak

      March 2, 2009 at 8:45 am

  4. If that’s the case, then Media Temple should be blamed for not upgrading the older clusters with better hardware. Blaming BlueArc just seems like they are trying to scapegoat them, something that’s going to cause even more backlash in the internet community I would imagine.

    Duane Storey

    March 2, 2009 at 8:49 am

  5. Brennan and all, as noted in the screen caps from the DM, the BlueArc team is very aware of Media Temple’s issues. I am glad you got the chance to speak to Alex on this – he is a great guy, and Media Temple is a good company with a unique product. That all rings hollow when there are outages, of course, but I can assure you that this is not only visible at the highest levels at both companies, but being worked by both at full capacity.

    Media Temple has a culture of transparency, in terms of them communicating to customers, and telling, to the best of their ability, as much as they know.

    Looking forward to good news from them tonight and tomorrow. For you and the rest, please do keep watch over the Media Temple twitter account – where you will likely get the most updates.

    Louis Gray (BlueArc)

    March 2, 2009 at 8:59 am

  6. Glad to hear that Alex reached out to you. I know for a fact that both BluArc and (mt) engineers have been on this issue all weekend and the coordination has been spectacular. I don’t think there are any hard feelings, probably more like feelings of camaraderie. 🙂

    We will be providing a detailed blog post very soon regarding the current state of the Grid. It will explain in great depth what has been going on and where things are going from here. For now, you can stay updated on the progress of getting things back online at http://weblog.mediatemple.net/weblog/category/system-incidents/2009-02-28-cluster-2-unavailable/

    Arman Zakaryan – (mt) Media Temple

    Arman Zakaryan (mt)

    March 2, 2009 at 9:46 am

  7. So everyone that is on the other Clusters that are not 3/4 are pretty much getting shafted with this ridiculous amount of downtime? Either upgrade the older clusters or move us over!

    Luis Mendoza

    March 2, 2009 at 10:16 am

  8. At least both MT and BA are starting to reach out to the community online.

    Luis Mendoza

    March 2, 2009 at 10:21 am

  9. “So everyone that is on the other Clusters that are not 3/4 are pretty much getting shafted with this ridiculous amount of downtime? Either upgrade the older clusters or move us over!”

    This very inquiry will be addressed in a future blog post. We’d very much like to flip a switch to fix all of this, but unfortunately it requires a lot of time and planning to do. Our customers will get a proper explanation shortly following the conclusion of this weekend’s incident.

    Arman.

    Arman Zakaryan (mt)

    March 2, 2009 at 10:29 am

  10. Hope (mt) does what I did about 10 years ago:
    ONLY BUY HDDs from WESTERN DIGITAL! 🙂

    I’m not gonna say anything, but my guess is those hard drives are either samsung or maxtor… To a lesser probability, maybe seagate (they’ve been getting better).

    Like I’ve ben saying on Twitter: I won’t be jumping ship! I’m more committed than ever to (mt). Great companies learn from their mistakes and I believe this could be a great turnkey point for (mt) and I’m willing to stick around and give them my support. I’m not dumb and I’m watching. Being a techno-geek/programmer and sysadmin myself, I know that some things cannot be predicted. Situations like these teach to create better redundancy and backup systems and I’m *sure* the great team at (mt) are already laying out the systems to avoid critical failures like we’ve seen this weekend.

    DISCLAIMER: No, I’m not sponsored, supported or remotely connected to (mt) other than by being a customer, who happened to be caught in this weekend’s failure, even after sensing something was going wrong a few days before and submitting a ticket asking to me moved clusters 3 days before the failure… I wasn’t moved and was down most of the weekend.

    Levi Figueira

    March 2, 2009 at 5:07 pm

  11. […] setup this blog. I got a lot of attention for rallying the twitter/blog-o-sphere. Back then I got a phone call from a V.P. at Media Temple followed by personal phone number for V.P. at Blue Arc (mt hardware supplier). They both sorta […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: