What Happened?

[As many of you know, MediaCommons disappeared for a couple of days this week. It’s (almost entirely) back now, but we wanted to explain what happened, and to start rebuilding not just the site but our relationship with our community. Thus this, cross-posted from MediaCommons.]

So… funny story.

You know how you get those email messages from your hosting provider, saying “hey, we’re going to upgrade some things on your server”? And you think “Bonus! Upgrades are good!”

We are here to tell you that sometimes upgrades are not so good.

The result of the upgrade that was conducted on the MediaCommons server over Tuesday night was, not to put too fine a point on it, a debacle.

We woke up Wednesday to a completely clean, shiny new server — with no files on it, no configuration settings, and no user accounts through which anyone could log in and, say, rebuild a totally missing website.

The day was a misery for our heroic development team, who tried absolutely everything to get things back up and running as quickly as they could. Thanks to some utterly abysmal customer service and misleading communication from our hosting provider, it wasn’t until the wee hours of Thursday morning that our folks found the source of the problem, which they spent all day Thursday addressing.

To cap everything off, a server configuration mystery prevented our even leaving anything like a meaningful error message on the site, so for an embarrassingly long period of time all we were able to provide were bad redirects, 404s, and our hosting provider’s generic “hey, site administrator, something’s not properly configured here” screen.

Happily, we’ve gotten the bulk of the network restored. There is a bit of recent content still missing, however, and a few other errors to address. If you notice something odd, please leave us information about it in the comments.

We are deeply sorry for any inconvenience that our downtime may have caused our community, and we hope, now that we’re back, that we can begin rebuilding your trust in the stability of our platform. The good news, though, is that we heard from a lot of you over the last couple of days, and so we got a very strong sense of the richness of the community that we’re serving through the various MediaCommons projects.

We are — it will probably not surprise you to hear — actively seeking a new hosting solution. We are hard at work on upgrades to the platform that will increase its stability and its user-friendliness. And we have some very exciting developments in our projects coming soon.

Thanks for sticking it out with us. Together, we can and will do better than this.