BuzzFeed Survives Sandy Outage With Instant Amazon Upgrade

It may be the quickest cloud conversion in history. Last night, news site BuzzFeed went down in the aftermath of Hurricane Sandy. By 6 a.m. EDT, BuzzFeed's flooded data center was still waiting to be bailed out by the Army Corps. of Engineers. But the site itself was back online -- hosted on Amazon's cloud computing platform. Startups like Amazon because they can get up and running quickly, but when Sandy struck last night, and power went out in Lower Manhattan, BuzzFeed found itself looking for the shortest go-live deadline imaginable.
Image may contain Text Label Advertisement and Poster
Photo:John Lopez/Flickr

It may be the quickest cloud conversion in history.

Last night, news site BuzzFeed went down in the aftermath of Hurricane Sandy. By 6 a.m. EDT, BuzzFeed's flooded data center was still waiting to be bailed out by the Army Corps. of Engineers. But the site itself was back online -- hosted on Amazon's cloud computing platform.

Startups like Amazon because they can get up and running quickly, but when Sandy struck last night, and power went out in Lower Manhattan, BuzzFeed found itself looking for the shortest go-live deadline imaginable.

The site started reporting problems around 7 p.m., according to BuzzFeed editor in chief Ben Smith. Around 9 p.m., he got a message from Datagram, the company that runs the Whitehall Street data center that houses BuzzFeed's servers. Con Edison -- the power company -- had cut off power, and a basement fuel pump designed to move diesel to the site's backup generators was submerged.

Datagram's techies struggled in the dark, trying desperately to get the pump back online, but by midnight, it was clear that they weren't getting anywhere.

Because BuzzFeed pages were largely cached on the Akamai content delivery network, most visitors could still view the site, but without their servers, editors couldn't post new items to Buzzfeed.com. They were still cranking out news, but on Tumblr and Facebook and Twitter.

For a site that considers social networking part of its DNA, this wasn't a big leap. "Our basic view of the world is that social distribution is how content gets around," says Smith. "We sometimes joke that the logical extension of that view is that you don't have a website anymore.... So in some ways this was a bit of an experiment that challenged us to put our money where our mouth was."

Eugene Ventimiglia.

Photo:BuzzFeed

By midnight, though, it was clear that Datagram wasn't coming back soon. And, experiment aside, BuzzFeed wanted its website back. "We made the decision right then to go to Amazon," Smith says.

The website's geeks then started the epic coding task of rebuilding BuzzFeed.com from backups on brand new virtual servers they'd bought online thorough Amazon. They worked through the night, and they worked through the storm. At one point, a tree crashed through the roof of BuzzFeed system administrator Eugene Ventimiglia.

He kept right on coding.

The geeks had a bare-bones version of BuzzFeed hosted on Amazon by 6 a.m. By noon -- twelve hours after they'd started -- the Amazon hosted site was pretty well fleshed out, Smith says.

BuzzFeed prides itself on taking a roll-your-own approach to product development, but this was quite a feat. But Smith gives full credit to the geeks. "Our developers can do anything as far as I'm concerned," he says.