How #PhillyTech grappled with an Amazon Web Services outage

A Slack tipster — namely Engine Room dev Mike Trischetta — slid into my DMs yesterday with a relevant question:

As it turns out: it did.

Amazon’s Simple Storage Service (S3) platform, used by companies like Netflix and AirBnB, had a serious crash yesterday afternoon. Per the site’s status page, there were “Increased Error Rates” present in the US-EAST-1 Region (based in North Virginia.)

As you can imagine, the outage made Tuesday a bit of a day for some of Philly’s tech firms, many of which rely on S3 — or services that also use it.

“Until [S3] is back up our clients are experiencing delays loading their data,” said Stitch CEO Jake Stein of the outage. “Fortunately, we’ve set things up such that no data is lost when there are outages like this. In addition to our product, a lot of the tools we use rely on S3, so it’s definitely impacting our productivity.”

We’re doing good – we’ve got our backup hot-standby physical data pipeline cranking away. pic.twitter.com/jA4AREk22y

— Stitch Data (@stitch_data) February 28, 2017

For Women in Tech Summit organizers (more on that event later this week,) the outage knocked out their ticketing sales.

. @TechnicallyPHL Our site & ticketing when down for a bit. Looks like it might be back up. Fingers crossed!

— Women In Tech Summit (@WomenTechSummit) February 28, 2017

To this, Ticketleap CEO Tim Raybould jokingly quipped:

or you could switch your ticketing 😉

— Tim Raybould (@timraybould) February 28, 2017

However, Ticketleap didn’t survive the outage unscathed: Raybould told Technical.ly that images on the company’s event pages weren’t loading and order confirmation emails were interrupted, but orders were still going through.

MailChimp, the broadly-used email blasting service, had trouble loading images onto newsletters.

Due to the AWS S3 outage, we're still seeing trouble with images. We’re actively monitoring AWS and will update as more info comes through.

— Mailchimp Status (@MailchimpStatus) February 28, 2017

This inconvenienced even the ever-serene Charlotte Lee, Benjamin’s Desk‘s director of programming.

Code for Philly comms lead and GIF connoisseur Pat Woods ran into some trouble on Slack.

missing my /giphy tools. pic.twitter.com/GprWGuosW7

— Pat Woods (@patwoodsdesign) February 28, 2017

Also heartbroken was former Technical.ly Delaware lead reporter Rana Fayez, who pivoted to the dev field.

Everything I need is hosted on AWS S3 and I don't know what to do with my life RN with this outage 💔

— Rana Fayez (@_tagine) February 28, 2017

CloudMine VP of Product Nick Borth said the firm hosts HIPAA compliant websites and backend storage in S3, so backup automation and the company’s failover strategy pretty much saved the day.

“This approach significantly lowers the risk of compromising data, and maintaining availability for our clients and partners in sensitive industries,” said Borth in an email.

The outage ended around 6:00 p.m., but before then, one solution was offered by Lancaster, Pa.- based entrepreneur Dave Conklin:

.@comcast … can you give @awscloud a call and ask them to unplug it for 10 seconds, and plug it back in please? #aws #awsdown

— Dave Conklin – Lancaster PA (@DaveConklin) February 28, 2017