How #PhillyTech grappled with an Amazon Web Services outage - Philly


Mar. 1, 2017 10:02 am

How #PhillyTech grappled with an Amazon Web Services outage

Devs across town spent most of Tuesday fretting due to a regional S3 outage in the popular online platform. Here's how some companies fared.

If only the other cloud had been moving like this.

(GIF via

A Slack tipster — namely Engine Room dev Mike Trischetta — slid into my DMs yesterday with a relevant question:

mike comment

As it turns out: it did.

Amazon’s Simple Storage Service (S3) platform, used by companies like Netflix and AirBnB, had a serious crash yesterday afternoon. Per the site’s status page, there were “Increased Error Rates” present in the US-EAST-1 Region (based in North Virginia.)

As you can imagine, the outage made Tuesday a bit of a day for some of Philly’s tech firms, many of which rely on S3 — or services that also use it.

“Until [S3] is back up our clients are experiencing delays loading their data,” said Stitch CEO Jake Stein of the outage. “Fortunately, we’ve set things up such that no data is lost when there are outages like this. In addition to our product, a lot of the tools we use rely on S3, so it’s definitely impacting our productivity.”

For Women in Tech Summit organizers (more on that event later this week,) the outage knocked out their ticketing sales.

To this, Ticketleap CEO Tim Raybould jokingly quipped:

However, Ticketleap didn’t survive the outage unscathed: Raybould told that images on the company’s event pages weren’t loading and order confirmation emails were interrupted, but orders were still going through.

MailChimp, the broadly-used email blasting service, had trouble loading images onto newsletters.

This inconvenienced even the ever-serene Charlotte Lee, Benjamin’s Desk‘s director of programming.


Code for Philly comms lead and GIF connoisseur Pat Woods ran into some trouble on Slack.

Also heartbroken was former Delaware lead reporter Rana Fayez, who pivoted to the dev field.

CloudMine VP of Product Nick Borth said the firm hosts HIPAA compliant websites and backend storage in S3, so backup automation and the company’s failover strategy pretty much saved the day.

“This approach significantly lowers the risk of compromising data, and maintaining availability for our clients and partners in sensitive industries,” said Borth in an email.

The outage ended around 6:00 p.m., but before then, one solution was offered by Lancaster, Pa.- based entrepreneur Dave Conklin:


Roberto Torres

Roberto Torres became Lead Philly Reporter in May 2016. Prior, he was a freelance contributor to and Al Dia News. The native Venezuelan moved to Philadelphia in 2015 after reporting on research at his alma mater, the University of Zulia. Whenever he's not fencing deadlines, he can be found standing in line at Overbrook Pizza in West Philly, running Netflix/Hulu marathons with his wife or reading news from Venezuela.


Sign-up for regular updates from

Do NOT follow this link or you will be banned from the site!