Why SEPTA.org crashed so often, and how it’s getting fixed

It’s been a long winter, and the tech staff at SEPTA knew it all too well.

That’s because every time it snowed, SEPTA’s website (plus its app and all other SEPTA apps) would crash.

“It was like a cruel joke,” said Mike Zaleski, SEPTA’s director of emerging technologies. “When you needed it most, it didn’t work.”

It’s been a problem for years, he said, but now it’s getting fixed. Zaleski and his team of about six staffers have been working long days for the last six weeks to make sure that riders can always get the information they need when they need it most. They’re a few weeks from finishing the project, he said.

We spoke to Zaleski about whether the next weather event will slow him down. Here’s what he told us.

Why SEPTA.org crashed all the time

There are two reasons SEPTA.org (and the API that powers its app and every other SEPTA tool out there) crashed every time it snowed:

• Problem 1: When there’s a major storm, everyone wants to know what’s going on with SEPTA. But SEPTA’s old database couldn’t handle all that traffic and would crash when it got hit with too many requests.
• Problem 2: The SEPTA server that handles all the requests for online transit information is out in a data center in West Chester, and every time there was a major storm, that data center would lose power.

How SEPTA is fixing those problems

The website and API could definitely still go down, but these fixes will make sure it doesn’t go down during normal and high-traffic usage, Zaleski said.

• Problem 1: SEPTA replaced their old database with two stronger ones that could handle more traffic and share the burden. They were pieces of hardware that SEPTA already owned, so SEPTA only had to buy a load balancer (about $40,000), a piece of hardware that distributes the traffic between the two databases.
• Problem 2: SEPTA added two more server locations so that they are no longer relying on that one space in West Chester — a level of redundancy that most network administrators will tell you is an absolute necessity. They added one at their headquarters at 1234 Market Street (about $6,000), as well as one in the cloud with Amazon Web Services (it’s hard to break down this cost because SEPTA has other servers in the cloud with AWS and the cost is pay-as-you-go, so the cost will be about $1000/month). They’re also putting the finishing touches on another load balancer for these servers.
• The main cost was staff time, Zaleski said.

Why it didn’t happen sooner

SEPTA knew about these problems and had discussed fixing them, but it didn’t happen until now for a few reasons, Zaleski said.

• SEPTA’s short on staff
• It’s hard to get buy-in for something that people don’t totally understand. Say the word “API” and people’s eyes glaze over. But once Zaleski put it in terms of SEPTA apps (think about your favorite SEPTA app, he would say, it’s SEPTA’s API that powers that), he was able to get support for the project.
• Part of the reason SEPTA made the fixes this winter was because the weather was so bad. The winters before that were milder. But after Philly got hit by storm after storm, the need became very clear.