Civic News

Why SEPTA.org crashed so often, and how it’s getting fixed

It's been a long winter, and the tech staff at SEPTA knew it all too well. That's because every time it snowed, SEPTA's website (plus its app and all other SEPTA apps) would crash.

The Chesapeake Bay. (Photo obtained via Creative Commons license by Wikimedia Commons user Farragutful)

It’s been a long winter, and the tech staff at SEPTA knew it all too well.

That’s because every time it snowed, SEPTA’s website (plus its app and all other SEPTA apps) would crash.

“It was like a cruel joke,” said Mike Zaleski, SEPTA’s director of emerging technologies. “When you needed it most, it didn’t work.”

It’s been a problem for years, he said, but now it’s getting fixed. Zaleski and his team of about six staffers have been working long days for the last six weeks to make sure that riders can always get the information they need when they need it most. They’re a few weeks from finishing the project, he said.

septa.org

SEPTA’s report from pingdom.com. Since Nov. 2012, SEPTA has had 11 hours and 52 minutes of downtime, or 99.9% uptime. Still, Michael Zaleski said, when there is downtime, it happens for a large chunk of time.

We spoke to Zaleski about whether the next weather event will slow him down. Here’s what he told us.

Why SEPTA.org crashed all the time

There are two reasons SEPTA.org (and the API that powers its app and every other SEPTA tool out there) crashed every time it snowed:

  • Problem 1: When there’s a major storm, everyone wants to know what’s going on with SEPTA. But SEPTA’s old database couldn’t handle all that traffic and would crash when it got hit with too many requests.
  • Problem 2: The SEPTA server that handles all the requests for online transit information is out in a data center in West Chester, and every time there was a major storm, that data center would lose power.

How SEPTA is fixing those problems

The website and API could definitely still go down, but these fixes will make sure it doesn’t go down during normal and high-traffic usage, Zaleski said.

  • Problem 1: SEPTA replaced their old database with two stronger ones that could handle more traffic and share the burden. They were pieces of hardware that SEPTA already owned, so SEPTA only had to buy a load balancer (about $40,000), a piece of hardware that distributes the traffic between the two databases.
  • Problem 2: SEPTA added two more server locations so that they are no longer relying on that one space in West Chester — a level of redundancy that most network administrators will tell you is an absolute necessity. They added one at their headquarters at 1234 Market Street (about $6,000), as well as one in the cloud with Amazon Web Services (it’s hard to break down this cost because SEPTA has other servers in the cloud with AWS and the cost is pay-as-you-go, so the cost will be about $1000/month). They’re also putting the finishing touches on another load balancer for these servers.
  • The main cost was staff time, Zaleski said.

Why it didn’t happen sooner

SEPTA knew about these problems and had discussed fixing them, but it didn’t happen until now for a few reasons, Zaleski said.

  • SEPTA’s short on staff
  • It’s hard to get buy-in for something that people don’t totally understand. Say the word “API” and people’s eyes glaze over. But once Zaleski put it in terms of SEPTA apps (think about your favorite SEPTA app, he would say, it’s SEPTA’s API that powers that), he was able to get support for the project.
  • Part of the reason SEPTA made the fixes this winter was because the weather was so bad. The winters before that were milder. But after Philly got hit by storm after storm, the need became very clear.
Companies: SEPTA

Before you go...

Please consider supporting Technical.ly to keep our independent journalism strong. Unlike most business-focused media outlets, we don’t have a paywall. Instead, we count on your personal and organizational support.

Our services Preferred partners The journalism fund
Engagement

Join our growing Slack community

Join 5,000 tech professionals and entrepreneurs in our community Slack today!

Trending

Expect even more drones over NJ, Delaware and PA this year — thanks to agriculture, the military and hobbyists

Four ways Trump could upend the tech world in his first 100 days

Federal agency selects Philadelphia for $12M award to support digital equity 

Why did North Philly shift toward Trump in 2024? ‘Social issues do not feed your family’

Technically Media