Why SEPTA.org crashed so often, and how it's getting fixed - Technical.ly Philly

Civic

Mar. 28, 2014 12:30 pm

Why SEPTA.org crashed so often, and how it’s getting fixed

It's been a long winter, and the tech staff at SEPTA knew it all too well. That's because every time it snowed, SEPTA's website (plus its app and all other SEPTA apps) would crash.

SEPTA.

(Photo by Flickr user @undo_everything, used under a Creative Commons license)

It’s been a long winter, and the tech staff at SEPTA knew it all too well.

That’s because every time it snowed, SEPTA’s website (plus its app and all other SEPTA apps) would crash.

“It was like a cruel joke,” said Mike Zaleski, SEPTA’s director of emerging technologies. “When you needed it most, it didn’t work.”

It’s been a problem for years, he said, but now it’s getting fixed. Zaleski and his team of about six staffers have been working long days for the last six weeks to make sure that riders can always get the information they need when they need it most. They’re a few weeks from finishing the project, he said.

septa.org

SEPTA’s report from pingdom.com. Since Nov. 2012, SEPTA has had 11 hours and 52 minutes of downtime, or 99.9% uptime. Still, Michael Zaleski said, when there is downtime, it happens for a large chunk of time.

We spoke to Zaleski about whether the next weather event will slow him down. Here’s what he told us.

Why SEPTA.org crashed all the time

There are two reasons SEPTA.org (and the API that powers its app and every other SEPTA tool out there) crashed every time it snowed:

  • Problem 1: When there’s a major storm, everyone wants to know what’s going on with SEPTA. But SEPTA’s old database couldn’t handle all that traffic and would crash when it got hit with too many requests.
  • Problem 2: The SEPTA server that handles all the requests for online transit information is out in a data center in West Chester, and every time there was a major storm, that data center would lose power.

How SEPTA is fixing those problems

Advertisement

The website and API could definitely still go down, but these fixes will make sure it doesn’t go down during normal and high-traffic usage, Zaleski said.

  • Problem 1: SEPTA replaced their old database with two stronger ones that could handle more traffic and share the burden. They were pieces of hardware that SEPTA already owned, so SEPTA only had to buy a load balancer (about $40,000), a piece of hardware that distributes the traffic between the two databases.
  • Problem 2: SEPTA added two more server locations so that they are no longer relying on that one space in West Chester — a level of redundancy that most network administrators will tell you is an absolute necessity. They added one at their headquarters at 1234 Market Street (about $6,000), as well as one in the cloud with Amazon Web Services (it’s hard to break down this cost because SEPTA has other servers in the cloud with AWS and the cost is pay-as-you-go, so the cost will be about $1000/month). They’re also putting the finishing touches on another load balancer for these servers.
  • The main cost was staff time, Zaleski said.

Why it didn’t happen sooner

SEPTA knew about these problems and had discussed fixing them, but it didn’t happen until now for a few reasons, Zaleski said.

  • SEPTA’s short on staff
  • It’s hard to get buy-in for something that people don’t totally understand. Say the word “API” and people’s eyes glaze over. But once Zaleski put it in terms of SEPTA apps (think about your favorite SEPTA app, he would say, it’s SEPTA’s API that powers that), he was able to get support for the project.
  • Part of the reason SEPTA made the fixes this winter was because the weather was so bad. The winters before that were milder. But after Philly got hit by storm after storm, the need became very clear.
Companies: SEPTA
People: Mike Zaleski
-30-
JOIN THE COMMUNITY, BECOME A MEMBER
Already a member? Sign in here

Advertisement

This cybersecurity firm is expanding its Philly HQ and adding a Rochester office

Here’s everything we know about the Philly courts website shutdown

Check out this data viz on SEPTA’s elevator accessibility

SPONSORED

Philly

This financial services firm offers global opportunity in the heart of Philly

Philadelphia

The Philadelphia Inquirer

Mobile Tech Lead

Apply Now
Blue Bell, PA

Pinnacle 21

Senior Software Engineers

Apply Now
Philadelphia

Guru Technologies

Marketing Web Engineer

Apply Now

Hear from the privacy pros at Security by the Schuylkill

Have a look at SEPTA Key’s revamped website in beta mode

Comcast just bought this cybersecurity firm from LLR Partners

SPONSORED

Philly

Guru is taking on growth with deliberate steps and an engaged team

Philadelphia, PA

PromptWorks

Senior UX Designer

Apply Now
601 Lee Road Chesterbrook, PA 19087

Deacom

Web Application Developer

Apply Now
Chesterbrook, PA

Deacom

Software Developer

Apply Now

Sign-up for daily news updates from Technical.ly Philadelphia

Do NOT follow this link or you will be banned from the site!