(Photo by Flickr user @undo_everything, used under a Creative Commons license)
It’s been a long winter, and the tech staff at SEPTA knew it all too well.
“It was like a cruel joke,” said Mike Zaleski, SEPTA’s director of emerging technologies. “When you needed it most, it didn’t work.”
It’s been a problem for years, he said, but now it’s getting fixed. Zaleski and his team of about six staffers have been working long days for the last six weeks to make sure that riders can always get the information they need when they need it most. They’re a few weeks from finishing the project, he said.
We spoke to Zaleski about whether the next weather event will slow him down. Here’s what he told us.
Why SEPTA.org crashed all the time
There are two reasons SEPTA.org (and the API that powers its app and every other SEPTA tool out there) crashed every time it snowed:
- Problem 1: When there’s a major storm, everyone wants to know what’s going on with SEPTA. But SEPTA’s old database couldn’t handle all that traffic and would crash when it got hit with too many requests.
- Problem 2: The SEPTA server that handles all the requests for online transit information is out in a data center in West Chester, and every time there was a major storm, that data center would lose power.
How SEPTA is fixing those problems
The website and API could definitely still go down, but these fixes will make sure it doesn’t go down during normal and high-traffic usage, Zaleski said.
- Problem 1: SEPTA replaced their old database with two stronger ones that could handle more traffic and share the burden. They were pieces of hardware that SEPTA already owned, so SEPTA only had to buy a load balancer (about $40,000), a piece of hardware that distributes the traffic between the two databases.
- Problem 2: SEPTA added two more server locations so that they are no longer relying on that one space in West Chester — a level of redundancy that most network administrators will tell you is an absolute necessity. They added one at their headquarters at 1234 Market Street (about $6,000), as well as one in the cloud with Amazon Web Services (it’s hard to break down this cost because SEPTA has other servers in the cloud with AWS and the cost is pay-as-you-go, so the cost will be about $1000/month). They’re also putting the finishing touches on another load balancer for these servers.
- The main cost was staff time, Zaleski said.
Why it didn’t happen sooner
SEPTA knew about these problems and had discussed fixing them, but it didn’t happen until now for a few reasons, Zaleski said.
- SEPTA’s short on staff
- It’s hard to get buy-in for something that people don’t totally understand. Say the word “API” and people’s eyes glaze over. But once Zaleski put it in terms of SEPTA apps (think about your favorite SEPTA app, he would say, it’s SEPTA’s API that powers that), he was able to get support for the project.
- Part of the reason SEPTA made the fixes this winter was because the weather was so bad. The winters before that were milder. But after Philly got hit by storm after storm, the need became very clear.
Hiring for cybersecurity is on the rise in Philadelphia: ‘There’s not an industry that’s immune’ from attacks
New app ClipDish saves only the recipe (and nothing else) from cooking sites
Now Siri can tell you just how badly SEPTA is f*cked today
SAP just opened a cyber defense center at its Newtown Square HQ
Sign-up for daily news updates from Technical.ly Philadelphia