Here’s what it takes to run the Philadelphia School District’s IT network

There’s a lot of color in Scott Fry’s life.

He’s the manager of network operations at the School District of Philadelphia, one of the country’s biggest school districts. And for all the budget battles and bureaucratic blockades, somebody has to keep the servers running, the internet connected and administrative IT services in place.
To do so, Fry monitors a live list on his laptop screen full of green and red boxes that occasionally switch colors. At his fingertips is the status of the entire internet infrastructure of the School District, which includes a network of fiberoptic cables that connects 200 facilities.
Those red boxes on his laptop are indicators that something is down. The very rare purple boxes are worse: they indicate that a large clump of the network is down at once. Fry explains that this program is an example of early-warning systems, which allows the members of the district’s Technology Services team to effectively use their time: they’re under-manned as is.
This status information will be used to dispatch one of five network engineers from the Technology Services office, which has a little more than 20 employees. That team oversees a network that includes a 7,000-square-foot data center at District headquarters on North Broad Street, featuring 500 terabytes of digital storage, a backup generator and 20 physical servers. The District also has a backup center in Northeast Philadelphia and 19 distributed core centers at various schools.
“It’s constant work. It’s never monotonous,” said Mike Reagan, one of the network engineers. “We’ve had our share of 30-plus-hour weekends.”
Technology Services additionally share the fate of other district departments in being persistently underfunded. In the Fiscal Year 2013 consolidated budget, the School District of Philadelphia allotted 0.7 percent of its budget to Technology Services. For comparison, IT spending at private companies can reach up to 9 percent of the company’s total budget. Other large school districts largely range from 1 to 4.5 percent.
Fry is aware of these issues, but isn’t intimidated.
“There’s always something to do,” Fry said. “Personally, I think we’re ahead. I’m sure there would be people who would love to have what we have.”
Despite the small size of the office and it’s budget, network speeds in the district are some of the highest in the country. In last year’s annual performance survey by the Council of the Great City Schools [PDF], the School District of Philadelphia was at the top of the list of bandwidth per 1,000 students. By contrast, just nine percent of American school districts said they had the bandwidth to support online assessments, according to a 2014 survey.

(Editor’s note: You can find the chart at right on page 161 of the above PDF. District names are withheld in the report, but District officials told us their code, which is used throughout. The District’s bandwidth gains are especially impressive considering IT spending data ranked in neighboring charts. The Notebook has more analysis of the District’s strengths and weaknesses relating to the recent report.)
This is a point of pride for the team and its leader, Bob Westall. Westall is the Deputy Chief Information Officer and directly oversees the Technology Services team. (Editor’s note: Westall was put on administrative leave in 2011 during an internal procurement investigation. Westall was cleared of any wrongdoing by an investigation from law firm Pepper Hamilton, according to an internal District document from July 2012 shared with Technical.ly Philly.)
Westall said the district’s fiberoptic network was created in 1999 and was a step ahead of other large urban school districts, which has contributed to the Philadelphia district’s edge with internet speeds. At the time, other school districts like the City of Chicago School District were buying hundreds of physical servers, a costly practice.
Philadelphia School District couldn’t afford this practice, Westall said, so they found another way: a high speed cable network allowing more effective use of their fewer physical servers.  The use of mass servers has since been outdated by faster wiring, opening the possibilities to virtual servers and cloud storage, putting Philadelphia on the right side of history.
About 85 percent of the network was built using federal dollars, according to the Notebook:

The School District has been among the best nationwide in securing federally subsidized E-rate dollars for its technology spending, said Chief Information Officer Harris. The District has gotten more than $312 million in E-rate reimbursements since that program for schools began in 1998.

“[The goal is to] design it from the beginning so it doesn’t cause issues,” Fry said. “As long as we stay ahead of the curve, we maintain the leading edge.”

Despite the successes of the district’s IT team, a technological problem that has continued to plague the district is its lack of funding for new hardware, said Fry. The average age of district devices — like computers — last year was more than five years old (we’ve seen the repercussions of that at various schools), which also ranks close to the bottom in comparison to other urban school districts nationwide.
Partly in response to this issue, the district has established a program that allows students and teachers to bring their personal devices to school and connect to the WiFi. The program, fitting with the national workplace Bring Your Own Device trend, allows for the use of laptops, tablets, cell phones, e-readers and iPods to connect to district WiFi. Some schools are using the model to get donated hardware into classrooms.
Additionally, Chromebooks have become a viable option for aiding schools in updating their hardware. They are cheaper than most computers since they largely use cloud storage for their data and applications. This plays to the bandwidth strength of the district.
In the last year, 175 schools opted in to using Chromebooks, accounting for 8,000 being received. For example, the Workshop School in University City uses them, which also makes it easier for their students to connect to Google Apps for school projects.
There’s innovation at the District in IT, something that Westall said he hopes more local technologists will come to know. Turns out it’s challenging to find the right candidate for district tech jobs because of Philadelphia’s active technology scene and less-competitive salaries available within the public school system.
“It’s so hard to fill a vacancy. I have to sell them on the mission,” Westall said. “Anyone here could walk to the Comcast building and double their salary.”
Richard Bateman, executive director of technical operations, said he sleeps easier working with the school system. He said he previously worked on re-entry systems for Intercontinental Ballistic Missiles, which were commonly designed to carry nuclear warheads.
“I definitely feel better working for students than working on missiles,” Bateman said, smiling.