Here's what it takes to run the Philadelphia School District's IT network - Technical.ly Philly

Dev

Mar. 26, 2015 12:45 pm

Here’s what it takes to run the Philadelphia School District’s IT network

On a fraction of the budget that a private company (or even other school districts) might have, 20 IT staffers are quietly maintaining one of the city's largest networks. For school districts, it's also one of the nation's fastest, according to a recent report.

School District of Philadelphia Network Operations Manager Scott Fry.

(Photo by Marcus McCarthy)

There’s a lot of color in Scott Fry’s life.

He’s the manager of network operations at the School District of Philadelphia, one of the country’s biggest school districts. And for all the budget battles and bureaucratic blockades, somebody has to keep the servers running, the internet connected and administrative IT services in place.

To do so, Fry monitors a live list on his laptop screen full of green and red boxes that occasionally switch colors. At his fingertips is the status of the entire internet infrastructure of the School District, which includes a network of fiberoptic cables that connects 200 facilities.

Those red boxes on his laptop are indicators that something is down. The very rare purple boxes are worse: they indicate that a large clump of the network is down at once. Fry explains that this program is an example of early-warning systems, which allows the members of the district’s Technology Services team to effectively use their time: they’re under-manned as is.

This status information will be used to dispatch one of five network engineers from the Technology Services office, which has a little more than 20 employees. That team oversees a network that includes a 7,000-square-foot data center at District headquarters on North Broad Street, featuring 500 terabytes of digital storage, a backup generator and 20 physical servers. The District also has a backup center in Northeast Philadelphia and 19 distributed core centers at various schools.

“It’s constant work. It’s never monotonous,” said Mike Reagan, one of the network engineers. “We’ve had our share of 30-plus-hour weekends.”

Technology Services additionally share the fate of other district departments in being persistently underfunded. In the Fiscal Year 2013 consolidated budget, the School District of Philadelphia allotted 0.7 percent of its budget to Technology Services. For comparison, IT spending at private companies can reach up to 9 percent of the company’s total budget. Other large school districts largely range from 1 to 4.5 percent.

Advertisement

Fry is aware of these issues, but isn’t intimidated.

“There’s always something to do,” Fry said. “Personally, I think we’re ahead. I’m sure there would be people who would love to have what we have.”

Despite the small size of the office and it’s budget, network speeds in the district are some of the highest in the country. In last year’s annual performance survey by the Council of the Great City Schools [PDF], the School District of Philadelphia was at the top of the list of bandwidth per 1,000 students. By contrast, just nine percent of American school districts said they had the bandwidth to support online assessments, according to a 2014 survey.

Bandwidth

(Courtesy of Council of the Great City Schools)

(Editor’s note: You can find the chart at right on page 161 of the above PDF. District names are withheld in the report, but District officials told us their code, which is used throughout. The District’s bandwidth gains are especially impressive considering IT spending data ranked in neighboring charts. The Notebook has more analysis of the District’s strengths and weaknesses relating to the recent report.)

This is a point of pride for the team and its leader, Bob Westall. Westall is the Deputy Chief Information Officer and directly oversees the Technology Services team. (Editor’s note: Westall was put on administrative leave in 2011 during an internal procurement investigation. Westall was cleared of any wrongdoing by an investigation from law firm Pepper Hamilton, according to an internal District document from July 2012 shared with Technical.ly Philly.)

Westall said the district’s fiberoptic network was created in 1999 and was a step ahead of other large urban school districts, which has contributed to the Philadelphia district’s edge with internet speeds. At the time, other school districts like the City of Chicago School District were buying hundreds of physical servers, a costly practice.

Philadelphia School District couldn’t afford this practice, Westall said, so they found another way: a high speed cable network allowing more effective use of their fewer physical servers.  The use of mass servers has since been outdated by faster wiring, opening the possibilities to virtual servers and cloud storage, putting Philadelphia on the right side of history.

About 85 percent of the network was built using federal dollars, according to the Notebook:

The School District has been among the best nationwide in securing federally subsidized E-rate dollars for its technology spending, said Chief Information Officer Harris. The District has gotten more than $312 million in E-rate reimbursements since that program for schools began in 1998.

“[The goal is to] design it from the beginning so it doesn’t cause issues,” Fry said. “As long as we stay ahead of the curve, we maintain the leading edge.”

Despite the successes of the district’s IT team, a technological problem that has continued to plague the district is its lack of funding for new hardware, said Fry. The average age of district devices — like computers — last year was more than five years old (we’ve seen the repercussions of that at various schools), which also ranks close to the bottom in comparison to other urban school districts nationwide.

Partly in response to this issue, the district has established a program that allows students and teachers to bring their personal devices to school and connect to the WiFi. The program, fitting with the national workplace Bring Your Own Device trend, allows for the use of laptops, tablets, cell phones, e-readers and iPods to connect to district WiFi. Some schools are using the model to get donated hardware into classrooms.

Additionally, Chromebooks have become a viable option for aiding schools in updating their hardware. They are cheaper than most computers since they largely use cloud storage for their data and applications. This plays to the bandwidth strength of the district.

In the last year, 175 schools opted in to using Chromebooks, accounting for 8,000 being received. For example, the Workshop School in University City uses them, which also makes it easier for their students to connect to Google Apps for school projects.

There’s innovation at the District in IT, something that Westall said he hopes more local technologists will come to know. Turns out it’s challenging to find the right candidate for district tech jobs because of Philadelphia’s active technology scene and less-competitive salaries available within the public school system.

“It’s so hard to fill a vacancy. I have to sell them on the mission,” Westall said. “Anyone here could walk to the Comcast building and double their salary.”

Richard Bateman, executive director of technical operations, said he sleeps easier working with the school system. He said he previously worked on re-entry systems for Intercontinental Ballistic Missiles, which were commonly designed to carry nuclear warheads.

“I definitely feel better working for students than working on missiles,” Bateman said, smiling.

You must appreciate accurate, relevant and productive community journalism.  Support this sort of work from professional reporters with seasoned editors.  Become a Technical.ly member for $12 per month -30-
JOIN THE COMMUNITY, BECOME A MEMBER
Already a member? Sign in here

Advertisement

Here’s a roadmap for increasing access to computer science

These Temple alumni are tackling IT security breaches in their new careers

Live streaming and in-app gifting helped Meet Group beat its 2018 revenue goals

SPONSORED

Philly

This startup is striving to deliver the future of freight

Philadelphia

Perpay

UX Designer

Apply Now
Philadelphia, PA - Center City

Odessa

Pre-Sales Business Analyst

Apply Now
Horsham, PA

Penn Mutual

Software Engineer

Apply Now

During CS4Philly Week, students and stakeholders will discuss computer science in schools

Today is the last day to register to vote in Pa.

Here’s why Minecraft matters for Philly public schools

SPONSORED

Philly

Why Deacom’s team prioritizes collaboration and continuous improvement

Horsham, PA

Penn Mutual

Software Engineer-Java

Apply Now
Horsham, PA

Penn Mutual

Product Systems Trainer-LEAP

Apply Now
Horsham, PA

Penn Mutual

Communications Specialist

Apply Now

Sign-up for daily news updates from Technical.ly Philadelphia

Do NOT follow this link or you will be banned from the site!