Here's what it takes to run the Philadelphia School District's IT network - Technical.ly Philly

Dev

Mar. 26, 2015 12:45 pm

Here’s what it takes to run the Philadelphia School District’s IT network

On a fraction of the budget that a private company (or even other school districts) might have, 20 IT staffers are quietly maintaining one of the city's largest networks. For school districts, it's also one of the nation's fastest, according to a recent report.

School District of Philadelphia Network Operations Manager Scott Fry.

(Photo by Marcus McCarthy)

There’s a lot of color in Scott Fry’s life.

He’s the manager of network operations at the School District of Philadelphia, one of the country’s biggest school districts. And for all the budget battles and bureaucratic blockades, somebody has to keep the servers running, the internet connected and administrative IT services in place.

To do so, Fry monitors a live list on his laptop screen full of green and red boxes that occasionally switch colors. At his fingertips is the status of the entire internet infrastructure of the School District, which includes a network of fiberoptic cables that connects 200 facilities.

Those red boxes on his laptop are indicators that something is down. The very rare purple boxes are worse: they indicate that a large clump of the network is down at once. Fry explains that this program is an example of early-warning systems, which allows the members of the district’s Technology Services team to effectively use their time: they’re under-manned as is.

This status information will be used to dispatch one of five network engineers from the Technology Services office, which has a little more than 20 employees. That team oversees a network that includes a 7,000-square-foot data center at District headquarters on North Broad Street, featuring 500 terabytes of digital storage, a backup generator and 20 physical servers. The District also has a backup center in Northeast Philadelphia and 19 distributed core centers at various schools.

“It’s constant work. It’s never monotonous,” said Mike Reagan, one of the network engineers. “We’ve had our share of 30-plus-hour weekends.”

Technology Services additionally share the fate of other district departments in being persistently underfunded. In the Fiscal Year 2013 consolidated budget, the School District of Philadelphia allotted 0.7 percent of its budget to Technology Services. For comparison, IT spending at private companies can reach up to 9 percent of the company’s total budget. Other large school districts largely range from 1 to 4.5 percent.

Advertisement

Fry is aware of these issues, but isn’t intimidated.

“There’s always something to do,” Fry said. “Personally, I think we’re ahead. I’m sure there would be people who would love to have what we have.”

Despite the small size of the office and it’s budget, network speeds in the district are some of the highest in the country. In last year’s annual performance survey by the Council of the Great City Schools [PDF], the School District of Philadelphia was at the top of the list of bandwidth per 1,000 students. By contrast, just nine percent of American school districts said they had the bandwidth to support online assessments, according to a 2014 survey.

Bandwidth

(Courtesy of Council of the Great City Schools)

(Editor’s note: You can find the chart at right on page 161 of the above PDF. District names are withheld in the report, but District officials told us their code, which is used throughout. The District’s bandwidth gains are especially impressive considering IT spending data ranked in neighboring charts. The Notebook has more analysis of the District’s strengths and weaknesses relating to the recent report.)

This is a point of pride for the team and its leader, Bob Westall. Westall is the Deputy Chief Information Officer and directly oversees the Technology Services team. (Editor’s note: Westall was put on administrative leave in 2011 during an internal procurement investigation. Westall was cleared of any wrongdoing by an investigation from law firm Pepper Hamilton, according to an internal District document from July 2012 shared with Technical.ly Philly.)

Westall said the district’s fiberoptic network was created in 1999 and was a step ahead of other large urban school districts, which has contributed to the Philadelphia district’s edge with internet speeds. At the time, other school districts like the City of Chicago School District were buying hundreds of physical servers, a costly practice.

Philadelphia School District couldn’t afford this practice, Westall said, so they found another way: a high speed cable network allowing more effective use of their fewer physical servers.  The use of mass servers has since been outdated by faster wiring, opening the possibilities to virtual servers and cloud storage, putting Philadelphia on the right side of history.

About 85 percent of the network was built using federal dollars, according to the Notebook:

The School District has been among the best nationwide in securing federally subsidized E-rate dollars for its technology spending, said Chief Information Officer Harris. The District has gotten more than $312 million in E-rate reimbursements since that program for schools began in 1998.

“[The goal is to] design it from the beginning so it doesn’t cause issues,” Fry said. “As long as we stay ahead of the curve, we maintain the leading edge.”

Despite the successes of the district’s IT team, a technological problem that has continued to plague the district is its lack of funding for new hardware, said Fry. The average age of district devices — like computers — last year was more than five years old (we’ve seen the repercussions of that at various schools), which also ranks close to the bottom in comparison to other urban school districts nationwide.

Partly in response to this issue, the district has established a program that allows students and teachers to bring their personal devices to school and connect to the WiFi. The program, fitting with the national workplace Bring Your Own Device trend, allows for the use of laptops, tablets, cell phones, e-readers and iPods to connect to district WiFi. Some schools are using the model to get donated hardware into classrooms.

Additionally, Chromebooks have become a viable option for aiding schools in updating their hardware. They are cheaper than most computers since they largely use cloud storage for their data and applications. This plays to the bandwidth strength of the district.

In the last year, 175 schools opted in to using Chromebooks, accounting for 8,000 being received. For example, the Workshop School in University City uses them, which also makes it easier for their students to connect to Google Apps for school projects.

There’s innovation at the District in IT, something that Westall said he hopes more local technologists will come to know. Turns out it’s challenging to find the right candidate for district tech jobs because of Philadelphia’s active technology scene and less-competitive salaries available within the public school system.

“It’s so hard to fill a vacancy. I have to sell them on the mission,” Westall said. “Anyone here could walk to the Comcast building and double their salary.”

Richard Bateman, executive director of technical operations, said he sleeps easier working with the school system. He said he previously worked on re-entry systems for Intercontinental Ballistic Missiles, which were commonly designed to carry nuclear warheads.

“I definitely feel better working for students than working on missiles,” Bateman said, smiling.

You must appreciate accurate, relevant and productive community journalism.  Support this sort of work from professional reporters with seasoned editors.  Become a Technical.ly member for $12 per month -30-
JOIN THE COMMUNITY, BECOME A MEMBER
Already a member? Sign in here
Connect with companies from the Technical.ly community
New call-to-action

Advertisement

Penn is launching a 24-week cybersecurity bootcamp

This tech founder will soon be growing her North Carolina nonprofit from Philly [Entrance Exam]

USciences just announced an MBA program for the business of cannabis

SPONSORED

Philly

Why this dev planted roots at The Philadelphia Inquirer after starting her career abroad

Philadelphia, PA

Technical.ly Match

Web Manager

Apply Now
Philadelphia, PA

PromptWorks

Senior Software Engineer

Apply Now
Chesterrbrook, PA

Deacom

Software Quality Analyst

Apply Now

College is creating poverty, so let’s make it free, says Temple’s Sara Goldrick-Rab

Tech education nonprofit Per Scholas is opening a new Philly location

ISTE 2019: How technology will change careers and industries over the next 10 years

SPONSORED

Philly

Meet these 10 hiring companies at Super Meetup

Philadelphia

Vistar Media

Front End Engineer

Apply Now
Philadelphia

Vistar Media

Sr. Software Engineer

Apply Now
Philadelphia, PA - Center City

Odessa

Sr. Project Manager

Apply Now

Sign-up for daily news updates from Technical.ly Philadelphia

Do NOT follow this link or you will be banned from the site!