The Tech Behind: What I learned about internet connectivity during a tour of a Comcast control center

The Tech Behind is a series in which we explore the technology that powers notable Philadelphia institutions. Have an idea for our next edition? Tell us.

Even though I’m a technology reporter, the development of the internet has somewhat been a mystery to me. In my early teen years, I depended on dial-up to do homework, and now, like many Americans, I use Wi-Fi to work remotely from home. Most activities I do in a day are internet enabled. Until it suddenly stops working, I do not think about how internet gets to my third-floor walk up. It’s a set-it-and-forget-it utility for me.

But throughout August, Technical.ly reporters have been diving into digital infrastructure — the concepts and realities that inform the internet today. We’ve talked funding for access, programs that keep school kids and others connected, and the security, equity, access, sustainability and related concerns of the digital world.

I hadn’t learned much about the physical architecture that actually powers the internet until last week. On a steamy August day, I drove to one of the Center City-headquartered Comcast’s headends, one of 11 Philly-based control centers for internet and cable. This one, a plain-looking facility with a collection of the company’s new, colorful technician trucks parked outside, is nestled in West Philadelphia. The 4,500-square-foot facility holds machines that power cable and internet to thousands of customers in the surrounding region and serves as an office space for those deploying out to the field.

A handful of Comcast employees — directors, technicians and communications pros — invited me on an on-background tour of the facility for a look into its processes. Xfinity, Comcast’s internet service, is currently the largest internet service provider in Philadelphia, and services 60 million homes and businesses in the country.

Servers in a Comcast headend. (Photo by Paige Gross)

At the start of the tour, we stepped into a large room, whirring with the sound of air conditioning fans. To keep the machines cool and operating at their best, a director tells me, they’re always kept at 68 degrees. Thousands of feet of colorful wires run overhead, color coded by service, so in the event of maintenance, technicians know where to target.

The room is filled with rows and rows of the hardware that send out Xfinity cable and internet signals to nodes out in the field. The transmitters, of which there are thousands, each correspond with a few blocks of signal out in the city. The skinny devices run both cable and internet through this system. Some rows of devices are sending out high-speed data signals to nodes out in the field, while other rows are receiving requests, like ordering up an on-demand video.

The director notes Comcast’s focus on redundancy — meaning that they have a backup plan on top of their backup plan. The facility houses generators and backup machines, plus newer versions for when the technology is ready to advance. The main hardware gets a tech refresh every five to seven years, the director said, and at any given time, at any site, they’re changing out hardware for updated versions.

Next-gen hardware at a Comcast headend. (Photo by Paige Gross)

On the other half of the whirring room are more lockers of devices, though these are poised for the future — the next-gen, server-driven architecture of Xfinity. The devices on this side of the room take up noticeably less space, and as hardware updates come, the company is able to do more with less machines, the director says. A quick scan of the whole room shows the evolution of hardware, and how capacity has changed (and shrunk) the amount of space needed to run service to thousands of customers.

In one of the rows of machines is a turn panel that sends the signal out to the field, with the blue, orange, white and yellow wires here in the West Philly headend correlating with minuscule wires in nodes out in the field. But the signal first stops in at a splice cabinet, where outside plant cable and inside plant cable meet and marry.

A turn panel. (Photo by Paige Gross)

A splice cabinet. (Photo by Paige Gross)

Minutes later, I’m on a field trip to a massive apartment complex off of City Avenue a few miles away. As we drive to the spot, I’m pointed to the telephone wires running along the street, which also carry Comcast fiber.

Soon, two network maintence technicians are deconstructing a node for me. We’re in a concrete room in the utilities part of the building, where an Xfinity node lives alongside other internet service providers’ nodes. The fibers run to the buildings — either on telephone poles or by running underground — and meet here, in the node. Every node out in the field is physically connected via wires and fiber back to one of Comcast’s headend facilities.

Headend employees deconstruct a node in the field. (Photo by Paige Gross)

Tiny fibers in a Comcast node. (Photo by Paige Gross)

The fibers here look minuscule compared to the ones in the headends, each about the width of a strand of hair, because they don’t have the same coating as those in the facility. I learn that each of the four buildings in this massive complex get their own node.

So when your internet or cable is out, I ask, what is actually happening out here? It could be any number of things, one employee says. It could be that a car crashed into a telephone poll carrying the fiber your router relies on, an issue with the website or VPN you’re using, or something within your home.

Comcast EVP and Chief Network Officer Elad Nafshi to me in an interview a few days later that the company rolled out XMF, a new network technology that helps reduce the amount of time it takes for a technician to find and isolate a fiber cut, last year.

“Picture, if you will, a truck that for whatever reason, ran into a pole right out Fairmount Avenue and now we have a fiber cut,” Nafshi said. “You’d be amazed how many times a day that happens.”

The AI they’ve developed can pinpoint the location of the fiber cut and how far away it is from a facility, and let a customer know an ETA when they might expect a fix. It’s taken the average time from two hours down to one-and-a-half minutes, the company said at its release.

Nafshi also underscored the massive change in network usage that happened when the pandemic hit: “We had to turn our smart network into a brilliant network,” he said. “No one has ever foreseen a scenario where we all go home, jump on video communication, and don’t leave for two years.”

In response, in 2020, the company rolled out Octave, a cloud-based AI platform that checks for technical issues on tens of millions of modems across Comcast’s network. It’s an example of Distributed Access Architecture, which removes inefficiencies and enables the company to boost its hardware to speeds akin to software.

When Xfinity expands its service to a new area, there’s a lot of physical infrastructure involved. The company has to get their fibers and nodes set up in new neighborhoods, and sometimes establish a new headend. But hardware evolution will bring some sustainability and power savings, Nafshi said, as they’re able to do more with less material. It explains why the newer side of the headend I toured was a lot more sparse, and why headends in the near future will likely be smaller.

Physical infrastructure and fibers. (Photo by Paige Gross)

In the 15-minute drive home from my tour last Thursday, I stare up at the telephone lines for much of my journey. I now know that I can see and hear and feel the internet, and that the digital and physical infrastructure that makes up one of the country’s largest ISPs is a colorful, massive, constantly changing system.