Philadelphia’s growing AI programs face scrutiny over governance and public input

Technical.ly is one of 30 news organizations powering the Philadelphia Journalism Collaborative. We do solutions reporting on things that affect daily life in our city where the problem and symptoms are obvious, but what’s driving them isn’t. Follow at @PHLJournoCollab.

Philadelphia is moving ahead with using AI in city services, but some residents worry the rollout has happened largely out of public view.

City Council held the city’s first major hearing on AI regulation last October. Representatives from Mayor Cherelle Parker’s administration outlined a high-level strategy, but stopped short of offering specifics about how the technology is actually being used.

For community advocates, including at Philly Tech Justice, those gaps have been impossible to ignore.

“Our concerns around the city’s use of technology is that it’s a black box,” Devren Washington, organizing director, told Jordana Rubenstein on PhillyCAM’s new monthly show, Philly Unpacked.

In October, city officials said they would create a framework guiding how public sector employees should use AI by late winter 2025 or early spring 2026, though it’s unclear how much progress they’ve made. 

Once that’s in place, leaders are planning more training opportunities and a cross-departmental AI governance committee to review and update future AI policies, according to Kristin Bray, chief legal counsel to the mayor and director of Philly Stat 360.

Putting these policies into place will hopefully teach city employees not to put government data into online tools like ChatGPT and Gemini, city Chief Information Officer Melissa Scott said at the hearing. 

Despite a request from at-large Councilmember Rue Landau, who leads Council’s Committee on Technology and Information Services, for a comprehensive inventory of all AI tools used by the city, the Parker administration did not provide one, Technical.ly reporter Sarah Huffman said on the show. 

“Both council members and members of the public had questions,” Huffman said, adding that many are still unanswered. For example, whether the city plans to get expert advice on AI policy or integration, how stakeholders will get public feedback and the timeline of rolling out more AI initiatives.

Another repeated concern about AI rollout is how data is being collected, used internally and shared with third parties.

“At the heart of it, this is about who controls our data,” said Washington, of Philly Tech Justice. “The reason why we are fighting for accountability with the city is because … these companies are getting more control over the data that we are producing, and it’s immensely valuable.”

For city AI, residents want to ensure ‘humans in the process’

For years preceding Parker, city agencies have experimented with the use of AI in public spaces. 

Through SmartCityPHL, the Streets Department worked with infrastructure inspection company GoodRoads in 2021 to deploy AI tools identifying potholes for repair. Comcast, Juganu and US Ignite all backed the SmartBlockPHL initiative in 2022, which tested smart sensors on streetlights capable of collecting data about people and objects in public spaces, plus environmental conditions. A $4.9 million state grant in 2022 funded SEPTA’s pilot with local startup ZeroEyes to install AI-powered gun-detection software across parts of the transit system.

Under Parker, initiatives have continued, like SEPTA and the Philadelphia Parking Authority adopting AI to issue tickets to cars in the bus lane. 

When nominated as the Democratic candidate for mayor in 2023, Parker shared some thoughts on how she would continue integrating tech in the city. She emphasized transparency with residents while prioritizing cybersecurity. 

When using technology to address crime and violence, Parker said in 2023 she would introduce the comprehensive neighborhood safety and community policing plan, which addresses prevention, intervention and enforcement.

Law enforcement use of AI was a specific concern from residents at City Council’s AI hearing in October. Testimonies from city officials did not address how AI is being used by the police department or if the administration is concerned about the technology being used by Immigration and Customs Enforcement in the city. 

Scott, the city CIO, said she was “unaware” of any instances where the police in Philadelphia deployed AI against residents, according to Axios reporting.

For Washington, the concerns extend beyond transparency to questions of budget priorities and equity, too. Data used to train certain algorithms can hold biases that lead to prejudiced results. 

Washington pointed to the city’s property assessment algorithm as an example. In 2022, the Philadelphia Inquirer reported that the city distributed the tax burden unequally, with systemic inaccuracies in neighborhoods with majority Black residents, and those with lower median household incomes.

Bray, chief legal counsel to the mayor, said at the October hearing that human judgment, public trust and accountability would be top of mind for officials.

“A large theme from all of that testimony was just keeping humans in the process,” Huffman said on the PhillyCAM show. “Make sure that humans are auditing the system and that everything is working properly.”