Hey, DE governments: this is why local devs are going to start scraping your websites - Technical.ly Delaware

Dev

Nov. 30, 2015 7:05 am

Hey, DE governments: this is why local devs are going to start scraping your websites

Former City of Philadelphia Chief Data Officer Mark Headd told a room full of developers that the best way to make their governments adapt more quickly to modern open data trends is to challenge them with action.
Former City of Philadelphia Chief Data Officer Mark Headd speaking at Delaware Innovation Week.

Former City of Philadelphia Chief Data Officer Mark Headd speaking at Delaware Innovation Week.

(Photo by Christopher Wink)

Making data open can mean different things to different people. Mark Headd says Delaware needs to expect more.

Shaved bald, with thick rimmed glasses and a hooded sweatshirt, he is a recognizable face to many who follow national open data and open government conversations. Headd is the fiery former embattled Chief Data Officer of the City of Philadelphia, past executive with Code for America and self-identified civil servant. He spent 10 years living in Wilmington while serving an array of Delaware state government jobs beginning in 1999. Now he lives in upstate New York, while traveling frequently to conferences and tech events to evangelize for Accela, a Silicon Valley government IT services company.

He was back in Delaware for the first time in several years last week to speak at our Delaware Innovation Week Dev Talks, which brought nearly 70 technologists to the coIN Loft in Wilmington.

“So I wanted to check in on how things were going,” Headd said of an open data movement that has really only started in earnest here in the last six months. “I self-identify as a civil servant. There are people in all governments working really hard to serve citizens in lots of ways. But we need better here.”

To show that, Headd took a look at the State of Delaware’s Food Establishment Inspection Records. Though some might say that a government site that lists such records could constitute open data, Headd said that it is practically unusable. It is clunky, hard to understand and lacks any useful structure — like geolocation to map the offenders or “fuzzy search” by restaurant name. There is only a first-letter alphabetized order.

“This bad version isn’t helping anyone,” said Headd. The state doesn’t get help enforcing records by putting pressure on restaurants with poor records, restaurant goers don’t get the benefit of being able to easily identify problem locales and the restaurants that are following state law don’t get the benefit of being able to be discovered for a strong track record.

Advertisement

So in a 20-minute case study, he showed how quickly something better could be created. He scraped the government site (“throwaway work” that all experienced developers hate doing, he said) and used Google’s OpenRefine to add longitude and latitude data to a spreadsheet that had the inspected locations. Then Headd was able to use geolocation service CartoDB to create a simple map of the locations.

Here's the simple map Mark Headd created of inspected restaurants according to state data.

Here’s the simple map Mark Headd created of inspected restaurants according to state data.

“A lot of people in government identify as the ones who build the apps. Open data is pushing them to see they are the enablers of apps,” said Headd. It’s a concept that Wilmington Office of Economic Development Director Jeff Flynn is trying to build a constituency around.

“We need this,” said Flynn, who spoke to Headd after the presentation but was not endorsing Headd’s approach.

The “this” is something that’s happening across the country — tech communities pushing their governments to supply the marketplace with data and information in formats that can be easily manipulated in new and creative ways (think .CSV files and APIs, not proprietary websites or text documents).

“I am not just calling out the state of Delaware,” said Headd. “Often, there is still not an awareness that this is good for government.”

So what could make the Food Inspections website better?

“The data is clearly already in a database,” said Headd. “Render it as a .CSV and allow us to download the whole file or, even better, put an API in front of it so an application we build using that data stays accurate.”

Look at Brixsy in Philadelphia as an example of a solution coming into the market because of shared data, Headd said. What can developers do to push government to modernize their data releases?

“Go scrape some data from the state,” said Headd, of using automated tools to pull down information. “There isn’t a more effective way to push open data forward than scrape it and share it. Then tell government: ‘I’ll stop scraping it when you start sharing it.'”

Want more richly researched and deeply sourced community reporting?  Become a Technical.ly member for $12 per month -30-
BECOME A CONTRIBUTOR TO THE JOURNALISM FUND TO COMMENT
Already a contributor? Sign in here
Connect with companies from the Technical.ly community
New call-to-action

Advertisement

Exploring Wilmington’s OpenGov

Do you understand the Freedom of Information Act?

1on1: Devs Steph Staub and Briana West on coding and career trajectories

SPONSORED

Delaware

Verizon is looking for the brightest ideas on how to use its 5G technology

Philadelphia, PA - Center City

Odessa

Business Architect

Apply Now

Offsite

Philadelphia Home

Social Media Catalyst

Apply Now

Baltimore, Brooklyn, D.C., Delaware, Philadelphia

Vistar Media

Software Engineer

Apply Now

CompStat helped lower Wilmington’s crime rate — but what the heck is it, exactly?

An open data silver lining for Wilmington

Fintech, mergers and HQ2: Delaware’s biggest tech stories of the last 10 years

SPONSORED

Delaware

Escape the August heat with cool AI tech

Malvern, PA

Vanguard

Senior Front End Engineer – CX Journey Lab

Apply Now

Wilmington

Marlette Funding

Production Support Analyst

Apply Now

Wilmington,DE

CSC

Associate Marketing Specialist

Apply Now

Sign-up for daily news updates from Technical.ly Delaware

Do NOT follow this link or you will be banned from the site!