Hey, DE governments: this is why local devs are going to start scraping your websites

Making data open can mean different things to different people. Mark Headd says Delaware needs to expect more.

Shaved bald, with thick rimmed glasses and a hooded sweatshirt, he is a recognizable face to many who follow national open data and open government conversations. Headd is the fiery former embattled Chief Data Officer of the City of Philadelphia, past executive with Code for America and self-identified civil servant. He spent 10 years living in Wilmington while serving an array of Delaware state government jobs beginning in 1999. Now he lives in upstate New York, while traveling frequently to conferences and tech events to evangelize for Accela, a Silicon Valley government IT services company.
He was back in Delaware for the first time in several years last week to speak at our Delaware Innovation Week Dev Talks, which brought nearly 70 technologists to the coIN Loft in Wilmington.
“So I wanted to check in on how things were going,” Headd said of an open data movement that has really only started in earnest here in the last six months. “I self-identify as a civil servant. There are people in all governments working really hard to serve citizens in lots of ways. But we need better here.”
To show that, Headd took a look at the State of Delaware’s Food Establishment Inspection Records. Though some might say that a government site that lists such records could constitute open data, Headd said that it is practically unusable. It is clunky, hard to understand and lacks any useful structure — like geolocation to map the offenders or “fuzzy search” by restaurant name. There is only a first-letter alphabetized order.
“This bad version isn’t helping anyone,” said Headd. The state doesn’t get help enforcing records by putting pressure on restaurants with poor records, restaurant goers don’t get the benefit of being able to easily identify problem locales and the restaurants that are following state law don’t get the benefit of being able to be discovered for a strong track record.
So in a 20-minute case study, he showed how quickly something better could be created. He scraped the government site (“throwaway work” that all experienced developers hate doing, he said) and used Google’s OpenRefine to add longitude and latitude data to a spreadsheet that had the inspected locations. Then Headd was able to use geolocation service CartoDB to create a simple map of the locations.

“A lot of people in government identify as the ones who build the apps. Open data is pushing them to see they are the enablers of apps,” said Headd. It’s a concept that Wilmington Office of Economic Development Director Jeff Flynn is trying to build a constituency around.
“We need this,” said Flynn, who spoke to Headd after the presentation but was not endorsing Headd’s approach.
The “this” is something that’s happening across the country — tech communities pushing their governments to supply the marketplace with data and information in formats that can be easily manipulated in new and creative ways (think .CSV files and APIs, not proprietary websites or text documents).
“I am not just calling out the state of Delaware,” said Headd. “Often, there is still not an awareness that this is good for government.”
So what could make the Food Inspections website better?
“The data is clearly already in a database,” said Headd. “Render it as a .CSV and allow us to download the whole file or, even better, put an API in front of it so an application we build using that data stays accurate.”
Look at Brixsy in Philadelphia as an example of a solution coming into the market because of shared data, Headd said. What can developers do to push government to modernize their data releases?
“Go scrape some data from the state,” said Headd, of using automated tools to pull down information. “There isn’t a more effective way to push open data forward than scrape it and share it. Then tell government: ‘I’ll stop scraping it when you start sharing it.'”