Hey, DE governments: this is why local devs are going to start scraping your websites - Technical.ly Delaware

Dev

Nov. 30, 2015 7:05 am

Hey, DE governments: this is why local devs are going to start scraping your websites

Former City of Philadelphia Chief Data Officer Mark Headd told a room full of developers that the best way to make their governments adapt more quickly to modern open data trends is to challenge them with action.

Former City of Philadelphia Chief Data Officer Mark Headd speaking at Delaware Innovation Week.

(Photo by Christopher Wink)

Making data open can mean different things to different people. Mark Headd says Delaware needs to expect more.

Shaved bald, with thick rimmed glasses and a hooded sweatshirt, he is a recognizable face to many who follow national open data and open government conversations. Headd is the fiery former embattled Chief Data Officer of the City of Philadelphia, past executive with Code for America and self-identified civil servant. He spent 10 years living in Wilmington while serving an array of Delaware state government jobs beginning in 1999. Now he lives in upstate New York, while traveling frequently to conferences and tech events to evangelize for Accela, a Silicon Valley government IT services company.

He was back in Delaware for the first time in several years last week to speak at our Delaware Innovation Week Dev Talks, which brought nearly 70 technologists to the coIN Loft in Wilmington.

“So I wanted to check in on how things were going,” Headd said of an open data movement that has really only started in earnest here in the last six months. “I self-identify as a civil servant. There are people in all governments working really hard to serve citizens in lots of ways. But we need better here.”

To show that, Headd took a look at the State of Delaware’s Food Establishment Inspection Records. Though some might say that a government site that lists such records could constitute open data, Headd said that it is practically unusable. It is clunky, hard to understand and lacks any useful structure — like geolocation to map the offenders or “fuzzy search” by restaurant name. There is only a first-letter alphabetized order.

“This bad version isn’t helping anyone,” said Headd. The state doesn’t get help enforcing records by putting pressure on restaurants with poor records, restaurant goers don’t get the benefit of being able to easily identify problem locales and the restaurants that are following state law don’t get the benefit of being able to be discovered for a strong track record.

Advertisement

So in a 20-minute case study, he showed how quickly something better could be created. He scraped the government site (“throwaway work” that all experienced developers hate doing, he said) and used Google’s OpenRefine to add longitude and latitude data to a spreadsheet that had the inspected locations. Then Headd was able to use geolocation service CartoDB to create a simple map of the locations.

Here's the simple map Mark Headd created of inspected restaurants according to state data.

Here’s the simple map Mark Headd created of inspected restaurants according to state data.

“A lot of people in government identify as the ones who build the apps. Open data is pushing them to see they are the enablers of apps,” said Headd. It’s a concept that Wilmington Office of Economic Development Director Jeff Flynn is trying to build a constituency around.

“We need this,” said Flynn, who spoke to Headd after the presentation but was not endorsing Headd’s approach.

The “this” is something that’s happening across the country — tech communities pushing their governments to supply the marketplace with data and information in formats that can be easily manipulated in new and creative ways (think .CSV files and APIs, not proprietary websites or text documents).

“I am not just calling out the state of Delaware,” said Headd. “Often, there is still not an awareness that this is good for government.”

So what could make the Food Inspections website better?

“The data is clearly already in a database,” said Headd. “Render it as a .CSV and allow us to download the whole file or, even better, put an API in front of it so an application we build using that data stays accurate.”

Look at Brixsy in Philadelphia as an example of a solution coming into the market because of shared data, Headd said. What can developers do to push government to modernize their data releases?

“Go scrape some data from the state,” said Headd, of using automated tools to pull down information. “There isn’t a more effective way to push open data forward than scrape it and share it. Then tell government: ‘I’ll stop scraping it when you start sharing it.'”

Want more richly researched and deeply sourced community reporting?  Become a Technical.ly member for $12 per month -30-
JOIN THE COMMUNITY, BECOME A MEMBER
Already a member? Sign in here

Advertisement

1on1: Devs Steph Staub and Briana West on coding and career trajectories

Delaware tech jobs by the numbers: How small is the pond?

CompStat helped lower Wilmington’s crime rate — but what the heck is it, exactly?

SPONSORED

Delaware

Hear from the privacy pros at Security by the Schuylkill

Philadelphia, PA

Orthly

Full-Stack Javascript Engineer

Apply Now
Philadelphia

Technically Media

Technical.ly Editorial Intern (Summer 2019)

Apply Now
Philadelphia

KickUp

Software Engineer

Apply Now

Fintech, mergers and HQ2: Delaware’s biggest tech stories of the last 10 years

Curious about open data? Join our civic tech Slack chat on Friday

realLIST: These are the top 10 startups in Newark, Del.

SPONSORED

Delaware

Learn to lead digital transformation at Phorum 2019

Malvern, PA

Vanguard

Senior Front End Engineer – CX Journey Lab

Apply Now
Baltimore, MD 21201

14 West

Senior Software Engineer (Java + Apache Spark)

Apply Now

Sign-up for daily news updates from Technical.ly Delaware

Do NOT follow this link or you will be banned from the site!