One study says DC’s gender pay gap is big, the other says small. What’s the truth?

It’s a classic tale, really. A tale of two surveys.

One reports that D.C.’s tech gender pay gap is virtually nonexistent, then another puts our fair city near the top of a list for having the biggest tech gender pay gaps. Which is correct? Well, it’s not easy to tell at first glance, so Technical.ly decided to take a deeper look.
Here’s the full story, from the top:
In May, an analysis from SmartAsset, a personal finance website, declared D.C. at the top of its list of “best cities for women in tech.”
In fact, out of a possible 100, the District scored a 91.98 on SmartAsset’s index of friendliness toward women, an index composed of numbers on the gender pay gap, income after housing costs, percentage of tech jobs filled by women and three-year tech employment growth.
D.C., of course, scored well in each of these categories. But notable among them was the tech gender pay gap — SmartAsset found that women in #dctech “earn roughly the same average income as men.”
The #dctech community seemed, in general, pleased to hear this news. “D.C. blew away the competition,” DC Inno wrote. This Facebook post got some love:

But there was also quiet skepticism — how could this be the case? What makes D.C. so different in terms of pay parity?
Then, in early June, a new study on the tech gender pay gap came to town and this time the outlook was far less rosy. A survey by Comparably, “a platform to provide anonymous and comprehensive data on compensation,” reported that women in #dctech make 51 percent less than their male counterparts.
Wow.
This was the juncture at which Technical.ly got curious about the statistics. Some difference in the numbers can be expected, but the difference between “roughly the same average income” and 51 percent less? That’s big. So we called up SmartAsset and Comparably to ask some questions about their respective methodologies.

###

The data for the SmartAsset report comes from the Census or, specifically, from the American Community Survey (ACS). The ACS is an “ongoing” survey by the Census Bureau, aimed at providing information about American’s health, wealth, education and more on a more regular basis (every year) than the traditional Census. The SmartAsset report released in May is based on ACS data from 2014.
AJ Smith, SmartAsset’s VP of content, told Technical.ly that her team started with a list of the 58 largest cities in the U.S. (so that each city would have a significant number of tech-focused employees) and then pulled the numbers on the gender pay gap, income after housing costs, percentage of tech jobs filled by women and three-year tech employment growth for each.
SmartAsset defined “tech” as the “computer and mathematical occupations” line item on the ACS. For D.C., this occupation, which falls under the larger category of “computer, engineering and science occupations,” boasts nearly 17,000 respondents.
Indeed, even a quick look at the data reveals only a small differential between median salary for men in “computer and mathematical occupations” ($78,372) and women in the same ($77,860). It’s worth noting, though, that the pay gap in the broader “computer, engineering and science occupations” category is much larger.
Also worth noting is a footnote on the SmartAsset study that mentions how the ACS used to look at computer occupations as a separate category before combining it with mathematics. “There is also greater gender equality in mathematical occupations, which means the data used in this year’s analysis paints a rosier picture of the tech industry’s gender dynamics,” the author of the report writes.
So the data might not be perfect, but then again what data is. When you’re relying on individuals to self-report information there will always be some mistakes — intentional or not. Hence statistical tools like the margin of error.
Regardless, the SmartAsset study does have some strengths. Chief among them is the data itself — it is legally mandatory for selected households to answer the ACS survey. As an attempt to eliminate selection bias this is meaningful, essentially because it means respondents aren’t just people who sought out an opportunity to make their position known.
Also, as mentioned above, the SmartAsset study worked with a decently large sample size (nearly 17,000 individuals in D.C. alone) and a large sample size, as any statistician will tell you, is key to trustworthy results. Census data might not be perfect, but it is one of the best sources for making a general statement about an entire population (e.g., women in tech) precisely because of large sample sizes.

###

Next, the Comparably survey.
To understand where the Comparably numbers come from, you’ve got to understand Comparably. The site, launched just this past March, is dedicated to transparency in pay and company culture. It’s similar to a company like Glassdoor, for example. For the moment, though, Comparably focuses solely on transparency in the tech industry.
Comparably members sign up for free, but must enter a bunch of data on their location, job title and salary. Comparably gathers this anonymous data to show members their respective place (salary-wise) in their industry with regards to gender, race, company size and more. Recently, though, Comparably started sharing this data more publicly.
This is where the report on the gender pay gap in tech came from. Comparably has over 10,000 users now, so, according to Tal Siach, the company decided it was time to start making data visualizations based on their numbers.
Comparably divided those 10,000 users by gender. Then the company used members self-identification to create other clusters such as race, age, education and location to spot differences between the genders. Lo and behold, Female Comparably members in D.C. make a median salary plus bonus that’s 51 percent less than male Comparably members in D.C.
The phrasing above is super important because here’s the crux of the issue with the Comparably survey — sure, we can believe that the results accurately reflect the position of Comparably’s members. But do they accurately reflect anything more than that?
One clue in answering this question is sample size. Sure Comparably based their numbers on the responses 10,000 users, but how many of those users are in D.C.? Siach did say that the cities mentioned in the report are cities where Comparably has “the most users,” but declined to offer any specific numbers on the District.
Also worth thinking about — the kind of data generated by a mandatory, government-run survey (the ACS) versus a private, interest-based website that users have to go (at least somewhat) out of their way to sign up for.

###

So here’s the big, lingering question: What’s the truth? Does D.C. have a big tech gender pay gap, or a small one?

Based on the two surveys at hand, SmartAsset comes out on top for employing somewhat smarter statistical practices. However, as Siach mentioned in conversation, a company like Comparably could be an interesting source of data, especially over time and with more users.

The main lesson, though, the moral of this tale of two surveys, seems to boil down to a single phrase: caveat emptor. The SmartAsset report has a decently thorough section on methodology, but not all reports do. And when they don’t, it’s up to the viewer — journalist, data scientist or curious citizen — to ask questions about where the data comes from, what the survey looked like, what the sample size was and more.

So here’s your friendly reminder: statistics are malleable. Ask questions.