Why would anyone build an AI that doesn’t favor them?

In 1983, the president of American Airlines went before Congress to defend a computer program his company had built for the entire airline industry, and admitted it was rigged.

Competitors had been complaining for years that SABRE, the reservation system every travel agent used but American owned, buried their flights. Continental Airlines claimed it was losing up to $55 million a day. Rather than deny it, Robert L. Crandall testified that favoring American’s flights “is the competitive raison d’etre for having created the system in the first place.” 

He seemed genuinely puzzled that anyone expected otherwise. Researchers later named the objection after him. Crandall’s Complaint: Why build an expensive system if you can’t tilt it your way? 

Which leads to the uncomfortable follow-up: Absent some outside check, shouldn’t we assume every system is designed with someone else’s goals in mind, over yours?

That question is a good starting point for your region’s AI conversation. American’s competitors compiled evidence, the Civil Aeronautics Board investigated, and the resulting rule became the first regulation in history to explicitly reference an algorithm.

The method is older than the machines

That story opens “Auditing AI,” a new MIT Press book written by 11 authors working as the Marquand House Collective. One of them, Philadelphia-based journalist Ellery Roberts Biddle, presented the book back in June at a monthly AI coffee chat we host in the Technical.ly newsroom. This month she hosted a book talk at Penn Bookstore, moderated by Technial.ly national and market editor Katie Malone.

I use a broad definition of AI itself: essentially any computer program that mimics human cognition. The coauthors define an AI audit in an equally broad way: a rigorous and systematic evaluation of the core claims, expectations or standards pertaining to an AI system.

The auditing method didn’t come from computer science, Biddle said. 

In the mid-20th century, tenants and homebuyers organized fair housing committees and sent two people — one Black, one white — to ask the same landlord about the same apartment at roughly the same time, then compared what each was told. They called each series an “audit study,” and they designed them, in one memorable line, “to be so simple that even a lawyer or judge can understand them.”

Next, paper replaced people. A landmark 2004 study titled “Are Emily and Greg more employable than Lakisha and Jamal?” mailed nearly 5,000 fictional resumes to more than 1,300 job postings and found that white-sounding names drew 50% more interview callbacks. Then software replaced the paper. The underlying method never changed: Vary the input, watch the output, look for patterns rather than single decisions.

Auditing AI co-author Danaé Metaxa’s team ran that exact play on ChatGPT. Prompting GPT-3.5 to generate 320 resumes for fictional applicants, they found that women’s names produced resumes with fewer years of experience on average, and that Hispanic names were far more likely to yield a candidate described as a green card holder — which only about 3% of the US population holds.

How to tell a real audit from a rubber stamp

The book’s most immediately useful chapter isn’t about running audits. It’s about reading them, and that’s the skill most local employers, agencies and boards actually need.

New York City in November 2021 passed Local Law 144, requiring anyone using a hiring algorithm to have it audited for racial and gender bias and to publish the result. The compliance reports are public, and two of them demonstrate a risk of the AI age: when an avalanche of words hides a lack of substance. 

To comply with the law, Paramount commissioned a 21-page report on its use of the vendor Pymetrics. It looks official. But the testing was conducted by Pymetrics itself, covering just one calendar year, and few demographics were collected. A third of applicants never reported race or gender. The passing grade may be accurate. The report just doesn’t let you check.

In contrast, NBCUniversal’s audit of its vendor SmartRecruiters fits on one page. Of the total candidate pool, just 2.5% skipped demographic data, and the hiring effects of the applicant sorting tools were reported for more identity and background categories than required. 

The shorter report was the better one.

RELATED NEWS

Comcast’s nationwide overhaul will cut 634 jobs at its Philly headquarters

Healthcare workers are being asked to use AI. Here’s a Baltimore-area effort to teach you how.

New threats, kind of like the old threats: Why AI makes speed the ultimate cybersecurity defense

Researchers use 0.80 as a rule of thumb for the effects of a hiring algorithm: if one group’s selection rate falls below four-fifths of the top group’s, that’s evidence of adverse impact. It’s also, as the “Auditing AI” authors note, merely a guideline from decades of still-ongoing employment discrimination work — imported into AI auditing largely because it was the only threshold anyone had already agreed on.

Forecasting versus measuring

The audit tradition treats AI as what it plainly is, Biddle said: a product somebody built, for somebody’s benefit, subject to the same question we ask of eggs, bridges and mortgage lenders. Does it do what the seller said it does?

Technology changes tasks first, then firms, and only later, under external pressure, entire industries. Audits can be that external pressure. And the task stage is the last point where a local employer, hospital or county still has leverage over the terms.

An audit tells you how a system behaved when tested, not why. Audits are not explanations. And they compel nothing: Researchers documented gender stereotypes in Google Image Search in 2015, Pew replicated it in 2018, others found them persisting in 2021. Google never formally responded. Then its AI Gemini overcorrected into generating Black soldiers in Nazi uniforms.

The best audits are boring. An egg inspector who finds every carton is Grade A has still done the work we’re paying for.

“Auditing AI” is free to read under a Creative Commons license; the authors would rather you use this than buy it.

Crandall’s real offense wasn’t rigging SABRE. It was saying so on the record, which is why it got regulated. Today’s version is quieter, better lawyered and running in your HR department. Assume the thumb is on the scale. The only open question is whether anybody near you can weigh it.