Digital revolution makes data consumers “hungry and aimless”: Judea Pearl

When it comes to the explosion of new data sources, we’re all “hungry and aimless.” That’s because, for those of us caught up in the power of statistics and analytics, we’re asking the wrong questions, says UCLA professor Judea Pearl.

Consumers of big data sets are interested in “interventionist questions,” such as “what will happen if we raise taxes?” or “what will happen if implement a congestion charge for traffic?”

But there is no way to really test or compare something like that because the variables can’t be held constant in any useful, consistent way, he said.

Professor Pearl gave his remarks as part of the 15th Annual Lynford Lecture at the Polytchnic Institute of New York Universiy (NYU-Poly). He was the first Lynford honoree to have actually graduated from NYU-Poly. Professor Pearl earned his doctorate at NYU-Poly in 1965. He’s been honored for his work on causation.

Some interesting points Prof. Pearl made during his talk:

• Humankind didn’t get where it is today by waiting for evolution. It got to where it is today through an exercise of the mind.
• Counter-factuals (that is, asking what would have happened if a different intervention had been pursued) are the building blocks of understanding. He called the biblical Abraham the first scientist, because he asked God how many righteous men he would have spared Soddom and Gomorrah for. He was exploring a counter-factual. “He was after the general rule of collective punishment,” the professor said.
• “The Physics you learned in high school is about counterfactuals.” In other words, you learn formulas that yield different outputs if different inputs are put in.
• He explained why a robot in a kitchen looking for the cause of a spilled glass of milk is like an economist in the economy.
• The trouble with running experiments where some ‘X’ is held constant is this: “In reality, Y variable listens to what X variable is doing.”

The question and answer period ranged from advice on changing the structure of experiments in order to generalize finding, to the brains of octopi to the Singularity. On the Singularity, Professor Pearl had this to say:

“I’m not a futurist,” he said. “This is a world I cannot put myself into [but] I am sure we will get to a point where robots will act as if they have free will.”