NYU is building a real-life holodeck

The prospect of sci-fi turning into real life is pretty familiar these days. In the past couple of years, several projects have claimed to point the way toward a real-life version of the holodeck from Star Trek: basically, an immersive virtual-reality experience in which participants can freely interact with their surroundings. For now, it’s still more hype than substance. But NYU got a $2.9 million grant from the National Science Foundation to make it happen.
When Winslow Burleson, director of the NYU-X Lab at the College of Nursing, announced the grant at the NYC Media Lab Summit last month, quite a few people in the audience, including this reporter, were incredulous. (“I’m still trying to wrap my head around that holodeck presentation,” one demo participant said, hours later.) So we tracked Burleson down for more details. Does the project actually intend to bring Star Trek to life? And besides the obvious cool factor, what will be the eventual payoff?

We very much believe in this as a large-scale open innovation endeavor.

The project involves several schools and departments within NYU, including the Media and Game Network (MAGNET), which is based at NYU Tandon in Brooklyn. Brooklyn-affiliated faculty on the project include computer science professor Ken Perlin, who is a member of MAGNET, and composer and Tandon associate professor R. Luke DuBois. In addition, NYU is partnering with 18 other universities, including MIT and UC Berkeley, as well as companies such as IBM and Sennheiser.
The National Science Foundation grant covers a five-year period, with the first two years designated for bringing the technology to initial maturity: in other words, one that other universities could investigate. Burleson said he expects that an initial prototype will be built by the end of 2017.

The NYU Holodeck isn’t a novel technology, per se, but a combination of many technologies under one platform: virtual and augmented reality, wearable devices, brain-computer interaction, 3D printing, robots, advanced sound systems and game controllers, all underpinned by IBM’s Watson. What’s novel, Burleson said, is how these technologies will work together. Interacting with a virtual reality display in the NYU Holodeck, for instance, could summon a social robot to perform a task.
“It’s like using the Force to bring it to you,” he said.
(Yes, we know that’s a Star Wars reference. Apologies for the mixed sci-fi metaphors.)
Burleson has previously worked on several of the technologies being incorporated into the NYU Holodeck. As a doctoral researcher at the MIT Media Lab, he developed a “multimodal affective learning companion” — in other words, a software-generated character that responds to users’ emotions in order to help them learn new skills. In his current role at the College of Nursing, he has worked on virtual-reality simulators that enable students to train for medical procedures.
With this project, the university hopes to expand upon its educational and research possibilities. One key aspect of the NYU Holodeck is that its platform will be distributed, so it won’t be confined to NYU’s campus. Instead, its partner institutions will be running experiments at the same time, and their results will be captured and shared on the platform. As a result, Burleson said, the Holodeck will become “smarter” in a sense, by having more simulations and virtual scenarios to draw upon as it undergoes testing. And, of course, universities and other institutions will be able to collaborate more efficiently, by using the same shared resource.
“We very much believe in this as a large-scale open innovation endeavor,” he said. “We hope it will pave the way for many other Holodeck nodes to be built.”
So how much does this line up with the Star Trek holodeck that inspired it — or, at least, its name? Note that there’s no mention of holographs here: one key feature of the original, which other supposed “real-life holodecks” have attempted to replicate. Burleson acknowledged as much.
“The current thinking is that’s further out than our project,” he said.
But, he added, with the distributed model, the NYU Holodeck may be even more sophisticated than the sci-fi version, at least in one respect.
Overall, he said, “We think that we’re spiritually in the ball park.”