Gaming / VR

We go behind the scenes of this dazzling VR performance of Hamlet

A team from NYU's MAGNET is combining virtual reality, motion capture and livestreaming to put a new twist on theater. Could this be the future of performance?

Christmas Sweater World, we hardly knew ye. (Courtesy photo)

Imagine standing within Elsinore Castle in Denmark behind a 30-foot-tall ghost as he reveals to his son the identity of his killer.
Shakespeare buffs will recognize the setting, from Act I, Scene 5 of Hamlet. A project at NYU’s Media and Games Network (MAGNET) aims to takes us there through a live production of the scene — translated into a virtual-reality setting. On Sunday, I visited MAGNET at 2 MetroTech, where two actors — Zachary Koval as Hamlet and Roger Casey as the ghost of his father — portrayed the scene in a dark room while bedecked in black body suits with small, round markers attached.
Those suits Koval and Casey wore served to track their positions and movements, which were projected onto a screen behind them. And to the left of that projection was the virtual-reality world onto which Koval and Casey’s characters had been transposed. There stood Hamlet, cowering under the Darth Vader-like presence of his dead father, amidst a scape of rocky cliffs and the Øresund. Through an HTC Vive headset, I got the chance to witness this VR production, entitled To Be With Hamlet, live.
To Be With Hamlet is led by Javier Molina, an engineer and NYU Tandon faculty member who specializes in VR and motion capture. Among the past projects he’s worked on is The Return, an art installation produced last year at the Metropolitan Museum of Art, which illustrated how the famed statue of Adam by Renaissance sculptor Tullio Lombardo was restored after being shattered more than a decade ago. Directing the theatrical action is David Gochfeld, a graduate of NYU’s interactive telecommunications program with a deep background in theater. One of his past projects is FutureMate, an apocalyptic-themed mashup of live theatre and online dating. The team also includes collaborators from many different departments within NYU, including the Game Center and Steinhardt.

‘A new art form’

So exactly how do Molina, Gochfeld and their collaborators transform Hamlet from a stage play into a live-action, virtual-reality world? It involves a combination of several technologies.
The virtual world of Elsinore Castle was created using Unreal, a popular software engine for developing video games. Unreal was also used to create avatars for both of the characters portrayed in To Be With Hamlet. For Hamlet, they scanned 3D body and facial images of Koval to develop a convincing doppelgänger — perhaps bringing a new meaning to the play’s line “God hath given you one face, and you make yourselves another.”
For Casey’s ghost, the team’s designers took some creative license. Though the character’s armor hews to Shakespeare’s description, the ghost itself is portrayed in To Be With Hamlet as a giant, which isn’t specified in the text. The avatar, Gochfeld said, offers a visual cue to convey the outsized impact of the character.
“In VR, you can play with scale. You can make a character really large or really small,” he said. “In this scene, [the ghost] is turning Hamlet’s whole life on its head.”

Zachary Koval and Roger Casey prepare for a live stream of To Be With Hamlet.

Zachary Koval and Roger Casey prepare for a live stream of To Be With Hamlet. (Photo by April Joyner)

The next step is getting the flesh-and-blood actors into that meticulously constructed world. That’s where motion capture technology — including those odd body suits — comes in. The ball-shaped markers that stick out from the suit track the movements of various parts of the actors’ bodies. Those movements are relayed to a software program, which in turn communicates them to the virtual world platform. So when Koval and Casey move, their avatars do, too.
(For more on the technical side of the production, check out this interview with Molina and developer Owen Bell at There Is Only R.)
The thing is, motion capture doesn’t deal so well with subtlety. So it takes some changes in the actor’s stage presence: basically, making their movements bigger than usual. Koval and Casey told Technical.ly they relish the challenge. Casey previously worked with Molina on The Return, while To Be With Hamlet marks Koval’s first time working with VR. Virtual reality, Koval said, involves a mashup of different acting disciplines.
“It’s kind of a new art form,” he said. “There’s a bit of puppetry, a bit of mask work, kind of walking this fine line between film and stage theater performance, because the audience can be anywhere.”
That said, Casey pointed out, there wasn’t a substantial difference from more traditional media in their approach to character development.
“It involves the same internal work,” he said. “Now we just have extra tools.”

Getting into Hamlet’s head

There has been at least one other project exploring the world of Shakespeare through motion capture and VR technologies. Developers at UC Davis created a VR game called Play the Knave that allows participants to reenact scenes from a selection of Shakespearean plays. In that game, participants’ movements are captured through a Microsoft Kinect camera.
But the trickiest piece of To Be With Hamlet, technically speaking, isn’t the virtual-reality or motion-capture components. It’s the live-action element, which Molina and his team were troubleshooting when I arrived at MAGNET. To broadcast the VR performance, they use a proprietary software platform called M3diate, developed by a group of NYU-affiliated technologists in Amsterdam as well as the university’s campuses in Abu Dhabi and Shanghai. On Sunday, the team was preparing to perform To Be With Hamlet live for VR Days, a festival in Amsterdam. Earlier that morning, M3diate wasn’t fully cooperating, but by the early afternoon, it was up and running. (Here’s a video of the actors giving their live streamed performance.)
Once all the kinks are out, ideally, viewers will be able to access performances by downloading the virtual world of To Be With Hamlet from a gaming hub such as Steam, then connecting to M3diate to see the live performance. The only data transmitted live is the actors’ movements: the Elsinore Castle setting and the avatars are included in the advance download. So, remarkably, the VR performance ends up taking much less bandwidth than live streamed video.
So how much of a thing do Molina and Gochfeld anticipate live VR theater becoming? Both cautioned against it becoming gratuitous.
“Not every case demands the same technology,” Molina said.
But for Hamlet, the interactivity may lend itself well to getting inside the head of the psychologically complex title character. Experiencing Hamlet’s interaction with his father’s ghost from his point of view, or simply seeing the environment he lives in, may fill in some blanks for the audience.
For my part, after being dazzled by the VR technology, especially the ability to “teleport” among different perspectives of the action using video game controllers, I found it intriguing to see the ghost of Hamlet’s father up close, as Casey delivered a rousing monologue, and imagine what encountering such an imposing presence might feel like.
“There’s such an emotional journey in this scene,” Gochfeld said.

What’s next for To Be With Hamlet and live VR

There’s one significant factor limiting the theatrical range of To Be With Hamlet, however, as it currently stands. The motion capture system isn’t tracking the movements of the actors’ mouths or faces, so the actors’ body language must do all the work. Right now, Hamlet isn’t quite ready for his close-up. After seeing the live VR performance, I watched a pre-recorded version of the production. At the end, I was face to face with the avatar of Hamlet, which was missing eyes. For a moment, I thought I had wandered into a production of zombie Shakespeare.
That shortcoming is primarily a cost issue, Molina said. The equipment required to capture facial movements exceeds the production’s budget right now. In the next few months, he told Technical.ly, he will launch a crowdfunding campaign on Indiegogo to finish development and raise funding for future productions. Molina’s hope, he said, is that face and mouth movements will eventually be added. He and Gochfeld said that they aim to stage a longer excerpt of Hamlet in a theatrical venue, where audience members can experience the virtual-reality production communally.
That’s where Molina and Gochfeld anticipate VR is headed. Most people, Molina said, probably won’t purchase a dedicated headset to watch or play in VR, at least not yet. He pointed to the emergence of VR arcades, more of which are beginning to open in large cities. (There’s a pop-up arcade at the World Trade Center mall running for the next two weekends, for instance.) Watching performances in groups has traditionally been a part of the theatrical experience, and that element, he and Gochfeld agreed, is equally valuable for virtual reality.
“The shared presence is one of the key elements,” Gochfeld said.
Molina said he is especially excited about his role in enhancing the VR experience through live production, which according to him, no one else has attempted.
“I’m sure other people will figure out our secrets,” he said, “but it’s good to be able to say it started here.”

Series: Brooklyn

Before you go...

Please consider supporting Technical.ly to keep our independent journalism strong. Unlike most business-focused media outlets, we don’t have a paywall. Instead, we count on your personal and organizational support.

Our services Preferred partners The journalism fund

Join our growing Slack community

Join 5,000 tech professionals and entrepreneurs in our community Slack today!

Technically Media