A real-life walkthrough of how to make cool things with augmented reality

We’ve been hearing a lot about augmented and virtual reality in the last few years but not seeing a ton of it in the real world. Now, for all the tinkerers and creatives out there, is a little guide on the problems and fixes that Brooklyn’s Jane Friedhoff encountered (and solved) while working on an AR prototyping project for the Google Creative Lab.

In a new post on Medium, Friedhoff talks about what was easy and what was hard in making an AR video experience of pop singer Grace VanderWaal’s song “Moonlight” with the lyrics written out in augmented reality that unfurl on the viewer’s screen as they move through the world.

“We came across Grace VanderWaal’s lyric videos and loved their style,” Friedhoff wrote. “Her videos frequently include her handwriting, and it made us wonder: could we make a handwritten lyric video, but explode it out into AR? What if the song wrote itself out, as it was being sung, all around you ?”

Today I can finally share my favorite bizarro tech hack that I've ever done: animated handwriting in augmented reality in C#, using tricks I learned from… CSS! https://t.co/EPjEjH86Pn

— jane frie(n)dhoff (@JFriedhoff) April 2, 2018

In the original video, the 14-year-old pop star VanderWaal, who came to fame competing on America’s Got Talent, walks and sings down and around what seems to be Kent Avenue on the Williamsburg waterfront. Friedhoff’s team at Google took the song and added hand-written neon lyrics to their own video shot in the perspective of walking around as well. In Friedhoff’s version, the viewer can look at, around, and through the lyrics on the screen as they move the screen.

Friedhoff, who is also a game developer, found a way to use game engine Unity to show the 2D handwriting in 3D space. Friedhoff’s previous creative work includes a favorite game of ours she released for free online called Handväska!, the point of which is to whack fascists with your purse.

Friedhoff’s whole post is worth a read if you’re curious about this stuff and shouts out another Technical.ly fave, Chris Whong, for learning how to get the handwriting right in CSS.

Snapshots from the video look cool. Maybe we’ll start watching some videos in AR sooner rather than later.

Read the full story