This company is using AR headsets to offer detailed insights on how customers shop 

A 16-year Pittsburgh consulting firm is pivoting to tech by launching a custom platform with a lofty goal: to revolutionize focus groups. 

Founded in 2008 to provide quality assurance and software testing, North Shore-based Rivers Agile hasn’t shared much about what it’s been up to over the past couple of years. Founder and CEO Ben Wilson is excited to change that with the Robotics Discovery Day launch. 

“We spent most of our 16 years of existence building great products for other people,” Wilson told Technical.ly, “and now we’re rolling out our own.”

That product is an augmented reality (AR) research tool called Tupleo. It’s software that can be installed on a VR headset customers wear while shopping for or interacting with a company’s product. It tracks how they react — verbally or through physical cues — to gather feedback for a company, similar to how a focus group works, but resulting in more granular, quantifiable data.

Rivers Agile recently joined the Pittsburgh Robotics Network (PRN), and its approach has been “very Pittsburgh,” according to PRN Executive Director Jennifer Apicella. 

“Between AI, computer vision and augmented reality, they are really maximizing advanced technology,” Apicella said. “Innovation first, innovation forward.”

Rivers Agile community involvement, especially in AI, goes back even further. In 2019, it announced a workshop to support local, early-stage AI, machine learning and fintech companies with its knowledge of the tech from its time consulting.

Building on existing tech for a smooth transition into the field 

Rivers Agile has been developing Tupelo for the past 15 months. 

It was initially designed for a large beverage manufacturer that wanted to analyze best practices for end caps, those displays at the end of supermarket or department store aisles that showcase high-demand items. The beverage company sought data that would help it determine the best placement for its most profitable products. 

The Rivers Agile team decided to use existing tech like Meta’s Oculus headset and the Unity programming platform to develop Tupelo, which uses AR to overlay digital content onto the real-world environment. 

It then combines eye tracking, mood detection and image recognition technology to gather in-depth user feedback. 

“We are using hardware and repurposing it in a way it was not necessarily intended,” said Wilson.

Tupelo tracks the user’s eye movements and marks their focus on specific objects as “events.” These actions are displayed as an arrow on the recording, indicating exactly what the user is looking at. This allows the platform to track customer behavior by recording data like how often a participant looks at a particular product or whether certain items go unnoticed entirely. 

AI programmed to understand how humans express themselves

The Rivers Agile team also added a sentiment analysis feature powered by AI. 

The program listens to what users say, transferring speech to text, and an algorithm interprets the feedback. Sometimes it can be simple, like a user saying, “I hate this” and the program registering the feedback as obviously negative, but it can also pick up on more subtle user signals using the sensors on the headset. 

The team associated ranging human emotions with subtle expressive cues, or the physical expressions of feedback. The headset senses those movements and is programmed to understand whether it’s positive or negative input.

“These headsets have a ton of sensors and some of them are exterior, so when you smile, your cheekbones go up,” Wilson said. “There’s no denying it, it’s a fact of being a human being. Understanding a number of other human emotions and defining those was a model that had to be created.” 

What’s next? Applying the tech to new industries, like robotics

Rivers Agile is now considering other use cases for AR to track human feedback. 

For example, the software could be used in store cameras to learn what customers are looking at or, it could help automakers determine how drivers interact with car features. 

“With each application to a different industry we’re going to find small pockets of new functionality to be developed,” Wilson said.

Looking ahead, Wilson said Rivers Agile is focused on forging new partnerships with interested companies and expanding Tupelo’s capabilities. 

One area it’s particularly excited about is remote troubleshooting, which would allow two people in different locations to wear VR headsets and guide each other through complex tasks, such as fixing a car or learning how to fly a plane.

“In the future, we’ll be placing further emphasis on deep tech,” Wilson said, “wrapping our arms around the robotics community and continuing to serve them.”