UserTesting bolsters ML-based UX testing with $1.3 billion acquisition by Sunstone, Thoma Bravo

Getting insights into how users experience and use software was previously only possible by having humans do all the user testing. With the advent of modern sentiment analysis and machine learning (ML), more information than ever before can be gained through testing.

User testing is among the pioneers in the space using ML techniques to help discover and analyze user behaviors. The past two years have been a whirlwind of activity for the company. In 2020, UserTesting raised $100 million dollars in funding, and a year later, in 2021, the company went public on the New York Stock Exchange (NYSE) under the symbol USER.

Today, UserTesting announced that it has reached an acquisition agreement for $1.3 billion by Thomas Bravo and Sunstone Partners. When the deal closes, the plan is to merge User Zoom – which Thoma Bravo acquired in April 2022 – with UserTesting, to create an even broader set of features for user experience testing.

“We’re in a space where we’ve built a set of technologies to capture a kind of feedback that we call a customer experience narrative,” Andy MacMillan, CEO of UserTesting, told VentureBeat. “UserZoom has a set of additional different research techniques and methodologies that could complement some of our customer experience stories.”

How UserTesting integrated ML

Over the past two years, UserTesting has made significant investments in technology that help it learn from its testing.

The test involves recording users to see how they interact with apps, including what they click on, and asking users to tell their stories. MacMillan said his company has invested in using ML to help extract insights from recorded user experience content.

“We really take unstructured content, but turning it into something structured,” MacMillan said. “We trained a set of machine learning models to help uncover what we call moments of insight.”

Moments of insight are those nuggets of information that can help identify trends that will improve the user experience. UserTesting uses several ML technologies, including natural language processing (NLP), computer vision and the analysis of intentions and behavior.

Among the things that ML allows for UserTesting is the ability to perform click path analysis, which can track where a user is going and what they are actually trying to do when they click something. User sentiment analysis is another key attribute that ML helps with, along with the ability to see if the user is happy with an experience.

Going one step further, UserTesting uses ML to help power a visualization that overlays intent and path behavior to gain insight into how people navigate a site or app.

“There’s a lot of things we can determine about the behaviors we see people exhibiting, as they go through a process,” he said.

The virtuous circle of ML

ML does not exist in a vacuum; by definition, they are machines learning from data.

MacMillan explained that the UserTesting approach to ML is a virtuous circle, where the models his company builds are continually validated and extended with new data from user testing sessions that already benefit from ML. He added that the ability of humans to validate ML models with their own eyes helps build trust in the models.

“We collect these customer experience scenarios — sort of like end-to-end videos — and use the machine learning models to direct people to the insightful moments,” MacMillan said. “But you can always dig in, you can always say ‘oh the model says this, let me look at some of this customer experience story’ and see if the intent really matches the sentiment.”

One of the biggest ML challenges for any organization, according to MacMillan, is having the right kind of training data. UserTesting already has a video capture, which shows what is happening on a screen, and the test also collects user click data. Testing is done against a test plan, so there is a baseline expectation for what users are expected to do. UserTesting has dedicated staff who also tag content as part of their daily work to help train and optimize models.

“The goal of the product is to help connect teams directly to real customers and real humans to get human insight into the product,” MacMillan said. “We think machine learning is really just a way to help people connect to those moments of insight, but those moments are still human.”