By Dr. Michael Housman, Co-Founder and Chief Data Scientist of RapportBoost.AI
For the sake of digging into the nuts and bolts of what we do at RapportBoost.AI, let’s establish some definitions:
Live Chat Optimization: the act of analyzing a company’s chat and messaging data in relation to Key Performance Indicators including sales, conversions, and customer satisfaction, and making suggestions to live chat agents about the language they use to life KPIs.
Natural Language Processing: any method of processing the unstructured data present in natural language. In conversational commerce a live chat agent uses entire phrases. When we break down these phrases into smaller parts, categorize and label them, the chat agent’s language can be interpreted to generate insights and deliver results.
Sentiment Analysis: a machine learning approach to taking unstructured language that has been labeled with its sentiment (e.g., positive, negative) and building classifiers that attempt to predict that sentiment.
Although a variety of platforms such as Google Cloud Natural Language API and Microsoft Azure now make it relatively easy to engage in sentiment analysis, this tool has limited ability to lift KPIs. Many chat vendors implement some form of sentiment analysis. Zendesk sells an app integration that identifies the sentiment of a customer before the live chat agent initiates conversation. LivePerson features a “Meaningful Connection Score” that shows the live chat agent an emoticon to represent the customer’s emotional state.
One dirty little secret is that the majority of these sentiment analysis tools are based on the tags connected to movie reviews. As part of the Stanford Core NLP, researchers used language from movie reviews on Rotten Tomatoes and IMDB to develop a Sentiment Treebank. Although movie reviews are readily available sources of unstructured data that have been labeled with the writer’s intended sentiment, a sentiment analysis tool based on movie reviews is not the best fit for a brand looking to improve consumer experience.
Sentiment Analysis is certainly a step in the right direction towards automating the interpretation of unstructured text, but it’s a relatively small piece of the puzzle. Live chat optimization isn’t just focused on analyzing sentiment, it’s focused on understanding the immediate drivers of consumer behavior — whether that consumer has a positive customer experience, whether that consumer decides to make a buying decision, and whether that consumer becomes a repeat customer. Large brands that are determined to build relationships with their customers need to know more than the customer’s emotional state. They need to understand the hundreds and thousands of factors that occur during a conversation that influence a consumer’s behavior. This difference is key.
Sentiment Analysis accounts for just a few of the many tools that comprise the RapportBoost.AI live chat optimization platform. We’ve done a variety of studies on conversational commerce that look at how different variables affect a customer’s Key Performance Indicators. We’ve found that sentiment accounts for 5 to 10% of the outcome. It’s not negligible but it’s not huge. We also utilize APIs that measure the customer’s personality and find that those variables collectively account for 15 to 25% of the outcome. That’s especially true when we combine personality variables with other indicators like emoticon usage, spelling, grammar, and capitalization. It’s possible to identify strict grammarians in our data and, for those people, it’s far more important to be grammatically correct than to monitor their emotional state.
In fact, it’s worth noting that there is huge variation in the efficacy of the sentiment APIs out on the market: not all sentiment analysis tools are created equal. Over the two years it took to build its proprietary platform, RapportBoost.AI tested a half-dozen of them. When we applied those sentiment analysis tools to chat data and then simply eye-balled the results, it was pretty obvious that some were designed incredibly well and some were no better than guessing. Some of this may be attributable to the level of rigor that went into designing those tools and some likely stems from the fact that movie reviews are incredibly different than chat data, which is far messier and often more subtle.
If you’re thinking about how to turbo-charge your chats with customers to drive a specific behavior, sentiment analysis may be a part of the solution but it cannot be the entire solution. Yes, brands do want happy customers but they’re typically more interested in customers that will tell their friends about the amazing experience they just had engaging with a company’s front line agents - and that's a task for live chat optimization. If we want to drive that behavior, we need to build our models off of that behavior instead of the emotional cues that partially drive it.