This month Brandwatch rolled out a whole new sentiment model across the more than 100m online sources we cover in Brandwatch Consumer Research as well as the apps powered by Brandwatch like Cision Social Listening and Falcon Listen. 

It is a big upgrade on Brandwatch’s existing world-class sentiment analysis, providing around 18% better accuracy on average across previously supported languages.

This new model is also multilingual, meaning:

  • Official support has been added for 16 new evaluated languages, with more to come (bringing the current total of officially supported languages to 44)
  • The model will also attempt to assign sentiment to posts in any other language (and posts with no language identified, like emoji-only posts) when it is confident enough

Sentiment is one of the key metrics Brandwatch customers rely on for a number of important tasks such as:

  • Assessing brand health
  • Identifying advocates or detractors 
  • Detecting emerging crises
  • Understanding positive and negative topics related to brand or topic conversations

I sat down with one of the data scientists leading the team who developed our new sentiment model, Colin Sullivan, to ask him how it works and how it will benefit Brandwatch customers.

Hi Colin! We’re really excited to be able to see the fruits of your labour now available in Brandwatch’s sentiment analysis. Before we talk about this new sentiment model, tell us a little about yourself and your background.

Thanks Nick, we’re excited too! I’m a Data Science Manager leading several different projects here at Brandwatch and my background is in linguistics and computational linguistics. 

Linguistics is essentially a social science involved in figuring out the patterns and rules that govern how language works looking at the theoretical background, syntax, and semantics of language.

Computational linguistics is the study of how computers can model these same structures and apply these models to things like Natural Language Processing, language identification, and how things get indexed. And it is also used to analyze things like sentiment and topics within large volumes of text data.

This sentiment update uses an entirely new model. Why build a new way of analyzing sentiment?

Two key reasons. 

1. We wanted to make a jump to some of the state-of-the-art methods emerging in the research world. There have been some really exciting new developments in recent years that can help us achieve even better results.

2. We also saw an opportunity to simplify how we do sentiment at Brandwatch. We used to do the same procedure for each language that we supported which involved gathering a whole bunch of training data for each language, getting it labelled, learning about its linguistic patterns and then building a supervised learning model for each one. Moving to this new setup we have a single methodology that works for many languages at once.

This new model uses ‘transfer learning’. What is that exactly?

Over the last few years, the field of AI has made exciting progress with transfer learning which basically involves first training a model to have a more general understanding and then transferring that learning and asking it to apply it to a different task. This is very different to training a model only to solve a single, specific problem, which is how we used to do sentiment analysis.

So our new model has first been trained to have a general sense of how language is used. We then take a secondary step to point that model at a task like sentiment analysis.

The first step is very similar to how next-word auto suggestion works. A model with enough experience of language being used by humans can start to predict what the next words are likely to be if you give it some text. Next, we ask it to ‘predict’ a topic that encapsulates the meaning of a whole sentence or social media post, in this case the topics are ‘positive’, ‘negative’ or ‘neutral’ – it re-uses all the same information from step one.

This is actually how your brain works when you listen to someone talk. You are, subconsciously, constantly trying to predict what they are going to say next in order to better hear and understand them.

How does this help Brandwatch define sentiment better than before?

One of the key advantages of this new approach is that it makes it more robust when dealing with more complex or nuanced language. The new model can see past things like misspellings or slang. 

Previously, supervised learning models would be restricted to a fixed set of known patterns during training, which did not come close to exhaustively capturing all linguistically plausible ways of expressing a concept. New state-of-the-art models are better able to re-use what it already knows when faced with new or rare patterns. 

The transfer learning approach means the model will take what it knows to fill in gaps. For example, it can break down words it doesn’t know into parts that might give it clues (just like you would!).

And it works in almost any language because we are not training for a new language each time. This also means it can handle a wider range of regional dialects and posts where someone switches between languages.

See Brandwatch’s sentiment analysis in action

What kind of things will cause a post to be labelled positive or negative? Does the model look for ‘good’ or ‘bad’ words or phrases? 

The model takes into account the complete context of the document. The things that matter will be word order – not just positive or negative words. For example, if something is ‘not good’ the negation is easily understood. It will also understand things like emojis, syntax, and case-sensitivity. 

When you look at the results it appears like real-world knowledge. It can start to disregard names of that use terms that would otherwise convey sentiment in typical usage (eg Kind bars). If it has seen enough examples of language being used in a particular way it will even be able to see past double negatives or occasionally sarcasm, though this is harder as it doesn’t always have the additional real-world knowledge needed.

That said, if you spend any length of time on Twitter you might conclude that humans themselves are not entirely adept at detecting sarcasm!

This new model is multilingual. How does that work?

The pre-training has been done with a huge amount of data in 104 languages. Our first step was to take the pre-trained model and feed it many more examples of text drawn from social media. This step is needed to improve its ability to model the kinds of linguistic patterns that appear on social compared to standard news or formal contexts. 

Then we did the supervised step, where we gave it sentiment data and pointed it at the problem of detecting positive or negative posts. We used just 12 languages in one go but evaluated the results for 44 languages and found that the model had learned to handle these with a really good level of accuracy. Because the model had sentiment examples in enough languages it could focus on what was being asked of it in others by using what it already knew.

We can now officially support 44 languages but the model will classify sentiment in any language if it is confident enough. In the future we’ll be able to add official support for more languages far more quickly than before.

How accurate is this new model? And how do you measure accuracy?

Sentiment is inherently a subjective task and people interpret the definition of this task differently in the first place. It has been shown, for example, that two humans only agree on the sentiment of something around 80% of the time – and that is on tweets which are relatively easy to assess. 

Brandwatch customers can typically expect to see an average accuracy of around 60-75% but this will always vary with the type of data being looked at. We could see much higher accuracy if we just evaluated based on a set of IMDB film reviews for example. My team is tasked with calculating sentiment for posts from over 100m data sources, so we try to evaluate with a wide variety of data sets. (In other words, we try to make it difficult for ourselves!).

How you evaluate will have a huge impact on results. We have been able to benchmark our models on several public datasets for which some of the biggest names in AI and NLP have also provided predictions. In this apples-to-apples comparison of overall performance, Brandwatch Consumer Research is consistently a leader amongst these highly respected data science companies.

As a data scientist, what tips would you give for people using sentiment analysis in their work?

First of all I’d say make sure you set out to clearly define what you’re trying to do. Often what people consider to be positive or negative comes through the lens of what they are trying to gain insights about. 

Sentiment analysis is a tool rather than a single unarguable ‘truth’. If you are assessing brand health, you will want to break your data down into categories, audiences, and topics to use sentiment to identify what exactly is driving public opinion and how you can improve it. If you are trying to predict potential crises, you need to be more focussed on shifting trends or spikes in your data so you can act more quickly.

Sentiment is most useful in aggregate – how the distribution changes over time. If there are peaks and valleys these are significant. It’s always important to set benchmarks and then investigate more closely when the data shifts from the norm. 

Colin, thank you for taking the time to talk and for your team’s hard work in bringing Brandwatch customers this exciting new development to the platform!

If you’d like to see Brandwatch’s sentiment analysis in action click here to book a meeting. If you are an existing Brandwatch customer, you will already be benefiting from the new sentiment model in your projects today.