The complete guide to sentiment analysis

Machine-based sentiment analysis is becoming ever more sophisticated, as deep learning allows programmes to accurately analyse content for context and meaning. But how does it work, what are the practical applications, and what could the future hold?

What is sentiment analysis

Sentiment analysis is the understanding of what the author of a piece of content felt about its subject when it was produced. Straightforward enough when you’re reading the morning paper or watching a news report, or even perusing an online review. More complex for a multinational corporation trying to understand the combined attitudes of its many stakeholders towards its business activities.

The latter requires the processing and interpreting of millions of pieces of content, encompassing multiple languages and formats to establish the sentiments, both positive and negative, contained within.

To achieve this, automated sentiment analysis is used to process vast quantities of data, and report on who is feeling what about given topics or events. With sophisticated sentiment analysis systems, nuances such as personal bias can be identified, and strength of feeling quantified.

This in-depth form of text analysis results in a numerical sentiment score being applied to the sum of the data.

Why your business needs sentiment analysis

A business requires sentiment analysis if it has any interest in understanding how its stakeholders feel about it. The perceptions of employees, shareholders, customers, the media and the wider community form the basis of corporate reputation. To protect that reputation from risk, businesses need to understand what it’s built on. Sentiment analysis is central to this.

Gathering content is just the start. A mass of data is of little value unless usable intelligence can be gleaned. To do this, all mentions of an organisation or brand across every media channel need to be captured, processed and analysed. This type of AI-led processing reveals who is saying what, where they are saying it, and to whom.

As such it gives vital insight into that organisation’s reputational standing among its many stakeholders.

Sentiment analysis is perhaps the most important element of a stakeholder intelligence framework, as it provides a detailed understanding of what a diverse base of stakeholders thinks and feels about a particular organisation, brand, product, service or event at any given time.

Applications and challenges of sentiment analysis

The practical application of sentiment analysis is increasingly common among businesses. At the top level, an accurate predictor of how different stakeholder groups will react to a particular corporate behaviour can inform boardroom decisions.

A more specific application could be enabling market analysts to understanding investor enthusiasm, or indifference, towards a corporate transaction.

It could also support communications teams performing market research on how consumers are responding to marketing campaigns.

The customer services team, responding to customer feedback about a product or service, will be better prepared if they know how people are reacting to it.

It can assist human resources in monitoring, and improving, workforce satisfaction.

And it provides public relations professionals with real-time coverage of whether their clients are being reported on positively or negatively in the media.

The advantages of an effective sentiment analysis tool are multiple, and so are the challenges of creating one.

In order to do all of the above, this tool must be able to comprehend the meaning of language in a given context: words mean different things depending on how and where they are used – and even by whom. In addition, it must correctly interpret irregular ways of communicating – such as slang, jargon or sarcasm – and have the ability to incorporate novel words and phrases into the model.

Adding to the complexity is the fact that smaller sentences or samples tend to yield less predictable results. The tool also needs to both identify and link particular subjects.

Ultimately, a sentiment analysis tool must be able to accurately follow an online conversation conducted in acronyms and abbreviations; or dissect a 300-page financial report peppered with technical language.

Earlier practitioners of sentiment analysis tended to underplay this complexity, creating “tone” scores which calculated a 3-point (positive, negative, neutral) understanding of a piece of content before applying this sentiment to any entity contained within the content. This broad-brush approach, allied with aggressive (and misleading) sentiment accuracy stats, have ironically caused damage to the industry’s reputation.

As technology and the market matures, greater nuance and sophistication of approach are becoming possible, meaning that none of these previous challenges is now insurmountable.

Currently, deep learning reproduces a reasonable proportion of sentiment as it is scored by human analysts. Larger data sets tend to provide consistent results, resulting in sentiment calculations well-aligned with training and test data sets.

We can also expect continued improvements in sentiment measurement, with future progress achieving greater nuance of understanding of how different stakeholders view a subject.

Types of sentiment analysis

Technology provides different routes to determining if content contains positive sentiments or negative sentiments – or even whether any sentiment is expressed. The current cutting edge comprises:

 Machine learning algorithms: ML algorithms employ support vector machines (SVMs) and the random forest technique. SVMs find groupings and relationships across many variables. For example, a model could be trained using words and phrases from corporate forecasts to calculate sentiment scores for new forecasts. The random forest is populated with decision trees that step through variables. Each step is a decision on positive, negative or neutral sentiment. The more trees, the greater the accuracy. 

• Deep learning: Deep learning uses neural networks, which function like a basic nervous system. An input, such as an article about a company, is converted into standardised terms, each of which is given a value. These are then weighted based on the sentiment expressed, and the resulting set of values used to assign an overall sentiment score to the article.

• Long Short Term Memory: Words also hold meaning based on their context. Sentiment applications use recurrent neural networks called Long Short Term Memory (LSTM) to ‘remember’ the way words have been used before. By retaining a memory of a prior relationship between words representing a particular sentiment, it makes sentiment calculations more accurate.

How sentiment analysis tools work

Sentiment analysis has evolved from basic, dictionary-based definitions of ‘good’ and ‘bad’ words into a powerful business tool. Today, alva’s sentiment analysis is underpinned by core proprietary technology, encompassing machine learning and Natural Language Processing (NLP). Above this, a layer of sector-expert analysts interpret the automated analysis and provide ongoing refinement.

Input to the solution comes from publicly available data sources incorporating over 25 million pieces of content per day. That content is sourced in over 100 languages and from more than 150 countries, from over 500,000 individual publications. Media channels include traditional print and broadcast, online reporting (including paywalled), and social media.

Real-time data monitoring enables the solution to identify relevant company and topic mentions from all of these data sources. Content analysis also tracks shifts in volume or sentiment associated with given clients, topics, stakeholders, regions, channels and keywords.

Machine learning-based topic modelling also tags repeat phrases appearing in conjunction with organisations or sectors, surfacing previously unknown issues.

Beyond set phrases, the way in which words combine also affects the sentiment of a statement. alva’s deep learning models can build sentiment representation of complete sentences based on their structure. Training with human-score real data, using neural networking, enables the sentiment of longer, more involved phrases to be understood. This is the most complex level of text analytics currently available.

The future of sentiment analysis

As the technology develops, and there is greater demand for more complex applications of it, sentiment calculations will evolve. Some of the areas alva is working on in this field include:

• Multi-stakeholder perspectives: While alva’s stakeholder solution will currently surface multi-stakeholder views, the relationships between particular stakeholder perspectives and markets are more complex. A streamlining drive that increases share price but reduces employment will be received very differently by employees and investors, for example. Coverage in the media, the reaction of local communities and even government is likely to focus on the job losses, resulting in a low sentiment score, even though shareholders benefit.
• Recognising topics as sentiment carriers: Sentiment has differing time scales. A company’s reputation may attract a long-term sentiment, with people slow to change their baseline opinion, but shorter-term sentiment is driven by events or headline topics. Rather than simply summing up sentiment toward a company at a point in time, the three-dimensional picture is of peaks and troughs on overlapping timelines, reflecting different stakeholder views of these topics.
• Authors and sources generating signals: Authors and sources have greater or lesser influence depending on the subject matter. Future sentiment analysis will incorporate exactly how much impact sentiment the author has. In 2019, the world’s leading epidemiologists had little media presence or popular influence. Since 2020’s coronavirus outbreak, they have shaped how everyday life is lived.
• Identifying specific emotional states: Rather than simply a good or bad effect, sentiment tools will be able to genuinely distinguish between the emotional state of the author, such as frustrated, sad, excited or aggressive.
• Image and video sentiment: A picture speaks a thousand words, and is much quicker to post to social media, so a complete sentiment solution must also measure the effect and impact of images, memes, or videos stories.