We have used virtually every social monitoring software – Radian6, Alterian, Spiral16, Sysomos and more. Although we’ve used some more than others, there’s one thing we’ve learned from experience with these tools – automated sentiment is about as useful as a chocolate teapot.

If you are shown a demo, you will be told that all you have to do is click a couple of buttons and you will get a report that spells out whether online conversations about your brand are positive or negative. Even better, you’ll be able to see how this trends from day to day and month to month.

So you’ll sign a contract, beaming with pride as you delight your boss with tales of your new tool’s capabilities. You will pay monthly invoices with a smile thinking about the beautiful dashboards you’ve seen. And one day, when you’ve got some time on your hands, you’ll look more closely at the data and realise you’ve been passing on garbage to your executives for several months. 

Defenders of automated sentiment analysis will say that no software can be 100% accurate. They follow with claims that they can achieve 70% accuracy. Even at this level, the attraction is still seductive. You might be thinking “70 percent accuracy? That’s close enough, I can live with that.”  The problem is that once you start to look a bit deeper this will drop, perhaps even as low as 25%. 

But accuracy is not the only issue. Even if you accept that accuracy doesn’t matter then there are still many other shortcomings of automated sentiment analysis. Here are a few.

1. Consistency

You may decide to live with a degree of inaccuracy. The problem is, it needs to be consistently inaccurate. But it rarely is and because there’s little consistency you can’t be confident that the trends in data are even accurate. If you can’t be confident that the trends are accurate you’ve got yourself a limited tool from an analysis standpoint.

2.  Sentiment is not created equal

A negative mention in a mainstream news article like The Times, Telegraph or even The Sun weighs more than a negative mention on Facebook made by an individual, sometimes by seismic magnitudes, and trying to make sense of the impact of a sentiment is hard enough without introducing authority and influence into the equation.

3. You are not the centre of their world

Despite what you might have heard very few people are talking about any one specific brand. Consequently they are comparing you in a set with others and automated sentiment analysis has no way of judging this and just reports scores without any important context.

4. Human or machine?

Many sentiment tools give a very high sentiment weighting and high influence scores to blogs. The problem with measuring blogs is that most of social monitoring tools measure as a “blog” wasn’t created by human being. As such attempting to ascribe any sentiment into these posts is a pretty pointless exercise.

5. Does it reflect the market?

The problem with any online channel is that its user base often isn’t fully representative of your average consumer. Consequently the mentions you’ll pick up are skewed towards the niche people that populate the channel. Plus, when you monitor individual channels, blogs or forums, you’re likely to get multiple mentions from the same user, which further disrupts getting an accurate read on what it is you’re really trying to measure.

6. Bots, Mambots and other online creatures

Lastly, much online content is created automatically by people with an interest to do so. So if someone wishes to spread their views widely they can. Thus a lot of the data collected by automatic sentiment tools is spam, pay per click or paid likes and follows. Take these into account and your accuracy drops even further.

So how do you get the best from sentiment analysis?

The answer is that there are no short cuts – you have to do it the hard way and in person. Nothing beats the human mind at discerning the true sentiment of a statement. But there are a number of ways to make things easier. 

First, limit yourself to a handful of social media channels like Facebook, YouTube and Twitter. That’s because lots of people use them and their content is still more likely to be made by a human than in any other channel you can monitor. Facebook is probably cleanest but you have to be a bit wary of comments on Twitter because more tweets are created by bots and spammers. 

Using Twitter therefore introduces more volatility because in addition to judging sentiment you first have to judge whether something came from a human being or not. Whatever you do, do not measure the sentiment of anything classified as a blog, a forum or a piece of mainstream news.

Second, don’t try to analyse everything. Use the market research principle of looking at a random sample. About 200-300 comments across the channels you are using should be more than enough. Just start counting them – mark negative, neutral and positive with coloured pens. If it sounds tedious, remember, you only really need a couple of hundred. Even if it takes you ten seconds per mention it will only take you a half hour to do. 

Now add them up and, voilà, you have an accurate sentiment analysis metric you can proudly present to your management team. You’ve now got something really simple yet more accurate than anything generated by automated tools. Here is another thing. If you have the software available, you can run some basic text analytics to look at word associations and relationships. So in addition to sentiment you will also see whether there are any common themes being discussed and how important are these really to the overall online conversation.