Guide 03 / 06
Reading sentiment on X without fooling yourself
Sample size, bots, brigading and the other ways a sentiment number can say something that isn't true.
A sentiment score looks objective because it is a number. It is really a summary of a sample, and every choice you made while collecting that sample is baked into it. Most bad readings of mood on X come from the sample rather than from the scoring method.
Here are the mistakes that come up most, and what to do about each one.
The sample is too small
With 20 replies, three people changing their mind moves the result by 15 percentage points. That is enough to turn "mostly positive" into "mixed".
A rough rule from survey statistics helps here. With a simple random sample of 100, a percentage has a margin of error of about plus or minus 10 points. With 400, it is about plus or minus 5. Replies on X are not a random sample of anything, so the real uncertainty is larger, but the rule tells you which differences are worth talking about. If one week shows 58 percent positive and the next shows 52 percent on 80 posts each, you have not found a trend.
When you only have a few dozen lines, report counts instead of percentages ("14 of 31 replies were negative") and read every line.
The replies are ranked
The reply list under a post is not in order of arrival. X ranks replies using its own signals, and it may push some down or hide them behind a prompt for possible spam or offensive content. The first screen of replies is a curated selection.
If you only copy what is visible without scrolling, you are measuring X's ranking. Scroll to the end, expand hidden replies if you want the whole conversation, and write down which you did.
The same applies to search. The Top tab favors popular posts, which are often the funniest or angriest. The Latest tab is noisier and closer to what ordinary users said.
A few accounts are doing most of the talking
Sort your collected posts by account before scoring. In heated threads, one or two people often post a dozen replies each. The checker counts lines, so one person arguing in ten replies counts ten times.
Decide on a rule and apply it every time. A simple one is to keep each account's first reply and drop the rest, or to cap each account at three. Say which rule you used when you share the number.
Bots and spam
Automated accounts reply to popular posts with promotions, scams and generic praise. Some bot replies are positive ("Great post! Check my profile"), which inflates the positive side. Others pile onto controversial topics in an organized way.
X's own rules ban platform manipulation and spam, and it removes some of this, but plenty gets through. Signs worth checking by hand:
- the same or nearly the same text across several accounts
- handles that are a name followed by a long string of digits, with no profile photo
- replies that have nothing to do with the post
- a burst of similar replies within a minute or two
You won't catch every bot, and you shouldn't try to guess too hard. Remove the obvious ones, count how many you removed, and keep that number in your notes.
Pile-ons and brigading
Sometimes a post gets shared in another community, on X or somewhere else, and a wave of people arrives together. They may never have heard of your brand before that day. The replies turn sharply negative within an hour, then go quiet.
This is real sentiment from real people, but it describes the group that arrived, not your customers. Look at the timing. If most negative replies landed in a short window after a large account quoted the post, note that and consider scoring the before and after separately. Quote posts from the account that started the wave will often explain what happened.
Comparing things that aren't comparable
A product launch thread and a support complaint thread will never have the same mood, and neither will a post about a price change and a post about a giveaway. Comparing their scores tells you about the topics.
Comparisons are more useful when you hold everything else steady: the same search query, the same collection method, the same day of the week, roughly the same number of posts. Then a change in the score might mean a change in mood.
The score ignores what people are upset about
Two threads can both be 40 percent negative for completely different reasons. One is angry about price, the other about a bug. The number is the same and the action you should take is not.
After scoring, filter the CSV to negative lines and read them. Group them by reason in a spare column: price, bugs, support, shipping, "just doesn't like it". A short list of reasons with counts is usually more useful to a team than the headline split.
The method has blind spots
Word-list scoring misses sarcasm and context, as the first guide explains. Twitterment flags lines that look sarcastic, and you should read every one of them. Also skim the neutral pile. It contains questions, jokes the list didn't understand, and replies like "lol same" that only make sense in context. If a large share of your sample is neutral, the real mood is hiding there.
A short checklist before you share a number
- How many posts, from how many different accounts?
- Which search query, which tab, which dates?
- Did you scroll to the end and expand hidden replies?
- How many duplicates and likely bots did you remove?
- Did a burst from one outside source dominate?
- What are the top reasons behind the negative lines?
If you can answer all six, your number will survive a skeptical question in a meeting. If you can't, share the counts and the reasons and leave the percentage out.
Questions
- What is brigading?
- Brigading is when a group of accounts, often organized somewhere else, arrives at the same post or account at the same time to flood it with replies, usually hostile ones. It can make a small group look like a crowd.
- Can I remove negative replies that look like bots?
- Remove accounts you have good reason to think are automated or duplicated, whichever direction they push, and say in your notes how many you removed and why. Dropping only the negative ones turns your measurement into a wish.
Twitterment is independent and not affiliated with X Corp. Features and search operators on X change from time to time; if something here doesn't match what you see, check X's own help pages.