Why Generic Sentiment Analysis Doesn’t Work for Trading — and What I Built Instead
I’ve been building an automated swing-trading system for NSE stocks — paper trading only, running daily on GitHub Actions. A few weeks ago, I decided to add a news sentiment layer, and my first instinct was to reach for an off-the-shelf tool like VADER or TextBlob. I’m glad I didn’t.
The Problem with Generic Sentiment Lexicons in Finance
In 2011, Loughran and McDonald published a paper in the Journal of Finance that’s since become foundational in financial NLP. They took the Harvard-IV General Inquirer — a standard, widely-used sentiment dictionary at the time — and checked how well its “negative” word list actually held up against real 10-K filings.
About three-quarters of the words it flagged as negative weren’t negative at all in a financial context.
Words like “liability,” “tax,” “cost,” “capital,” and even “president” (as in “vice president”) read as bad news in everyday English, but are completely routine vocabulary in business writing. A generic model scoring financial headlines is, in a very literal sense, scoring the wrong language.
Get Tanmay Kaper’s stories in your inbox
Join Medium for free to get updates from this writer.
So instead of importing a generic library, I built a small sentiment engine from the ground up following the LM methodology, creating separate word categories for:
- Negative
- Positive
- Uncertainty
- Litigious (the category that turned out to matter most)
Two Free Sources, on Purpose
Headlines come from two independent free sources:
- Yahoo Finance’s news feed (via
yfinance) - Google News RSS
No API keys, no paid tiers. Two sources isn’t redundancy for its own sake — I’d already been burned once elsewhere in this project by depending on a single free data source that silently changed its response shape. If one schema shifts without notice, the other source still covers you.
The Negation Bug That Taught Me the Most
Early testing threw a sentence at the scorer that should obviously read negative:
“Quarterly results were not impressive, margins weak.”
It came back positive.
- The Bug: My negation check looked backward a few words from any sentiment-bearing term to see if a negator preceded it. “Not” was technically within that window of “weak” — it just belonged to a different clause, negating “impressive,” not “weak” at all.
- The Fix: Scoping negation to stop at clause boundaries (commas, “but,” “however”) instead of just counting tokens. A small detail, but it’s exactly the kind of bug that looks fine on easy test sentences and quietly wrecks real headlines.
Why One Dramatic Headline Shouldn’t Move the Needle
A single very negative headline about a stock with otherwise thin news coverage gets shrunk toward neutral before it’s trusted — using empirical-Bayes shrinkage toward a prior. This is the same statistical device used elsewhere in the project to stop a small sample of trades from over-swinging a pattern’s calibrated weight.
- A cluster of five corroborating headlines gets trusted much more than one.
- Headlines also decay in relevance: today’s news outweighs a five-day-old story on an exponential half-life, ensuring stale coverage doesn’t linger in the score forever.
Two Different Jobs, Kept Separate
The sentiment score performs two distinct roles, and I deliberately didn’t collapse them into a single number:
- A Soft Factor: A small (10%-weighted) input into a larger cross-sectional conviction score that also includes momentum, trend quality, volume, and relative strength. Mildly positive news nudges conviction up a little. It’s a weak, noisy signal next to price/volume factors with decades of published evidence, and it’s sized accordingly.
- A Hard Veto: An independent, absolute-threshold check that can block a trade outright on a fresh, corroborated cluster of litigious/fraud-flagged headlines — regardless of how good everything else looks. This handles the asymmetric risk a pure price-based system is structurally blind to until the news is already priced into the chart.
Where This Fits Into the Bigger Picture
The sentiment engine is one of five layers a trade candidate has to clear:
- Technical Pattern Scanner
- Fundamental Soft-Gate
- Cross-Sectional Alpha Score
- Sentiment Layer
- Multi-Tier Risk Manager (portfolio risk budget, sector concentration caps, drawdown circuit breaker)
Every layer can be independently toggled on or off, specifically so I can A/B test whether each one is actually contributing anything, instead of just trusting that a sophisticated-sounding feature is helpful.
Looking Ahead
I’m not going to claim this makes money — the live paper-trading sample is still small, and I’d rather build the validation tooling properly than dress up a handful of trades as a track record. That’s genuinely the more interesting engineering problem anyway: not “does this sound smart,” but “how do I actually find out.”
- Repository (Fully open, MIT licensed): github.com/tanmaykaper/Paper-Trading-Bot
Would genuinely welcome pushback on the lexicon design or the shrinkage constants — still tuning both!
