Backend Performance Optimization: What Actually Moved the Needle
Backend performance optimization has no shortage of advice.
“Add caching.”
“Shard the database.”
“Use async everywhere.”
Most of it sounds smart. Some of it helps. A lot of it doesn’t move the needle at all.
After working on production systems at scale, I’ve learned one uncomfortable truth:
Most performance wins come from boring, unsexy fixes — done at the right time.
This post cuts through theory and focuses on optimizations that measurably improved latency, throughput, or stability, and a few that didn’t.
Start With a Latency Budget (Or You’re Guessing)
Before touching code, we defined a latency budget.
Example (p95 target = 200ms):
- Network + LB: 20ms
- App layer: 80ms
- Database: 80ms
- Buffer: 20ms
This single step changed everything.
Why?
- It forced ownership (“DB is blowing the budget” vs “backend is slow”)
- It stopped random optimization
- It made trade-offs explicit
If you can’t say where time is allowed to go, you’ll optimize the wrong thing.
Query Tuning Beat Caching (More Often Than Expected)
Caching is the default instinct. But in many cases, fixing the query was cheaper and safer.
What actually helped
- Removing unused JOINs added “just in case”
- Replacing
SELECT *with required columns - Fixing filters that prevented index usage
- Breaking one monster query into two predictable ones
Real outcome
- 40–70% query time reduction
- Zero cache invalidation complexity
- No stale data bugs
Rule of thumb
If a query runs on every request and hits the DB anyway, optimize it first. Cache later if needed.
Indexing Mistakes Were Costlier Than Missing Indexes
Adding indexes is easy. Removing bad ones is where the wins were.
Common mistakes we found:
- Indexes on low-cardinality columns (
status,is_active) - Composite indexes with wrong column order
- Indexes never used in real query patterns
- “Just in case” indexes added years ago
What worked
- Checking actual query plans (
EXPLAIN ANALYZE) - Tracking index usage stats
- Designing indexes around real WHERE + ORDER BY clauses
Result:
- Faster writes
- Smaller indexes
- More predictable performance under load
Indexes are not free. Treat them like production code.
N+1 Problems: Logging Beat Tooling
Everyone talks about N+1 detection tools. Useful, yes — but logs caught more issues than dashboards.
Get Sowndapan M V’s stories in your inbox
Join Medium for free to get updates from this writer.
We added:
- Query count per request
- Slow request logs with query summaries
- Alerts on sudden query count spikes
This surfaced:
- ORM lazy-loading surprises
- Accidental loops calling repositories
- “It’s just one more query” situations
The key insight
N+1 problems often appear only under real traffic patterns, not in dev or staging.
Observability > assumptions.
When NOT to Cache (Yes, Really)
Caching is powerful — and dangerous.
We stopped caching when:
- Data was highly volatile
- Cache invalidation logic exceeded query cost
- Bugs were harder to detect than DB issues
- Cache misses caused traffic spikes
In some cases, a fast DB query was more reliable than a flaky cache.
Cache when:
- Reads vastly outnumber writes
- Staleness is acceptable
- Invalidation is simple and deterministic
If your cache needs a wiki page to explain, rethink it.
Async Didn’t Fix a Slow System
Async, queues, and background workers are great — when used intentionally.
They did NOT help when:
- The bottleneck was still the database
- Downstream services were slow
- We just moved latency out of sight
They DID help when:
- Work was genuinely non-critical to the response
- Retries and backpressure were well designed
- Failure modes were clear
Async is not a performance fix. It’s a load-shaping tool.
The Biggest Win: Measuring the Right Thing
The single biggest improvement came from changing what we measured.
Instead of:
- Average latency
- CPU usage
We focused on:
- p95 / p99 latency
- Saturation signals
- Error rate under load
- Latency per dependency
Once metrics matched user pain, optimizations became obvious.
Final Takeaways
What actually moved the needle:
- Clear latency budgets
- Query tuning before caching
- Fewer, better indexes
- Observability over assumptions
- Saying “no” to unnecessary complexity
Backend performance isn’t about clever tricks.
It’s about clarity, restraint, and relentless measurement.
