Sharding is Harding
Database Sharding
Sharing is Caring, And if you add a d for database it becomes Shar-d-ing.
The name itself suggests to share the data because if you try to eat a pizza whole without sharing it with your loved ones, you choke and similarly so does a Database when you try to force everything on it.
In Simple terms, Sharding is splitting one huge slice of pizza into multiple i.e.. you try to split your one big database into multiple small databases known as shards.
The interesting thing is, the word Sharding is not an official English word so you can’t find it in a dictionary.
Okay just imagine that you found the ultimate recipe of the most delicious and scrumptious food mankind/womenkind has ever know and you open up a restaurant, it goes Viral and orders start to pile up. What do you do? Exactly! You start to shard your restaurant that is, you open up multiple small restaurants in different locations so the load on each restaurant is distributed equally. This is what sharding does to a Database Server.
There is always a possibility that you could increase the capacity of your existing restaurant but you can only do that so much that there is a limit to it and this what we callb vertical sharding. This blog rather focuses more on Horizontal Sharding.
In a more technical sense, Horizontal Sharding means splitting one giant database into smaller pieces (called shards), each living on its own server. Together they act like one database, but no single machine has to hold or serve everything.
How it works:
- Pick a shard key. This is the column that decides which shard a row goes to (user_id, country, etc.).
- Apply a rule. A function maps each key to a shard, Ex: if we have four database instances then four shards (sh1, sh2, sh3, sh4)
- Route the query. A router (or your app) sends each query to the right shard.
Think of it as a post office sorting mail by zip code. The zip code is the shard key, and each sorting bin is a shard.
1. Range-based sharding
Split by value ranges.
- Shard 1: users with ID 1 to 100
- Shard 2: users with ID 2 101 to 200
2. Hash-based sharding
Run the key through a hash function and use the result to pick a shard.
Get Kunalagarwal’s stories in your inbox
Join Medium for free to get updates from this writer.
Ex: if user_id is the shard key
shard_number = hash(user_id) % 43. Geo-based sharding
Users in India go to the Mumbai shard, users in Germany go to the Frankfurt shard.
Sharding is powerful, but it’s not easy peasy:
- Cross-shard joins: Joining data on shard 1 with data on shard 5 is like organizing a group project where everyone lives in a different country and nobody believs in email or phone calls because everyone is each other’s ex.
- Transactions: Keeping changes atomic across shards is hard
- Resharding: Moving data later is like rearranging furniture while people are still sitting on it and the furniture is on a mountain.
- Bad shard key: Choose a column where low cardinality exists and you are screwed.
- Complexity: More servers, more things to break at 3 a.m.
Don’t shard until you must. Try indexing, caching, read replicas, and a bigger server first. Sharding is like moving from an apartment to the same apartment but each one in a different city, amazing when you’ve outgrown a single apartment, but ridiculous if you just own a mixer.
