Your Database has Another Database

Good design adds value faster than it adds cost ~ Thomas C. Gale

Wait what ? What are you even saying, how can my database have another database when I have not assigned any database to it ? What is it actually then ?

Well this is what we are going to talk about here that what happens if we have not assigned it necessarily.

But let us first talk about why is it even having a backup database important, is out primary database not good enough ? Well the thing is our primary database could go very wrong in almost all ways, it can crash, it can be late, there may be conflicts and what not. So yes some of this reasons could be our “primary” reason for having a secondary database.

Firstly Why do we need Replication of Databases ?

Let’s think of it in a simple way, we have one database which is very simple and beautiful right ? But suddenly we get 100000+ reads/sec or lets say 10000+writes/sec. Now even the database is worried that what is even happening. So this leads to few problems like we may reach 100 % database downtime which is not a good thing.

So due to problems like this we decide to go for database recovery and thus we turn our focus to replication. And yes this is not that simple we will definitely run into issues when even we use replication.

So what are Database Replication

Replication is the process of maintaining multiple copies of database across a multiple nodes.

Instead of physically shipping the database to every replica, databases commonly replicate using logs.

So to see what things look like using WAL.

So a very important thing that you might think here is what are Leaders ? From where has leaders come into out topic. What is happening ?

Well let’s talk about Leader than now. Yes actually we are going to talk about Leaders. And just to make it sound more “funny” we will also talk about followers here.

Leaders and Followers

Let’s first see how the thing looks like, then maybe we can talk more about it.

Yes looks very much similar to our previous picture right ?

So basically what happens is that leaders receive the writes, then followers maintain the leaders data. But it is not always “ instantaneous” or we can say not always “ synchronous”.

But we will talk about synchronous and asynchronous replication some time later. First let’s talk more about how leaders and followers work.

So basically the Leader is the primary database and the Followers keep the copies of the data. Yes “ REPLICATION”.

So when a user wants to write something it is usually like

1. The user sends write request to the leader
2. The leader saves changes locally
3. The leader sends those changes to a replication log
4. Followers receive the data and apply changes

Now even the Leaders are of many types but before talking about that let’s talk about how replication may not be “synchronous”.

Synchronous and Asynchronous

So think of it like this if you are uploading a profile picture and have clicked Save. So the backend receives and then sends the request to the database for it to be updated. Now this is fine right ? There should not be any issues right ?

But no the main issue that comes up is that the database itself thinks of it like

Whether it would wait for the replicas to come or just inform the user that the write is done. It leads to latency, lag, durability and tons of other issues.

Let us understand it simply so imagine that there is a database with one leader and two followers all right. Now what happens is that the leader handles writes, followers follows the changes done by the leader and the user updates profile but the thing is that, the leader has to wait for both followers to be done with there replication works which leads to the replication methods.

Synchronous replication

So in Synchronous replication the leader waits for the replica or the followers to confirm the status before the leader can confirm about it to the user.

So every follower basically has to tell the leader that there part is done or not.

It is very much preferred because of durability. If the replication is synchronous and the required follower has acknowledged it, then we get another copy of the committed data.

So even if now the leader crashes, we can easily promote the synchronized follower, which now become the new leader.

So when the write is acknowledged, the required replica also gets to know about it.

But there is definitely an issue with it. What happens if the synchronized follower is unavailable then the leader keeps waiting for the follower and the user does not get the data. But to deal with such issues we can just make every follower synchronous right ? It does solve the major issues right ? like let’s just give both the followers as the synchronous followers it becomes very much efficient right ? But well no, that is not the thing happening here. Think what will happen if the second follower crashes then suddenly all the writes will be very much blocked what to do then. If one replica is having issues our entire database will have issues.

So how do we deal with such issues then ?

So what is Semi Synchronous Replication and how does that solve our issue here. Well let’s first again see how it looks like

So here the leader requires one follower to acknowledge the write then the others can do the work asynchronously.

So basically Asynchronous Replication the leader does not have to wait for followers before acknowledging the write.

It does not have to wait for the followers to confirm, which is why Asynchronous Replication so much faster.

Get Sankalp’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

But does this all make replication not have any issues, well no, in fact replication has lots of issues. Let us talk about them here.

Replication Issues

1. Replication lag the changes on the primary or the leader databases may take time to reach the replicas but withing that time the user may read the data after writing it.
2. Data inconsistency if the replication fails or conflicts occur then the different database nodes can contain different version of the data.

And there are many other issues also. But to consistently deal with this issues we just have to work in between consistency, availability, performance. A system can’t have all three at the same time, so we decide to just give certain system certain features at a time and don’t give every feature to the same system.

Now let us talk about another important issue or like maybe a feature

How do we create a New Follower

We can think setting up new followers might be an easy thing like you may just take the leaders data, copy it to the new node, and let replication do the rest then copy it to the new node and then replication solves the issues.

But replication or rather databases here are not that much supportive for us for that matter. Yeah it’s quite disappointing.

Why does that not work ?

Think of it like you have two databases one for the users and one for the orders.
If you copy details from the the first database at a certain time, then again in the second database you start copying details from the second database at another time. By that time more payments will be processed. Which is causing problems for the database as it is experiencing two things simultaneously which leads it to not have any valid point at a certain time.

So let’s not think of any other way and talk about a very much useful and simple way to solve this issue.

Snapshot and Logs

So what we are doing is we combine two things together.

Consistent Snapshot + Changes since Snapshot = Current database state

Now let’s talk about another very important thing. Yes now we will talk about

Multi Leader Replication

So till now what happens is that there is only one leader and all the writes goes through it. But we can clearly think that what will happen if we can’t reach the leader ?

So then we think about a way to solve it, we will use multi leader replication

So what happens basically here is that every leader also acts as a follower of the other leaders a write perform with leader A gets replicated to leader B and a write written in leader B gets replicated to leader A.

The changes are replicated asynchronously between the centers also.

The user don’t have to wait for for anyone else to agree before getting the results.

It makes writes local so after every place there is a write the user does not have to wait for the full completion of the data.

It can also deal with center outages like if the center of place A goes then only that place does not accept writes or replications but place B has center going on so we can easily deal with it and when the place A returns then the replications also happen. But this does not happen in Single Leader Replication.

Surprisingly Multi Leader Replication has a very good usage for us like our own devices act as a local database and on connection it syncs the data.

Leaderless Replication

So till now we have talked about Single leader and Multi leader replications and both of which are extremely useful but may also cause issues. Now let’s talk about what will happen if there is no leader. Sounds weird right ?

Like what do you even mean that there would be no leader, how would replications work then like whom would the followers follow even.

So that is what we will talk about now but before that let’s make this very clear to ourselves that whatever the replication method some sets of nodes will definitely be responsible for deciding the order in which writes should be applied.

But again let’s think that why do we even need a leader ? Like there should be no problems if we don’t have a leader right ?

In a Leaderless Replication there is no single replica that owns the write paths, so any replica can accept writes from the user, the user can send the writes directly to the replicas or it can send to a single node which can forward those writes to the required replicas but that node is not the Leader.

Who controls the things then ?

So what happens basically here is that multiple replicas can process the writes independently. Yes that is so very democratic right ?

But who settles the debates now ? Who decides like what values should stay when being updated.

Well to say the least Leaderless replications are very good in fault tolerance and availability. What this means is that even if one replica goes out the other replicas don’t have to wait for them instead they can easily just continue accepting requests.

But how do they do it ? What do they use ?

Quorums

Quorums come in and save us from the same old issue of machines crashing or any other issues.

So what does it basically do ?

Let’s think of in this way that what happens if we have three replicas and we are trying to write a data to it we might think that we need all the replicas to confirm before we can confirm to the user. Well but no if any of the two replicas have accepted the write then one replica can be stale which brings us to say that if we read from two replicas then one of them does contain the latest value.

n=total number of replicas

w=number of replicas that must confirm a write

r=number of replicas we query during a read

so the condition becomes

w+r >n

Sloppy Quorums and Hinted Handoff

What happens here is basically that let’s think like we have three replicas but the issue is that the user has access to only two replicas which are different from the replicas we have. Yes it’s very tense to think about, what do we do now ?

But Sloppy Quorums themselves don’t really provide us the access to consistency availability and latency together we can’t have all three together.

Concurrent writes, partial failures, stale replicas can all cause issues.

So the Dynamo databases usually contain eventual consistency rather than promising that every read will return the latest values we lead to eventual consistency .

Yes that is it for this one. Thanks for reading folks. You guys are very awesome.

Thank you. Hope you had a good read 😊