Messeging Queue BreakDown

So Recently i have had a chance to work on messeging queue during building one of my project. And I got to know that messeging queue is much more than i anticipated and its really a dark hoarse.

So today i am going to discuss what messeging queue is, how it works , what happens when something breaks and then how to fix it

Alright so first of all we should understand why do we even need a message queue ?

What problem does it solve ?

Let’s look at one diagram and figure out what’s wrong in that..

Now this is the basic architecture that you can design and here a client is uploading an image to the server and the server is doing three synchronous tasks which takes total of ~6s to complete and once the tasks are completed then only the server returns some response.

But this architecture has few limitations as well like —

1. Latency, a client stares at the spinner(loader) for like ~ 6s just to get the confirmation that all right image is uploaded or if there is a network hicup then that response time might increase even more !
2. The second limitation is that this architecture can break in a lot of places like resize image is done but server crashed during filtering.Then all the work of resizing the image is gone and we have to do that again when the server is up.
3. The third limitation is high throughput, right now there is only one client but what if there are 100000 clients uploading the image on the server and as we know that our server is processing the task synchronously, then that would really overwelhm our server and server will either drop request or through internal server error or might crash in worst case.

The above limitation are the real deal in a production envirnoment so to solve this we can introduce something called as MESSAGE QUEUE.

So how does Queue solve the above problems , now what happens is if a client comes and uploads the image he immedietly gets a response that image is uploaded. And the resizing, filter etc goes to the queue and are processed asynchronoulsy by some worker. Basically the server is not doing any computation on its end it is delegating that to the message queue.

So if thousands of request comes in then server basically delegates them to the message queue and moves on with the new task. In worst case if a message queue is full the response is delayed but no messages are dropped. because there are multiple worker consuming from the queue that is the reason the server is not overwhelmed.

So lets formalize it and understand what is a message queue :

A Message Queue is just a buffer of async tasks that sits between the producer and consumer.

Producer — Producer produce the task in the above example it was the API server that was handling upload. The producer pushes the task in the queue and then forgets it.

Consumer — Consumer is the server the pulls the task from the queue and process them in the above example it was the pool or workers.

So the whole flow goes like this —

Producer pushes the task in the queue and forgets about it , then the queue holds those task until someone is ready to process them and then a consumer pulls the task from the queue and process them and ACKs the queue about task completion.

The key thing to note here is that both consumer and producer are decopuled and that is the key thing here. It helps in indivual scaling of both.

Alright now we know what a message queue is … but how does it work under the hood ???

ACK(acknowledgement) — So let’s say that a worker is currently processing a task and during mid execution the worker crashed for whatever reason’s now if the queue deleted the task after handing it off to the worker then that task is lost cause now the worker is crashed and queue deleted the task … so what to do ?? That is why ACK are used , when a consumer pulls the task from the queue the queue does not immedietly deletes the task, when a worker finshed executing the task it sends an ACK to the queue and then the task is deleted…

But if the task is still on the queue then two workers can pickup the same task and process them right ?

Get Rohit Singh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

That will be wastage of resources and duplicate processing also it could be catastrophic if its related to money…..

So different queues handle it differently for example —

SQS basically makes a task invisible for a configurable amount time until a worker is processing the task and if the ACK is recieved within that duration then the task is deleted otherwise it becomes visible again and gets pickup up by other worker.

Kafka on the other hand follows the mechanism of partition, kafka has seperate partition for seperate worker so the issue of two worker picking the same task does not happen in kafka…

But there is one tricky thing here ….

Now what happens ?

The queue thinks that the task is still not completed and waits for the ACK which the queue is not going to recieve because the worker crashed. So the queue makes the task visible again after the configured duration and new worker picks the task and executes it again….

If its about resizing the image and filter then duplicate processing does not matter at all but…

Think what if task is deduct $500 from accout X , that means accout X will be debited twice which is unacceptable.

This problem in messaging queue is called delivery gaurantees —

At-least Once — Message may be delivered more than once.

At-most Once — Message may never be delivered at all.

Exactly Once — Message will only be delivered once.

In Our Case we are going with At-least Once with Idompotent workers, which will solve our problem that we discussed above.

Idompotent — Processing the same message twice produces the same result.

Alright we got that right….

Now lets discuss one more scenario —

What if we have more producer and less consumers so the producers produces a lot of messages at once which the consumers are not able to handle that means the consumer and cosuming slower that the producer is producing… which could could problem of message lost if the queue becomes full or it reaches its maximum depth…

We solve this with Back Pressure , back pressure means slowing down the producer producing rate to match the consumer consuming pace.

We can also solve this by adding more partitions to the queue or by adding more workers though…

The next problem that might come is — what if a task failed after max attempts .. what will happen to it ?

DLQ — Dead letter queue is a special queue that keeps the failed messages and the messages are later inspected by the admin and are discarded or replayed.

Alright now the main problem is — What if the Queue goes down??

That means the producer won’t be able to produce and the consumer won’t be able to consume and the messages in the queue are all gone….

During my own project i had used redis as a queue and redis by default does not persist the data so to presit the data we can use its feature such as AOF or RDS.

And we can also use a replica or queue which is always at stand by and takes over if the main queue goes down….

That was it about the message queue.