Message Queue and Task Queue: When to use them?

While working with tools like Celery, Redis, RabbitMQ, Amazon SQS, and Google Cloud Tasks, you’ve probably came across terms like task queue, job queue, message queue, and message broker. If you’re a beginner, these terms can feel confusing as many posts on the internet uses them interchangeably but these are not the same. Each of them solves a different problem, and using them without understand what they does would be waste of resources.

In this blog, we’ll start from the basics and gradually build up to real systems. We’ll understand what each of these concepts actually does, how they differ, and when you should use them. We’ll also map them to design patterns like Producer Consumer and Pub/Sub, so you can see how everything fits together.

Introduction

Whenever you visit Celery website you see “A simple distributed Task Queueing service”, when you read a bit of docs then you see “It works with message broking services like Redis Queue, RabbitMQ, Amazon SQS”. When you visit RabbitMQ site you see “It is a messaging and streaming broker”. Similarly when you visit Redis site you see “It is a message queueing and message broking service”. Let’s first understand what message queue, task queue and message brokers are. Then we’ll discuss it further.

If you are not familiar with producer consumer pattern and pub/sub. Read it first click here

Message Queues

Many articles and posts on the internet uses message queues and task queues interchangeably but there are many differences between both of them.

Message: A message can be a request, reply , error or just plain text information.

There could be multiple producers and consumers sharing the same queue.

A message queue is a temporary storage which keeps the message sent by the producer until any consumer picks it for execution. Many producers and consumers can use this queue but only a single consumer can read a message. For this reason we call it a one to one or point to point communication. When we need to process the message with multiple consumers we need to use pub/sub pattern alongside this.

Some message queues have the feature of resending the messages if success acknowledgement is not received.

Message Brokers

While message queues perform the functionality of queueing messages and retry mechanisms perfectly. With scale you need to add multiple queues and managing each of them individually becomes very complex. The simplest solution for this would be to keep all the message queues inside a centralised system with logic to manage all those queues inside that system.

Message Broker does this. It is a kind of middleman which takes the messages from Producer and stores them temporarily in its respective queue. The consumers continuously poll the Broker for tasks and fetches the task from queue. After successful execution Consumer sends an acknowledgement for the task.

Features of Message Brokers

1. Messages can be routed based on routing rules (e.g. round robin) , headers and other attributes.
2. It can act as a translator between multiple protocols.
3. It provides message persistence and retry mechanisms until it is successfully acknowledged
4. Messages can be monitored and analysed.

How Message Broker Works?

1. Producers send the messages to the broker. In some brokers like RabbitMQ, the messages are received by a component called an exchange which is used an entry point inside the broker. The exchange routes messages to multiple Queues based on routing rules. Some brokers like Redis directly pushes messages inside data structures like List having LPUSH and RPOP operations without separate routing rules.
2. Consumers then either poll(uses short polling or long polling) for messages or just simply receive messages via the broker and executes them.
3. If execution is successful, then Consumer sends an acknowledgement for success and pops the queue. If it fails the Queue does not pop the message and Consumer process it again until it reaches the retry limit.
4. In simple Redis implementation, once the message is fetched by the Consumer, it is immediately removed and there is no inbuilt acknowledgement or retry mechanism.

Task Queues (Job Queues)

A Task Queue is a system that stores task and execute them using workers. Similar to Message Queue it temporarily holds data but instead of storing just messages it stores jobs where set of execution steps are defined. It can be a function calling or api call or database queries. The Task queue holds these jobs until a worker picks and executes it.

In addition to this, a Task Queue can handle retries, errors, scheduling, and coordination between multiple brokers.

Worker : A Worker is responsible for executing the tasks. It is a specific type of consumer that the job queue manages. It is handled by the job queues. In message queues you need to manually define your own consumers.

Get Yash Vaishnav’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Task : A Task or a Job is a generally a description of work that needs to be done. A task may include sending emails, generating reports, image creation, data processing, generating logs and much more. A message is just a piece of information.

Working of Task Queue

How Task Queue Works?

1. The producer sends tasks to the task queue which acts as an entry point where these tasks are stored temporarily until they are executed.
2. Workers continuously check the task queue for new tasks. They either poll the queue or receive tasks directly depending on the architecture. When a task is available, a worker picks it up from the queue and starts execution.
3. The worker reads the task details, maps it to the corresponding function, and executes it independently of the original request. Multiple workers can process tasks in parallel, allowing the system to handle large workloads efficiently.
4. If execution is successful, the task receives success acknowledgement and popped from the queue. If execution fails, the task queue system can retry the task based on retry policies until it reaches retry limit.
5. Task queues also support job scheduling, time based scheduling and tracking of task states such as pending, running, failed, or completed using acknowledgements.

Examples of Task Queues are Celery, APScheduler, Amazon SQS and Google Cloud Task Queues

Is Task Queue used in production?

Yes, Task Queues are heavily used in production. In large scale systems tasks are not executed immediately. They are pushed into a queue, and processed by multiple workers running concurrently. For example a user uploads an image. This upload does not lock the server, instead it queues the image upload request and the user gets “Your upload is in progress”. A worker picks this task and after successful execution he gets “Upload Successful” response.

Task Queues also support Batch Processing. Instead of processing large amount of data once, the system breaks it into multiple smaller tasks and push them to the queue. Workers process these tasks asynchronously. Backing up your data, Transferring your cloud data to another device are some of its examples as they work as background processes.

Can I have multiple concurrent workers in single core system?

You can have multiple workers in single core systems too but instead of working concurrently, your workers use context switching. One worker works for sometime and saves state and other starts working. This way worker switches multiple times and completes execution. This is handled by Operating System. Though it is recommended to not use more than 2 workers per core.

Basic flow of execution using a Task Queue and a Message Broker

Message Queues, Task Queues and Message Brokers are not alternatives of each other. They work together in production. Producers send the jobs to the message broker which stores and deliver them using queues and workers fetch those jobs and executes them. The execution logic, retries, and task handling are managed by the workers or the task queue system

Task Queue is built on top of Message Queue if you are not using Redis. Redis has its own Data Store(primarily uses List Data Structure).

When you should not choose these tools?

1. Task Queue : You should not use a task queue when the operation requires an immediate response such as payments or authentication. It adds useless complexity for such kind of tasks.
2. Message Queue: If your systems deals with tight coupling(components are dependent on each other) or direct communication like api calls is sufficient, then it is not a good job to use queues. Also it is not ideal for immediate responses.
3. Message Broker : If a direct communication with a simple queue (or data store in Redis) is sufficient for your architecture then adding a broker adds a overhead and complexity. Avoid it in such cases.

References:

1. https://docs.celeryq.dev/en/stable/
2. https://www.rabbitmq.com/
3. https://redis.io/solutions/messaging/
4. https://ademcatamak.medium.com/what-is-message-broker-4f6698c73089
5. https://medium.com/must-know-computer-science/system-design-message-queues-245612428a22
6. https://tsh.io/blog/message-broker
7. https://medium.com/basecs/to-queue-or-not-to-queue-2653bcde5b04
8. https://thedataengineer.substack.com/p/optimizing-task-throughput-in-celery