Messaging Queue Details

RabbitMQ & Kafka Both

A Message Queue is a communication method used in software systems where messages are stored in a queue and later processed by a consumer. It is a form of asynchronous communication, which means the sender (producer) and receiver (consumer) do not need to be directly connected at the same time.

Key Components

1) Producer:

• This is the component (could be an app, service, or system) that creates messages and sends them to the queue.
• Also known as Publisher
• The producer doesn’t need to wait for the consumer to process the message; it just sends the message to the queue.

2) Queue:

• The queue is like a waiting line where the messages are stored temporarily.
• The middleware that receives messages from producers and directs them to the appropriate consumers.
• Messages are processed in the order they arrive (FIFO: First In, First Out). The first message to enter the queue will be the first one to be processed.

3) Consumer:

• This is the component that receives the messages from the queue and processes them.
• Also known as Subscriber.
• The consumer retrieves the messages one at a time and does something with them (e.g., performs some calculations, stores them in a database, etc.).

Why Message Queue is Needed?

Asynchronous Communication:

• In many systems, you may not want to block one part of your application while waiting for another part to complete. Message queues allow asynchronous communication, where a producer (sender) sends a message to the queue and moves on to other tasks without waiting for the consumer (receiver) to process it.
• This improves system performance and responsiveness.

Decoupling of Components:

• In complex systems, different components (e.g., producer, consumer) should work independently. With message queues, the producer doesn’t need to know who or when the consumer is going to process the message.

Reliability and Persistence:

• If the consumer is down or slow, messages can accumulate in the queue and be processed later when the consumer is ready. This ensures that no messages are lost, even during system failures or delays.
• The message queue can store messages persistently (even across system restarts) to guarantee delivery.

Load Balancing:

• Multiple consumers can pull messages from the same queue. This allows you to distribute the processing load among multiple workers, making your system more efficient and scalable.

Scaling and Handling Traffic Spikes:

• When there is a sudden surge in traffic, the message queue can help manage the excess load. It stores the messages temporarily, allowing the system to process them as it can handle the load. This helps prevent system overloads.

Real-World Example:

Imagine an e-commerce website:

• Producer (Order Service): When a customer places an order, the system sends an “order placed” message to the message queue.
• Queue: The queue holds the message until the order is processed.
• Consumer (Inventory Service): The inventory system picks up the order from the queue and reduces the stock count, processes shipping, etc.

This system ensures that the order service doesn’t need to wait for the inventory system to complete its task. It simply sends the message and continues working, while the inventory service processes the orders asynchronously.

Now, What is point to point and pub-sub message type?

Point-to-Point (P2P) and Publish-Subscribe (Pub-Sub) are two common messaging patterns used in message queuing systems. They define how messages are sent and received between producers and consumers.

Let’s go over each type in detail:

1. Point-to-Point (P2P) Messaging:

• In the Point-to-Point model, one producer sends a message to one consumer through a message queue.
• The message queue acts as a buffer where messages are temporarily stored until a consumer retrieves and processes them.
• Delivered to one consumer only.

Characteristics of Point-to-Point (P2P):

• Single Consumer: A message sent to a queue is consumed by only one consumer. If multiple consumers are listening to the same queue, each message will be delivered to only one of them.
• Message Processing: The consumer processes the message and then the message is removed from the queue.
• FIFO (First In First Out): Messages are typically processed in the order they are placed in the queue. The consumer picks up messages in the order they arrive.
• Load Balancing: Multiple consumers can work on the same queue, but each message is processed only once by one consumer, providing some level of load distribution.

Example Scenario (P2P):

Imagine an Order Processing System:

• The Order Service (Producer) sends a message with an order to a queue.
• The Inventory Service (Consumer) picks up the order from the queue, processes it (reduces stock, prepares shipping), and deletes it from the queue once completed.
• The message is processed by one Inventory Service instance only.

2. Publish-Subscribe (Pub-Sub) Messaging:

• In the Publish-Subscribe model, one producer (publisher) sends messages to multiple consumers (subscribers). All subscribers receive the message and can process it independently.
• The messages are published to a topic, and any subscriber that is subscribed to that topic will receive the message.

Here, In the Publish-Subscribe (Pub-Sub) model, an exchange is a key component in the messaging system.

Get Mayesha Mariza’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

An exchange is responsible for receiving messages from the producer (or publisher) and routing those messages to the appropriate queue(s), which then deliver them to the consumers (or subscribers). The exchange determines how messages are distributed and which queues they should be forwarded to, based on the configuration or binding rules.

Characteristics of Publish-Subscribe (Pub-Sub):

• Multiple Consumers: A message sent by the producer is delivered to all consumers who have subscribed to that topic. This is in contrast to the P2P model, where the message is delivered to only one consumer.
• Message Distribution: The publisher does not know about the subscribers, and the subscribers do not know about each other. They are only aware of the topic they are subscribing to.
• Event Broadcasting: This model is ideal for event-driven architectures where multiple systems need to react to the same event.
• No FIFO Guarantee: Since multiple consumers can receive the message simultaneously, the order of message delivery may not always be guaranteed.

Example Scenario (Pub-Sub):

Imagine a Weather Notification System:

• The Weather Service (Publisher) publishes weather updates (e.g., “Rain expected tomorrow”) to a Weather Updates Topic.
• Multiple Consumers (e.g., Mobile App, Website, Notification Service) are subscribed to the Weather Updates Topic.
• Each consumer gets the message and performs its action (e.g., display the weather update on the app, send an SMS alert to users).

When to Use P2P vs Pub-Sub?

• Point-to-Point is ideal for task-based scenarios where one consumer needs to handle each message, such as order processing, job queues, or task scheduling.
• Publish-Subscribe is best when you need to broadcast messages to multiple consumers, such as sending notifications, event logging, or streaming data.

How messaging queue works?

Now we will discuss about two types of message queues like Kafka and RabbitMQ.

Apache Kafka:

Kafka is a distributed event streaming platform, and it works as a message queue, but it is designed to handle high throughput and real-time data streaming. It was originally created by LinkedIn and later open-sourced as an Apache project.

Kafka high level architecture

Key Components of Kafka:

1. Producer:

• A Producer in Kafka is any application or service that sends messages (or records) to a Topic. The producer is responsible for producing data, which is then written to Kafka for processing by consumers.
• Producers don’t need to worry about how the data will be consumed or who will consume it; their job is simply to push data into Kafka.
• A Producer can send messages to a specific partition of a topic, which can help with load balancing and distributing messages across brokers in the Kafka cluster.

2. Broker:

Broker is nothing but a kafka server. Broker has Topic.

• Each broker manages a subset of topic partitions, and when a producer sends data to a topic, Kafka determines which broker (and partition) the message should go to.
• A Broker has a unique ID, and Kafka clusters typically have multiple brokers to distribute load and provide redundancy.

3. Topic:

• A Topic in Kafka is a logical channel where producers send messages and consumers read them from. It is essentially a category or feed name to which messages are published.
• Kafka topics are partitioned, meaning that each topic can have multiple partitions, which allows Kafka to scale horizontally across multiple brokers (servers).
• Producers publish messages to a topic, and consumers subscribe to one or more topics to consume those messages.

4. Partition:

• A Partition is a division of a Kafka topic. It allows Kafka to scale by splitting topics into multiple partitions that can be stored across multiple brokers in the Kafka cluster.
• Each partition is an ordered, immutable sequence of messages. Each message in a partition has an associated offset, which acts as a unique identifier for the message.
• Partitions enable parallelism because multiple consumers in a consumer group can read from different partitions of the same topic simultaneously, increasing throughput and scalability.

within a same consumer group , they always read different partition of a particular topic

5. Offset:

• An Offset is a unique identifier for each message within a partition. It’s a sequential number that allows consumers to track their position within a partition.
• When a consumer reads a message, it remembers the offset of the last message it successfully processed. Kafka doesn’t automatically delete messages from a partition unless the retention period or disk space limits are exceeded, so consumers can rewind or replay messages by seeking to a specific offset.
• Offsets can be managed automatically by Kafka, or consumers can manage them manually. In the case of automatic offset management, Kafka keeps track of the last consumed offset for each consumer group.

6. Cluster:

• A Kafka Cluster is a set of Kafka brokers that work together to store and process data. A Kafka cluster can consist of one or more brokers.
• The Cluster also handles balancing the load, as the number of brokers can be increased or decreased based on demand.
• Kafka ensures that if a broker goes down, the remaining brokers in the cluster will continue to handle requests, and data can still be served from other replicas.

Kafka cluster

7. Consumer:

• A Consumer is any application or service that reads messages from a Kafka Topic.
• Kafka consumers pull data from topics (not pushed like in some other systems). They can retrieve messages from one or more topics at their own pace.
• A Consumer can process messages in the order they appear in the topic partitions, but it’s the consumer’s responsibility to manage offsets to know where it left off.

8. Consumer Group:

• A Consumer Group is a group of consumers working together to consume messages from one or more Kafka topics.
• Each consumer in a group processes a subset of partitions of the topic(s) they are subscribed to. The key feature of consumer groups is that each message is processed by only one consumer in the group.
• Kafka guarantees that each message in a partition will be consumed by only one member of a consumer group.
• If there are more consumers than partitions, some consumers will remain idle.

9. Zookeeper:

Different brokers how will they interact with each others, something called zookeeper. Zookeeper helps them(brokers) to interact with each other. All the brokers interact will zookeeper first.

How does Kafka Work?

1. Producer Sends Messages to Kafka Topics:

• The producer publishes messages (or records) to Kafka topics.
• Each message is sent to a specific partition within a topic. Kafka uses round-robin or custom partitioning logic to determine which partition the message should go to.
• Producers do not wait for the consumer to process the message. Kafka is asynchronous, so the producer can send messages and continue with other tasks.

2. Kafka Stores Messages in Partitions:

• Kafka stores the messages in partitions within topics. Each partition is an ordered where messages are stored in the order they arrive.
• Messages within a partition are indexed by a unique number called offset. This allows consumers to track which messages they have already processed.

3. Consumers Retrieve Messages:

• The consumer subscribes to a Kafka topic and retrieves messages from the partitions of the topic.
• Kafka allows parallel consumption by assigning each consumer in a consumer group to a different partition of the topic.
• Each consumer maintains its offset (the position within the partition). This allows it to keep track of the last message it consumed, so it can resume from that point if it crashes or is restarted.

4. Consumer Groups and Load Balancing:

• A consumer group is a collection of consumers that share the same group ID. Kafka guarantees that each message in a partition is consumed by only one consumer in the group. This ensures that messages are processed in parallel and provides load balancing.
• If there are more consumers than partitions, some consumers will remain idle.
• If there are more partitions than consumers, some consumers will handle multiple partitions.

RabbitMQ:

RabitMQ Architecture

RabbitMQ is an open-source message broker software that facilitates communication between different components in a distributed system through message queuing. It implements the Advanced Message Queuing Protocol (AMQP) and is widely used for managing and routing messages between producers (senders) and consumers (receivers).

Key Concepts of RabbitMQ:

Producer:

• The producer is an application or service that sends messages to a RabbitMQ exchange. The producer does not need to know how or when the message will be consumed, just that the message is sent to the queue.

Consumer:

• The consumer is an application or service that retrieves and processes messages from the RabbitMQ queue. The consumer subscribes to a queue and processes the messages in the order they arrive.

Queue:

• A queue is a buffer that holds messages until they are consumed. RabbitMQ queues store messages temporarily until the consumer retrieves and processes them.
• Queues are FIFO (First In, First Out) by default, meaning that messages are processed in the order they arrive.

Exchange:

• An exchange is responsible for receiving messages from the producer and routing them to one or more queues. It determines how the messages are routed based on specific rules.

Types of exchanges:

1. Direct Exchange: Routes messages with a specific routing key to a queue.

Direct exchange

2) Fanout Exchange: Broadcasts the message to all queues bound to the exchange, ignoring the routing key.

Fanout Exchange

3) Topic Exchange: Routes messages based on patterns(Wild Card like * and #) in the routing key.

Binding:

• A binding is a relationship between an exchange and a queue. It defines the rules for how messages should be routed from an exchange to a queue.

Routing Key:

In some exchange types (e.g., Direct and Topic), the routing key is used to determine where to route the message.

Message Acknowledgments:

• RabbitMQ can ensure message reliability using acknowledgments. A consumer must send an acknowledgment after successfully processing a message. If an acknowledgment is not received, RabbitMQ will requeue the message for retrying by another consumer.

Dead Letter Queue (DLQ):

• If a message cannot be processed by a queue (due to failures, retries, etc.), it can be redirected to a Dead Letter Queue. This allows the system to handle failed messages separately for future inspection or reprocessing.

Publisher Confirms and Consumer Acknowledgments:

• Publisher confirms provide assurance that a message was successfully received by the broker.
• Consumer acknowledgments confirm that a message was processed successfully.

How RabbitMQ Works?

1. Producer Sends Messages to Exchange:

• The producer sends a message to an exchange, specifying the routing key (if needed) and the message content.

2) Exchange Routes the Message:

• The exchange uses the routing key and the type of exchange (direct, fanout, topic, or headers) to determine how to route the message to one or more queues.

3) Message Stored in Queue:

• The message is routed to a queue based on the binding rules between the exchange and the queue. If the queue is full, the message may be dropped or delayed, depending on configuration.

4) Consumer Retrieves and Processes the Message:

• The consumer retrieves the message from the queue and processes it. Once the message is processed, the consumer sends an acknowledgment to RabbitMQ, signaling that the message was successfully handled.

5) Message Deletion:

• After the message is acknowledged by the consumer, it is removed from the queue. If there is no acknowledgment, RabbitMQ may requeue the message for retry or place it in a dead letter queue.

Summary:

• Kafka is optimized for high-throughput, distributed event streaming, and large-scale message processing.
• RabbitMQ is more suited for reliable message delivery, task queueing, and inter-service communication in smaller to moderate scale systems.