MQTT for Backend Engineers: Understanding the Protocol Before Writing Code (Part 1 of 2)

Photo by Jakub Żerdzicki on Unsplash

Most backend engineers learn HTTP first and MQTT later, if at all. That order makes sense for web development, but it creates a blind spot when you move into IoT, telemetry, or event-driven systems. You start expecting request-response semantics from a protocol that was never built for them, and things stop making sense.

This article is Part 1 of a two-part series on MQTT. It covers the protocol itself: what problem it solves, how publish/subscribe works, what a broker actually does, and how topics, wildcards, and QoS levels behave. Part 2 takes this theory and builds a real Spring Boot project on top of AWS IoT Core, with working code, a full project setup, and a step-by-step simulation you can run on your own machine.

If you already know MQTT well, you can skip to Part 2. But if terms like QoS 1 or wildcard subscription feel fuzzy, stay here first. The code in Part 2 will make a lot more sense once the mental model is in place.

What MQTT Actually Is

MQTT stands for Message Queuing Telemetry Transport. It is a lightweight, binary messaging protocol built for machines to exchange small amounts of data over networks that are slow, expensive, or unreliable.

It was designed in the late 1990s by IBM engineers working on satellite pipeline monitoring. The constraint was brutal: sensors sat in remote locations and had to send readings over satellite links where every byte cost money and connections dropped constantly. A protocol like HTTP, with its verbose headers and repeated handshakes, was not an option. MQTT was built specifically to survive that environment.

Today, those original constraints (low bandwidth, unreliable networks, resource-starved devices) describe most of the IoT world. That is why MQTT became the default choice for IoT communication, decades after it was first written.

Why Not Just Use HTTP

To understand MQTT, it helps to be specific about what HTTP does badly in this context.

HTTP was designed around a browser asking a server for a document. The browser initiates, the server responds, and the connection ends. That pattern breaks down for machine-to-machine telemetry for a few concrete reasons:

1. A temperature sensor does not wait for someone to ask it a question. It produces readings continuously, whether anyone is listening or not.
2. Every HTTP request carries overhead in headers, TLS handshakes, and connection teardown that is often larger than the actual sensor reading.
3. If hundreds of sensors all try to push data to one HTTP endpoint at once, that endpoint has to juggle hundreds of separate connections.
4. HTTP has no native way for a server to push data to a client that is already connected and waiting.

MQTT flips the model around. A device connects once and stays connected. When it has something to say, it sends a small message. Anyone interested receives it automatically, with no polling involved. The overhead on a minimal MQTT message can be as small as 2 bytes, which is not an exaggeration you often hear about HTTP.

The Publish/Subscribe Model

HTTP is point-to-point. Client A talks to Server B directly. Server B has to be reachable and listening right now for the exchange to work.

MQTT works differently. It uses a publish/subscribe model with a broker sitting in the middle, decoupling the sender from the receiver.

Here is the mechanics of it:

1. A publisher sends a message to a named destination called a topic.
2. The publisher has no idea who, if anyone, is listening.
3. Subscribers declare interest in one or more topics.
4. The broker takes incoming published messages and forwards copies to every subscriber whose interest matches.
5. Publisher and subscriber never talk to each other directly. Neither one needs to know the other exists.

Picture a temperature sensor publishing readings to home/living-room/temperature. A display app, a logging service, and an alerting system can all subscribe to that same topic independently, without the sensor code ever changing. Add a new consumer, and nothing on the publishing side needs to know.

The Role of the Broker

The broker sits at the center of every MQTT deployment. Its job is narrow but essential:

• Accept connections from clients.
• Authenticate clients where security is configured.
• Receive published messages.
• Route each message to every subscriber whose topic filter matches.
• Enforce access control rules if any are set up.

Notice what is missing from that list: business logic. The broker does not transform messages, inspect payloads, or make decisions about content. It routes bytes based on topic matching, nothing more.

In the project built in Part 2, the broker is AWS IoT Core. Instead of standing up and maintaining your own broker software, you use AWS’s managed service, which behaves as a fully managed MQTT broker while also plugging into the rest of the AWS ecosystem.

Clients: Publishers, Subscribers, or Both

Anything that connects to an MQTT broker is a client. A client can act as:

• A publisher, sending messages out.
• A subscriber, receiving messages in.
• Both at the same time.

That last point matters more than it sounds. The same application can publish to a topic and also be subscribed to that exact topic. In Part 2, the Spring Boot application does precisely that: it publishes a message when an HTTP request comes in, and it has also subscribed to the topic it just published to. The message leaves the app, passes through AWS IoT Core, and comes right back to the same app.

Get Aadarsh Pandey’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Every client also needs a unique Client ID so the broker can tell connections apart. In the reference project, this value RoomTempMonitor.

Topics and How They Work

Topics are just strings that act as addresses for messages. They are hierarchical, separated by forward slashes, similar in shape to a file path:

RoomTempMonitor/kitchen

RoomTempMonitor/bedroom

home/floor2/room3/temperature

Topics are not registered in advance. There is no admin panel where you create a topic before using it. A topic comes into existence the instant a message is published to it, and it stops mattering the moment nothing is publishing or subscribing to it anymore.

Subscribers can use wildcards to match multiple topics in a single subscription:

1. +matches exactly one level. RoomTempMonitor/+ matches RoomTempMonitor/kitchenand RoomTempMonitor/bedroom, but not RoomTempMonitor/floor2/kitchen.
2. # matches any number of levels from that point onward. RoomTempMonitor/# matches everything published anywhere under RoomTempMonitor.

The reference project uses the # wildcard. On startup, it subscribes to RoomTempMonitor/#, so it receives messages from any room, kitchen, bedroom, living room, or anything else, all through one subscription.

Quality of Service (QoS) Levels

MQTT defines three QoS levels. They control how reliably a message travels between a client and the broker, and each one comes with a tradeoff between speed and guarantees.

QoS 0, At Most Once.

The message goes out once with no acknowledgment at all. If the network drops the packet, it is gone and nobody retries. This fits data that arrives so frequently that losing one reading barely matters, like a sensor firing every 100 milliseconds.

QoS 1, At Least Once.

The sender keeps retransmitting until the broker acknowledges receipt. This guarantees the message arrives, but it also means duplicates are possible if an acknowledgment gets lost on the way back. Applications receiving QoS 1 messages need to be built to tolerate duplicates. The reference project defaults to QoS 1 for telemetry.

QoS 2, Exactly Once.

A four-step handshake makes sure the message arrives exactly once, no duplicates, no drops. It is the most reliable option and also the most expensive in network overhead. It suits cases like financial transactions or control commands, where a duplicate could cause real harm.

The reference project exposes QoS as a field in the HTTP request body. If the caller does not set it, the service falls back to QoS 1.

The MQTT Communication Lifecycle

A typical MQTT session moves through these steps:

1. Connect. The client opens a TCP connection, or a WebSocket tunnel in the case of the project in Part 2, and sends a CONNECT packet with its Client ID, optional credentials, and session preferences.
2. Session establishment. The broker replies with a CONNACK packet, confirming success or reporting failure.
3. Subscribe. The client sends SUBSCRIBE packets naming topic filters and the QoS level it wants for each.
4. Publish. Publishers send PUBLISH packets containing the topic, the payload, the QoS level, and a retain flag.
5. Message delivery. The broker forwards matching messages to every subscriber.
6. Disconnect. Either side sends a DISCONNECT packet and closes the connection.

The detail that trips up people coming from HTTP: steps 1 through 6 span the entire life of the application, not one request. The connection opens once at startup and stays open until shutdown. Thousands of messages can flow over that single connection, none of them opening a new one.

Why MQTT Fits IoT So Well

Put all of this together and the fit with IoT becomes obvious:

• Constrained devices with only kilobytes of RAM can run a minimal MQTT client. An HTTP stack needs far more resources than that.
• Unreliable networks are handled through QoS levels and session persistence, both built with packet loss and reconnection in mind from the start.
• Many-to-many communication maps naturally onto publish/subscribe: many sensors, many consumers, all loosely connected through one broker.
• Real-time push means subscribers get data as it happens instead of polling on a fixed schedule.
• Low overhead means a minimal PUBLISH packet can weigh only a few bytes, while HTTP headers alone often outweigh that many times over.

A Note on Gaps in This Explanation

A few things are worth flagging honestly rather than glossing over.

This article focuses on the core protocol concepts relevant to the project in Part 2 (publish/subscribe, brokers, topics, QoS, and the connection lifecycle).

It does not cover MQTT’s last will and testament feature, retained messages in depth, MQTT 5.0-specific features like shared subscriptions and message expiry, or the wire-level packet format in byte-by-byte detail.

If you want to go deeper into the specification itself, the official MQTT.org site and the OASIS MQTT 5.0 standard are the right next stop.

None of those gaps affect your ability to follow Part 2, but they are gaps nonetheless, and it would not be honest to pretend otherwise.

What Comes Next

With the concepts in place, Part 2 walks through an actual Spring Boot application that publishes telemetry to AWS IoT Core, receives it back through a subscription, and demonstrates the full round trip. It includes the complete project setup, every file explained, a running demo you can execute with curl, and a full simulation of the message traveling from your terminal to AWS IoT Core and back.

Continue to Part 2: MQTT with Spring Boot and AWS IoT Core (a hands-on project).