<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Erkan Demirel on Medium]]></title>
        <description><![CDATA[Stories by Erkan Demirel on Medium]]></description>
        <link>https://medium.com/@erkndmrl?source=rss-6349117db536------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3f0Z88fvRF-CUXGI0pPMew@2x.jpeg</url>
            <title>Stories by Erkan Demirel on Medium</title>
            <link>https://medium.com/@erkndmrl?source=rss-6349117db536------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:44:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@erkndmrl/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond Text Generation: Tool Calling with Spring Boot and OpenAI]]></title>
            <link>https://erkndmrl.medium.com/beyond-text-generation-tool-calling-with-spring-boot-and-openai-0dc6b434e46b?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/0dc6b434e46b</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[calling-tool]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 09:38:32 GMT</pubDate>
            <atom:updated>2026-11-10T09:38:32.222Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vyi78OqSh5s83c21Kfdz_w.png" /></figure><p>In my previous project, I explored <a href="https://erkndmrl.medium.com/rag-in-java-building-document-search-with-spring-boot-openai-and-pgvector-f29c7d672c67">RAG with Spring Boot, OpenAI, and pgvector</a>. This time, I wanted to understand another important AI concept: <strong>tool calling</strong>.</p><p>Instead of only generating text, an AI agent can decide when it needs external data and call Java methods to retrieve information or perform an action.</p><p>The complete source code is available on <a href="https://github.com/erkandemirel/spring-boot-openai-tool-calling-agent-demo">GitHub</a>.</p><h4>What Is Tool Calling?</h4><p>A language model does not directly execute Java code.</p><p>We define available tools and their descriptions. The model analyzes the user’s request and decides which tool should be called. Spring AI executes the selected method and sends its result back to the model.</p><p>The flow looks like this:</p><pre>User Request<br>    ↓<br>Spring Boot REST API<br>    ↓<br>OpenAI<br>    ↓<br>Tool Selection<br>    ↓<br>Java Method Execution<br>    ↓<br>Final AI Response</pre><h4>Defining the Tools</h4><p>In this project, I created three tools:</p><ul><li>Get the current order status</li><li>Get the estimated delivery date</li><li>Cancel an order</li></ul><p>Spring AI allows Java methods to be exposed using the @Tool annotation:</p><pre>@Tool(description = &quot;Gets the current status of an order by its order ID&quot;)<br>public String getOrderStatus(<br>        @ToolParam(description = &quot;The unique order ID&quot;) Long orderId) {<br><br>    return orders.getOrDefault(orderId, &quot;ORDER_NOT_FOUND&quot;);<br>}</pre><p>The description is important because the model uses it to understand when the method should be called.</p><p>The cancellation tool also contains a business rule:</p><pre>@Tool(description = &quot;Cancels an order only when its current status is PREPARING&quot;)<br>public String cancelOrder(Long orderId) {<br><br>    String currentStatus = orders.get(orderId);<br><br>    if (!&quot;PREPARING&quot;.equals(currentStatus)) {<br>        return &quot;CANCELLATION_NOT_ALLOWED&quot;;<br>    }<br><br>    orders.put(orderId, &quot;CANCELLED&quot;);<br>    return &quot;ORDER_CANCELLED&quot;;<br>}</pre><p>The AI decides to call the tool, but the application still controls the actual business rules.</p><h4>Connecting the Tools to OpenAI</h4><p>The tools are registered while creating the ChatClient:</p><pre>this.chatClient = chatClientBuilder<br>        .defaultSystem(&quot;&quot;&quot;<br>                You are a helpful order support assistant.<br>                Use the available tools when order data is required.<br>                Call a state-changing tool only when the user<br>                explicitly requests that action.<br>                &quot;&quot;&quot;)<br>        .defaultTools(orderTools)<br>        .defaultAdvisors(<br>                MessageChatMemoryAdvisor.builder(chatMemory).build()<br>        )<br>        .build();</pre><p>The project also uses conversation memory. Requests with the same conversationId share their previous context.</p><p>For example:</p><pre>User: What is the status of order 1002?<br>Agent: Order 1002 is currently PREPARING.<br><br>User: Cancel it.<br>Agent: Order 1002 has been cancelled.</pre><p>The user does not need to repeat the order number because the agent remembers the previous message.</p><h4>Calling the API</h4><p>The application exposes a single endpoint:</p><pre>POST /api/agent/chat</pre><p>Example request:</p><pre>{<br>  &quot;conversationId&quot;: &quot;demo-1&quot;,<br>  &quot;message&quot;: &quot;When will order 1002 be delivered?&quot;<br>}</pre><p>The model can call both the delivery-date and order-status tools, then combine their results:</p><pre>{<br>  &quot;answer&quot;: &quot;Order 1002 is estimated to be delivered on October 20, 2026. Its current status is PREPARING.&quot;<br>}</pre><p>Tool executions are also logged, making the model’s decisions visible:</p><pre>Tool called: getEstimatedDeliveryDate, orderId=1002<br>Tool called: getOrderStatus, orderId=1002</pre><h4>Final Thoughts</h4><p>Tool calling creates a clear boundary between the language model and application logic. The model interprets the request and selects the appropriate operation, while Spring Boot executes the tool and enforces the business rules.</p><p>Although this demo uses in-memory data, the same approach can be applied to databases, internal APIs, and microservices. For production use, state-changing tools should also include authentication, authorization, confirmation, and audit logging.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0dc6b434e46b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG in Java: Building Document Search with Spring Boot, OpenAI and pgvector]]></title>
            <link>https://erkndmrl.medium.com/rag-in-java-building-document-search-with-spring-boot-openai-and-pgvector-f29c7d672c67?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/f29c7d672c67</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 08:59:53 GMT</pubDate>
            <atom:updated>2026-10-30T08:59:53.479Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FXhX4MehkJa7fTGd2x1GXw.png" /></figure><p>AI and LLM applications are a new area for me. Since my background is mostly Java, Spring Boot, and microservices, I wanted to learn these concepts by building a small project with technologies I already use.</p><p>For this project, I chose RAG because it demonstrates an important real-world use case: answering questions using application-specific documents instead of relying only on the model’s existing knowledge.</p><p>The application stores documents in PostgreSQL using pgvector, finds the most relevant content for a question, and sends that content to OpenAI to generate an answer.</p><h4>What Is RAG?</h4><p>RAG stands for <strong>Retrieval-Augmented Generation</strong>.</p><p>A language model does not automatically know our private documents or business data. With RAG, we first retrieve relevant information from our own data and then provide it to the language model as context.</p><p>The flow in this project looks like this:</p><pre>Add document<br>→ Split the text into chunks<br>→ Generate embeddings<br>→ Store them in PostgreSQL<br><br>Ask a question<br>→ Generate an embedding for the question<br>→ Find similar document chunks<br>→ Send the context and question to OpenAI<br>→ Return the answer and its sources</pre><p>The language model does not connect directly to the database. Spring Boot performs the search and sends only the relevant content to the model.</p><h4>Project Technologies</h4><p>The project uses:</p><ul><li>Java 21</li><li>Spring Boot</li><li>Spring AI</li><li>OpenAI</li><li>PostgreSQL</li><li>pgvector</li><li>Docker Compose</li></ul><p>Spring AI provides the integrations for chat models, embedding models, and vector stores.</p><h4>Adding a Document</h4><p>Documents can be added through the following endpoint:</p><pre>POST /api/documents</pre><p>Example request:</p><pre>{<br>  &quot;source&quot;: &quot;return-policy&quot;,<br>  &quot;title&quot;: &quot;Return Policy&quot;,<br>  &quot;content&quot;: &quot;Customers have 30 days to return unopened products with the original receipt.&quot;<br>}</pre><p>The source field identifies the document, while title provides a readable name.</p><p>The content is converted into a Spring AI Document and split into smaller chunks:</p><pre>Document document = new Document(<br>    request.content(),<br>    Map.of(<br>        &quot;source&quot;, request.source(),<br>        &quot;title&quot;, request.title()<br>    )<br>);<br><br>List&lt;Document&gt; chunks = textSplitter.apply(List.of(document));<br>vectorStore.add(chunks);</pre><p>Splitting is important for large documents. Instead of retrieving an entire document, the application can find only the sections related to the question.</p><p>Each chunk is converted into an embedding and stored in PostgreSQL.</p><h4>Embeddings and pgvector</h4><p>An embedding is a numerical representation of a text’s meaning.</p><p>In this project, the text-embedding-3-small model converts each document chunk into a vector containing 1536 numbers.</p><p>Texts with similar meanings produce vectors that are closer to each other. PostgreSQL uses the pgvector extension to store and compare these vectors.</p><p>The vector store configuration is defined in application.yml:</p><pre>spring:<br>  ai:<br>    vectorstore:<br>      pgvector:<br>        initialize-schema: true<br>        index-type: HNSW<br>        distance-type: COSINE_DISTANCE<br>        dimensions: 1536</pre><p>Spring AI automatically creates and configures the PgVectorStore using the PostgreSQL datasource.</p><h4>Asking a Question</h4><p>Questions are sent through another endpoint:</p><pre>POST /api/questions</pre><p>Example:</p><pre>{<br>  &quot;question&quot;: &quot;How many days do customers have to return a product?&quot;<br>}</pre><p>The application first searches for relevant document chunks:</p><pre>List&lt;Document&gt; documents = vectorStore.similaritySearch(<br>    SearchRequest.builder()<br>        .query(question)<br>        .topK(4)<br>        .similarityThreshold(0.35)<br>        .build()<br>);</pre><p>Here:</p><ul><li>query contains the user’s question.</li><li>topK(4) returns at most four relevant chunks.</li><li>similarityThreshold(0.35) filters out weak matches.</li></ul><p>The threshold is not a fixed value for every project. It should be adjusted according to the model, documents, and expected search results.</p><h4>Building the Context</h4><p>The selected document chunks are combined into a context:</p><pre>Source: return-policy<br>Title: Return Policy<br>Content: Customers have 30 days to return unopened products with the original receipt.</pre><p>This context is sent to OpenAI together with the question:</p><pre>String answer = chatClient.prompt()<br>    .user(&quot;&quot;&quot;<br>        Context:<br>        %s<br><br>        Question:<br>        %s<br>        &quot;&quot;&quot;.formatted(context, question))<br>    .call()<br>    .content();</pre><p>The model receives only the document parts selected by the vector search. Even if the database contains thousands of chunks, only the most relevant ones are included in the prompt.</p><h4>Returning the Sources</h4><p>The API returns both the generated answer and the documents used as sources:</p><pre>{<br>  &quot;answer&quot;: &quot;Customers have 30 days to return unopened products.&quot;,<br>  &quot;sources&quot;: [<br>    {<br>      &quot;source&quot;: &quot;return-policy&quot;,<br>      &quot;title&quot;: &quot;Return Policy&quot;,<br>      &quot;score&quot;: 0.44<br>    }<br>  ]<br>}</pre><p>The score represents the similarity between the question and the document chunk.</p><p>Returning sources makes the response easier to understand and verify. It also shows that the answer was generated using the stored document rather than only the model’s general knowledge.</p><h4>Running with Docker</h4><p>PostgreSQL, pgvector, and the Spring Boot application run through Docker Compose:</p><pre>docker compose up --build</pre><p>The OpenAI API key is provided through a local .env file:</p><pre>OPENAI_API_KEY=your-api-key</pre><p>After startup, the application is available at:</p><pre>http://localhost:8094</pre><h4>Conclusion</h4><p>This project helped me understand the main building blocks of a RAG application without moving too far away from the Java and Spring Boot ecosystem.</p><p>The most important point for me was understanding that the LLM does not search the database itself. The application creates embeddings, performs the vector search, prepares the context, and then asks the model to generate an answer.</p><p>This is a small demo, but the same structure can be extended with PDF uploads, document management, conversation history, authentication, and different data sources.</p><p>The source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-openai-rag-pgvector-demo">Spring Boot OpenAI RAG pgvector Demo</a></p><p>Thanks for reading.</p><p>Happy coding! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f29c7d672c67" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transactional Outbox and Idempotent Consumer Together with Spring Boot, Kafka and PostgreSQL]]></title>
            <link>https://erkndmrl.medium.com/transactional-outbox-and-idempotent-consumer-together-with-spring-boot-kafka-and-postgresql-4897180b9d8a?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/4897180b9d8a</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[idempotent-consumer]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[outbox-pattern]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 09:37:49 GMT</pubDate>
            <atom:updated>2026-10-17T09:37:49.702Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUS1ajy1q0HhJiAs4U5zfg.png" /></figure><p>Building reliable event-driven systems is not only about publishing events successfully. We also need to handle events that may be published or consumed more than once.</p><p>In this project, we combine two patterns:</p><ul><li>Transactional Outbox on the producer side</li><li>Idempotent Consumer on the consumer side</li></ul><p>Together, they create a more reliable end-to-end event flow.</p><p>GitHub repository: <a href="https://github.com/erkandemirel/spring-boot-kafka-outbox-idempotent-consumer-demo">spring-boot-kafka-outbox-idempotent-consumer-demo</a></p><h4>The Problem</h4><p>When an order is created, the application needs to perform two operations:</p><ol><li>Save the order to PostgreSQL.</li><li>Publish an OrderCreatedEvent to Kafka.</li></ol><p>A simple implementation could look like this:</p><pre>orderRepository.save(order);<br>kafkaTemplate.send(&quot;order-created&quot;, event);</pre><p>But these operations do not share the same transaction.</p><p>The order may be committed successfully while Kafka publishing fails. In that case, the database contains the order, but other services never receive its event.</p><p>This is known as the dual-write problem.</p><h4>Transactional Outbox</h4><p>Instead of publishing directly to Kafka, the Order Service saves both the order and its event in the same PostgreSQL transaction.</p><pre>@Transactional<br>OrderEntity create(String productId, int quantity, BigDecimal amount) {<br>    OrderEntity order = orderRepository.save(<br>            new OrderEntity(orderId, productId, quantity, amount)<br>    );<br><br>    outboxRepository.save(<br>            new OutboxEvent(<br>                    eventId,<br>                    orderId,<br>                    &quot;OrderCreatedEvent&quot;,<br>                    EventTopics.ORDER_CREATED,<br>                    objectMapper.writeValueAsString(event)<br>            )<br>    );<br><br>    return order;<br>}</pre><p>Because both records use the same database transaction, they either succeed together or roll back together.</p><p>The result is stored in two tables:</p><pre>orders<br>outbox_events</pre><p>A scheduled publisher periodically reads NEW and FAILED outbox records and sends them to Kafka.</p><pre>@Scheduled(fixedDelayString = &quot;${outbox.publisher.fixed-delay:2000}&quot;)<br>@Transactional<br>public void publishPending() {<br>    repository<br>            .findTop50ByStatusInOrderByCreatedAtAsc(PENDING)<br>            .forEach(this::publish);<br>}</pre><p>After a successful publish, the outbox record becomes PUBLISHED. Failed records remain available for another attempt.</p><h4>Why Can Duplicates Still Happen?</h4><p>The Outbox Pattern prevents event loss, but it does not guarantee that an event will be published only once.</p><p>Consider this situation:</p><pre>Event published to Kafka<br>        ↓<br>Order Service crashes<br>        ↓<br>Outbox status was not updated<br>        ↓<br>The same event is published again</pre><p>This is normal in an at-least-once delivery system.</p><p>For this reason, the consumer must also be idempotent.</p><h4>Idempotent Consumer</h4><p>The Inventory Service consumes OrderCreatedEvent messages from Kafka.</p><p>Before creating an inventory reservation, it attempts to insert the event ID into the processed_events table.</p><pre>INSERT INTO processed_events (<br>    event_id,<br>    order_id,<br>    consumer_name,<br>    topic,<br>    partition_number,<br>    offset_number,<br>    processed_at<br>)<br>VALUES (...)<br>ON CONFLICT (event_id) DO NOTHING</pre><p>The event_id is the primary key.</p><p>If the insert succeeds, this is a new event and the inventory reservation is created.</p><pre>int claimed = processedEventRepository.insertIfAbsent(<br>        event.eventId(),<br>        event.orderId(),<br>        CONSUMER_NAME,<br>        record.topic(),<br>        record.partition(),<br>        record.offset(),<br>        Instant.now()<br>);<br><br>if (claimed == 0) {<br>    log.info(&quot;Duplicate event {} ignored&quot;, event.eventId());<br>    return;<br>}<br><br>reservationRepository.save(<br>        new InventoryReservation(<br>                event.orderId(),<br>                event.eventId(),<br>                event.productId(),<br>                event.quantity()<br>        )<br>);</pre><p>The processed-event record and inventory reservation are written within the same database transaction.</p><p>Therefore, the consumer either completes both operations or rolls both of them back.</p><h4>Application Flow</h4><p>The complete flow looks like this:</p><pre>Order Service<br>    ↓<br>Order + Outbox Event<br>    ↓<br>Outbox Publisher<br>    ↓<br>Kafka<br>    ↓<br>Idempotency Check<br>    ↓<br>Inventory Reservation</pre><p>The producer prevents event loss, while the consumer prevents duplicate business operations.</p><h4>Running the Project</h4><p>Start the complete environment:</p><pre>docker compose up --build -d</pre><p>Create an order:</p><pre>curl -X POST \<br>&quot;http://localhost:8092/orders?productId=laptop&amp;quantity=2&amp;amount=1499.90&quot;</pre><p>Check the outbox:</p><pre>curl http://localhost:8092/outbox</pre><p>Check the inventory reservation:</p><pre>curl http://localhost:8093/inventory/reservations</pre><p>You can also see the processed event:</p><pre>curl http://localhost:8093/inventory/processed-events</pre><h4>Testing Duplicate Delivery</h4><p>Copy the event ID from the outbox response and publish the same event again:</p><pre>curl -X POST \<br>http://localhost:8092/outbox/{eventId}/replay</pre><p>The Inventory Service receives the event again, but recognizes the existing eventId.</p><p>The log shows:</p><pre>Duplicate event ignored</pre><p>The Kafka topic contains two messages, but PostgreSQL still contains only one inventory reservation.</p><h4>Conclusion</h4><p>In this article, we combined the Transactional Outbox Pattern and the Idempotent Consumer Pattern in a Spring Boot application.</p><p>In this example:</p><ul><li>Order Service stores the order and Outbox event in the same PostgreSQL transaction.</li><li>Outbox Publisher sends pending events to Kafka.</li><li>Failed publications remain in the Outbox table and can be retried.</li><li>Inventory Service stores processed event IDs.</li><li>Duplicate Kafka messages do not create duplicate inventory reservations.</li></ul><p>The Transactional Outbox Pattern prevents database changes from being committed without the information required to publish an event.</p><p>However, the same Outbox event may still be published more than once. The Idempotent Consumer Pattern handles this possibility by making sure that each event affects the consumer’s business data only once.</p><p>Together, these patterns provide at-least-once delivery with idempotent processing. They do not create a distributed transaction or true end-to-end exactly-once delivery, but they provide a practical way to build reliable event-driven microservices.</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-kafka-outbox-idempotent-consumer-demo">Spring Boot Kafka Outbox Idempotent Consumer Demo</a></p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4897180b9d8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Idempotent Kafka Consumer Pattern with Spring Boot]]></title>
            <link>https://erkndmrl.medium.com/idempotent-kafka-consumer-pattern-with-spring-boot-a7ce02516c25?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7ce02516c25</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[idempotency]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[idempotent-consumer]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 12:38:48 GMT</pubDate>
            <atom:updated>2026-10-08T12:38:48.577Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOHcsgwvAumYiwSN_jGnyw.png" /></figure><p>In an event-driven microservice architecture, Kafka consumers may receive the same event more than once.</p><p>For example, Inventory Service may receive an OrderCreatedEvent and reserve inventory for the order.</p><p>If the same event is delivered again, the service must not reserve the inventory a second time.</p><p>Without duplicate protection, repeated delivery may cause:</p><ul><li>Charging the same payment twice</li><li>Refunding the same payment twice</li><li>Reserving the same inventory multiple times</li><li>Sending duplicate notifications</li><li>Creating duplicate database records</li></ul><p>Kafka commonly provides at-least-once delivery. This means that an event is not lost, but it may be delivered more than once.</p><p>This is where the Idempotent Consumer Pattern becomes useful.</p><p>In this article, we will implement an idempotent Kafka consumer using Spring Boot, Apache Kafka, and PostgreSQL.</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-kafka-idempotent-consumer-pattern-demo">Spring Boot Kafka Idempotent Consumer Pattern Demo</a></p><h4>What Is Idempotency?</h4><p>An operation is idempotent when executing it multiple times produces the same final result as executing it once.</p><p>Consider the following operation:</p><pre>Set order status to COMPLETED</pre><p>Executing this operation several times may produce the same final state.</p><p>However, the following operation is not naturally idempotent:</p><pre>Reduce inventory by 2</pre><p>If it runs three times, the inventory is reduced by 6.</p><p>An idempotent consumer must identify previously processed events and prevent their business operations from being executed again.</p><h4>Why Can Kafka Deliver the Same Event More Than Once?</h4><p>A Kafka consumer normally performs two separate operations:</p><pre>Process the message<br>        ↓<br>Commit the Kafka offset</pre><p>Consider the following failure:</p><pre>Receive OrderCreatedEvent<br>        ↓<br>Reserve inventory in PostgreSQL<br>        ↓<br>Commit database transaction<br>        ↓<br>Application stops before offset commit</pre><p>The inventory transaction has already succeeded, but Kafka does not know that processing was completed.</p><p>After the consumer restarts or the partition is reassigned, Kafka may deliver the same message again.</p><p>Without idempotency, the inventory operation would run twice.</p><h4>Project Scenario</h4><p>The project contains two Spring Boot services:</p><ul><li>Order Producer Service</li><li>Inventory Consumer Service</li></ul><p>Order Producer Service creates an OrderCreatedEvent.</p><p>For demonstration purposes, it can intentionally publish multiple identical copies of the event.</p><p>Inventory Consumer Service:</p><ol><li>Receives the event.</li><li>Attempts to record its eventId.</li><li>Checks whether the event was already claimed.</li><li>Creates the inventory record only for the first delivery.</li><li>Ignores duplicate deliveries.</li></ol><p>The complete flow is:</p><pre>Order Producer Service<br>        |<br>        | Same event published multiple times<br>        v<br>Kafka: idempotent.order-created<br>        |<br>        v<br>Inventory Consumer Service<br>        |<br>        | Atomic event claim<br>        v<br>processed_events<br>        |<br>        +-- New event ------&gt; Create inventory record<br>        |<br>        +-- Existing event -&gt; Ignore duplicate</pre><p>The inventory operation and processed-event record are stored in the same PostgreSQL transaction.</p><h4>Technologies</h4><p>The project uses:</p><ul><li>Java 17</li><li>Spring Boot 3.5</li><li>Spring Kafka</li><li>Spring Data JPA</li><li>Apache Kafka 4.1</li><li>PostgreSQL 17</li><li>Docker Compose</li><li>Kafka UI</li></ul><p>Kafka runs in KRaft mode, so ZooKeeper is not required.</p><h4>Project Structure</h4><p>The project is implemented as a Maven multi-module application.</p><pre>spring-boot-kafka-idempotent-consumer-pattern-demo<br>├── event-messages<br>├── order-producer-service<br>├── inventory-consumer-service<br>├── docker-compose.yml<br>└── pom.xml</pre><p>The root pom.xml defines the modules:</p><pre>&lt;modules&gt;<br>    &lt;module&gt;event-messages&lt;/module&gt;<br>    &lt;module&gt;order-producer-service&lt;/module&gt;<br>    &lt;module&gt;inventory-consumer-service&lt;/module&gt;<br>&lt;/modules&gt;</pre><h4>event-messages</h4><p>Contains the shared event contract and Kafka topic name.</p><h4>order-producer-service</h4><p>Creates events and publishes one or more identical copies to Kafka.</p><p>It also provides an endpoint for replaying a previously published event.</p><h4>inventory-consumer-service</h4><p>Consumes events and stores:</p><ul><li>Inventory records</li><li>Processed event identifiers</li></ul><p>Both records are managed in the consumer’s PostgreSQL database.</p><h4>Defining the Kafka Topic</h4><p>The project uses one Kafka topic:</p><pre>public final class EventTopics {<br><br>    public static final String ORDER_CREATED =<br>            &quot;idempotent.order-created&quot;;<br><br>    private EventTopics() {<br>    }<br>}</pre><h4>Creating the Event Contract</h4><p>Every event contains a unique eventId.</p><pre>public record OrderCreatedEvent(<br>        UUID eventId,<br>        UUID orderId,<br>        String productId,<br>        int quantity,<br>        Instant occurredAt<br>) {<br>}</pre><p>The difference between eventId and orderId is important.</p><pre>eventId → Identifies the individual event<br>orderId → Identifies the business aggregate</pre><p>When the same event is redelivered, its eventId must remain unchanged.</p><p>A new event related to the same order should have a new eventId.</p><h4>Publishing Duplicate Events</h4><p>Order Producer Service stores published events in memory so they can be replayed during the demonstration.</p><pre>@Service<br>public class OrderEventPublisher {<br><br>    private final KafkaTemplate&lt;String, String&gt;<br>            kafkaTemplate;<br><br>    private final ObjectMapper objectMapper;<br><br>    private final Map&lt;UUID, OrderCreatedEvent&gt;<br>            publishedEvents = new ConcurrentHashMap&lt;&gt;();<br><br>    public void publish(<br>            OrderCreatedEvent event,<br>            int copies<br>    ) {<br>        publishedEvents.put(event.eventId(), event);<br>        sendCopies(event, copies);<br>    }<br>}</pre><p>The sendCopies method serializes the event once and publishes the same payload multiple times.</p><pre>private void sendCopies(<br>        OrderCreatedEvent event,<br>        int copies<br>) {<br>    try {<br>        String payload =<br>                objectMapper.writeValueAsString(event);<br><br>        for (int copy = 0; copy &lt; copies; copy++) {<br>            kafkaTemplate.send(<br>                    EventTopics.ORDER_CREATED,<br>                    event.orderId().toString(),<br>                    payload<br>            ).get(10, TimeUnit.SECONDS);<br>        }<br>    } catch (JsonProcessingException exception) {<br>        throw new IllegalStateException(<br>                &quot;Could not serialize the order event&quot;,<br>                exception<br>        );<br>    } catch (Exception exception) {<br>        throw new IllegalStateException(<br>                &quot;Could not publish the order event&quot;,<br>                exception<br>        );<br>    }<br>}</pre><p>Every copy contains:</p><ul><li>The same eventId</li><li>The same orderId</li><li>The same Kafka key</li><li>The same JSON payload</li><li>A different Kafka offset</li></ul><p>The order ID is used as the Kafka key. This keeps events for the same order in the same Kafka partition.</p><h4>Creating the Producer Endpoint</h4><p>The controller creates an event and accepts a copies parameter.</p><pre>@PostMapping<br>@ResponseStatus(HttpStatus.ACCEPTED)<br>public PublishResponse publish(<br>        @Valid<br>        @RequestBody CreateOrderEventRequest request,<br><br>        @RequestParam(defaultValue = &quot;1&quot;)<br>        @Min(1)<br>        @Max(10)<br>        int copies<br>) {<br>    OrderCreatedEvent event =<br>            new OrderCreatedEvent(<br>                    UUID.randomUUID(),<br>                    UUID.randomUUID(),<br>                    request.productId(),<br>                    request.quantity(),<br>                    Instant.now()<br>            );<br><br>    publisher.publish(event, copies);<br><br>    return new PublishResponse(event, copies);<br>}</pre><p>The default value is one.</p><p>Setting copies=3 publishes three Kafka records containing the same event.</p><h4>Replaying an Existing Event</h4><p>The producer also provides a replay endpoint.</p><pre>@PostMapping(&quot;/{eventId}/replay&quot;)<br>@ResponseStatus(HttpStatus.ACCEPTED)<br>public PublishResponse replay(<br>        @PathVariable UUID eventId,<br><br>        @RequestParam(defaultValue = &quot;1&quot;)<br>        @Min(1)<br>        @Max(10)<br>        int copies<br>) {<br>    return new PublishResponse(<br>            publisher.replay(eventId, copies),<br>            copies<br>    );<br>}</pre><p>Replay does not generate a new event.</p><p>It sends the original event with the same eventId.</p><p>This allows us to test whether the consumer safely ignores an event that was already processed.</p><h4>Inventory Record</h4><p>Inventory Service stores the business result in the inventory_records table.</p><pre>@Entity<br>@Table(name = &quot;inventory_records&quot;)<br>public class InventoryRecord {<br><br>    @Id<br>    private UUID orderId;<br><br>    private UUID sourceEventId;<br><br>    private String productId;<br><br>    private int quantity;<br><br>    @Enumerated(EnumType.STRING)<br>    private InventoryStatus status;<br><br>    private Instant createdAt;<br>}</pre><p>The sourceEventId shows which event created the inventory record.</p><p>The demo uses a simple inventory status:</p><pre>public enum InventoryStatus {<br>    RESERVED<br>}</pre><h4>Processed Event Entity</h4><p>Processed event IDs are stored in a separate table.</p><pre>@Entity<br>@Table(name = &quot;processed_events&quot;)<br>public class ProcessedEvent {<br><br>    @Id<br>    private UUID eventId;<br><br>    private UUID orderId;<br><br>    private String consumerName;<br><br>    private String topic;<br><br>    private int partitionNumber;<br><br>    private long offsetNumber;<br><br>    private Instant processedAt;<br>}</pre><p>The eventId column is the primary key.</p><p>This database constraint prevents the same event ID from being inserted twice.</p><p>The table also records:</p><ul><li>Consumer name</li><li>Topic</li><li>Partition</li><li>Offset</li><li>Processing time</li></ul><p>These fields are useful when investigating event processing.</p><h4>Why a Simple Exists Check Is Not Enough</h4><p>A basic implementation may use the following logic:</p><pre>if (repository.existsById(eventId)) {<br>    return;<br>}<br><br>repository.save(processedEvent);</pre><p>This looks correct, but it contains a race condition.</p><p>Two consumer threads could execute the existence check at the same time:</p><pre>Consumer A checks eventId → not found<br>Consumer B checks eventId → not found<br>Consumer A processes event<br>Consumer B processes event</pre><p>A unique database constraint prevents duplicate rows, but the business operation may already have been executed by both consumers.</p><p>The event must be claimed atomically.</p><h4>Atomically Claiming the Event</h4><p>The project uses PostgreSQL’s ON CONFLICT DO NOTHING.</p><pre>public interface ProcessedEventRepository extends JpaRepository&lt;ProcessedEvent, UUID&gt; {<br><br>    @Modifying<br>    @Query(value = &quot;&quot;&quot;<br>        INSERT INTO processed_events<br>            (<br>                event_id,<br>                order_id,<br>                consumer_name,<br>                topic,<br>                partition_number,<br>                offset_number,<br>                processed_at<br>            )<br>        VALUES<br>            (<br>                :eventId,<br>                :orderId,<br>                :consumerName,<br>                :topic,<br>                :partitionNumber,<br>                :offsetNumber,<br>                :processedAt<br>            )<br>        ON CONFLICT (event_id) DO NOTHING<br>        &quot;&quot;&quot;, nativeQuery = true)<br>        int insertIfAbsent(<br>            @Param(&quot;eventId&quot;) UUID eventId,<br>            @Param(&quot;orderId&quot;) UUID orderId,<br>            @Param(&quot;consumerName&quot;) String consumerName,<br>            @Param(&quot;topic&quot;) String topic,<br>            @Param(&quot;partitionNumber&quot;) int partitionNumber,<br>            @Param(&quot;offsetNumber&quot;) long offsetNumber,<br>            @Param(&quot;processedAt&quot;) Instant processedAt<br>    );<br>}</pre><p>The returned insert count determines whether the event was successfully claimed.</p><pre>Insert count 1 → First delivery<br>Insert count 0 → Duplicate delivery</pre><p>This combines the duplicate check and insert into one atomic database operation.</p><h4>Implementing the Idempotent Consumer</h4><p>Inventory Consumer Service receives the complete Kafka record.</p><pre>@KafkaListener(topics = EventTopics.ORDER_CREATED)<br>@Transactional<br>public void onOrderCreated(<br>        ConsumerRecord&lt;String, String&gt; record<br>) throws JsonProcessingException {<br><br>    OrderCreatedEvent event =<br>            objectMapper.readValue(<br>                    record.value(),<br>                    OrderCreatedEvent.class<br>            );<br><br>    int claimed =<br>            processedEventRepository.insertIfAbsent(<br>                    event.eventId(),<br>                    event.orderId(),<br>                    &quot;inventory-consumer-service&quot;,<br>                    record.topic(),<br>                    record.partition(),<br>                    record.offset(),<br>                    Instant.now()<br>            );<br><br>    if (claimed == 0) {<br>        log.info(<br>                &quot;Duplicate event {} ignored at offset {}&quot;,<br>                event.eventId(),<br>                record.offset()<br>        );<br><br>        return;<br>    }<br><br>    inventoryRepository.save(<br>            new InventoryRecord(<br>                    event.orderId(),<br>                    event.eventId(),<br>                    event.productId(),<br>                    event.quantity()<br>            )<br>    );<br><br>    log.info(<br>            &quot;Event {} processed; &quot; +<br>            &quot;inventory reserved for order {}&quot;,<br>            event.eventId(),<br>            event.orderId()<br>    );<br>}</pre><p>The method is annotated with @Transactional.</p><p>The processed-event marker and inventory record therefore belong to the same PostgreSQL transaction.</p><p>The transaction has two possible results.</p><p>First delivery:</p><pre>Insert processed event → successful<br>Create inventory record<br>Commit transaction</pre><p>Duplicate delivery:</p><pre>Insert processed event → conflict<br>Skip inventory operation<br>Commit without business changes</pre><h4>Why Both Records Must Use the Same Transaction</h4><p>The processed-event marker and business operation must commit together.</p><p>Consider storing the event ID first in a separate transaction:</p><pre>Store eventId<br>    ↓<br>Commit<br>    ↓<br>Application fails before inventory reservation</pre><p>When Kafka delivers the event again, the consumer sees the stored event ID and ignores it.</p><p>The inventory operation is never completed.</p><p>Now consider performing the inventory operation first:</p><pre>Reserve inventory<br>    ↓<br>Commit<br>    ↓<br>Application fails before storing eventId</pre><p>Kafka redelivers the event, but there is no processed-event marker.</p><p>The inventory operation may run again.</p><p>Using one local transaction solves both problems:</p><pre>PostgreSQL Transaction<br>    ├── Claim eventId<br>    └── Reserve inventory</pre><p>Both operations commit or roll back together.</p><h4>Consumer Configuration</h4><p>The Inventory Consumer uses manual offset management through the listener container.</p><pre>spring:<br>  kafka:<br>    bootstrap-servers:<br>      ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}<br><br>    consumer:<br>      group-id: inventory-consumer-service<br>      auto-offset-reset: earliest<br>      enable-auto-commit: false<br><br>      key-deserializer:<br>        org.apache.kafka.common.serialization.StringDeserializer<br><br>      value-deserializer:<br>        org.apache.kafka.common.serialization.StringDeserializer<br><br>    listener:<br>      ack-mode: record</pre><p>The Kafka offset is committed after the listener successfully processes the record.</p><p>However, the database commit and Kafka offset commit are still separate operations.</p><p>The consumer may stop between them.</p><p>This is why the database-based idempotency check remains necessary.</p><h4>Docker Compose Environment</h4><p>The complete environment contains five containers:</p><ul><li>Order Producer Service</li><li>Inventory Consumer Service</li><li>PostgreSQL</li><li>Apache Kafka</li><li>Kafka UI</li></ul><p>The services are available at:</p><pre>Order Producer     http://localhost:8090<br>Inventory Consumer http://localhost:8091<br>Kafka UI           http://localhost:8088<br>Kafka              localhost:9092<br>PostgreSQL         localhost:5433</pre><h4>Running the Project</h4><p>Clone the repository:</p><pre>git clone \<br>https://github.com/erkandemirel/spring-boot-kafka-idempotent-consumer-pattern-demo.git</pre><p>Open the project directory:</p><pre>cd spring-boot-kafka-idempotent-consumer-pattern-demo</pre><p>Build all Maven modules:</p><pre>mvn clean package</pre><p>Start the environment:</p><pre>docker compose up --build</pre><p>Check the containers:</p><pre>docker compose ps</pre><h4>Testing a Single Event</h4><p>Publish one event:</p><pre>curl -X POST \<br>  &quot;http://localhost:8090/order-events&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-100&quot;,<br>    &quot;quantity&quot;: 2<br>  }&#39;</pre><p>The response contains the generated IDs:</p><pre>{<br>  &quot;event&quot;: {<br>    &quot;eventId&quot;: &quot;0f015434-a009-4dff-b8e7-40f9f35e0f0a&quot;,<br>    &quot;orderId&quot;: &quot;cc492eb9-c434-4aa8-bc7b-ebc6810091a2&quot;,<br>    &quot;productId&quot;: &quot;product-100&quot;,<br>    &quot;quantity&quot;: 2<br>  },<br>  &quot;publishedCopies&quot;: 1<br>}</pre><p>Check the inventory records:</p><pre>curl http://localhost:8091/inventory-records</pre><p>Check the processed events:</p><pre>curl http://localhost:8091/processed-events</pre><p>Expected row counts:</p><pre>inventory_records  1<br>processed_events   1</pre><h4>Testing Duplicate Delivery</h4><p>Publish the exact same event three times using the copies parameter:</p><pre>curl -X POST \<br>  &quot;http://localhost:8090/order-events?copies=3&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-duplicate-demo&quot;,<br>    &quot;quantity&quot;: 4<br>  }&#39;</pre><p>The producer sends three Kafka messages.</p><p>All three messages contain the same:</p><pre>eventId<br>orderId<br>Kafka key<br>JSON payload</pre><p>Only their Kafka offsets are different.</p><p>The consumer processes the first message:</p><pre>Event ... processed;<br>inventory reserved for order ...</pre><p>The remaining messages are ignored:</p><pre>Duplicate event ... ignored at offset ...<br>Duplicate event ... ignored at offset ...</pre><p>Check the database again:</p><pre>curl http://localhost:8091/inventory-records<br>curl http://localhost:8091/processed-events</pre><p>Although Kafka contains three messages, only one inventory record and one processed-event record are created.</p><pre>Kafka messages        3<br>Inventory records     1<br>Processed-event rows  1<br>Ignored duplicates    2</pre><h4>Testing Event Replay</h4><p>Copy the eventId from the previous response.</p><p>Replay the same event three more times:</p><pre>curl -X POST \<br>  &quot;http://localhost:8090/order-events/{eventId}/replay?copies=3&quot;</pre><p>The original event is published again without changing its identity.</p><p>After replay, the complete result is:</p><pre>Kafka deliveries      6<br>Inventory records     1<br>Processed-event rows  1<br>Ignored duplicates    5</pre><p>The final database state remains unchanged.</p><p>This is the expected behavior of an idempotent consumer.</p><h4>Inspecting Kafka Messages</h4><p>Open Kafka UI:</p><pre>http://localhost:8088</pre><p>Select the topic:</p><pre>idempotent.order-created</pre><p>The duplicated messages have different offsets but the same event ID.</p><pre>Offset 0 → eventId: 0f015434...<br>Offset 1 → eventId: 0f015434...<br>Offset 2 → eventId: 0f015434...</pre><p>Kafka stores every published record.</p><p>The consumer decides whether processing the record should produce a new business side effect.</p><h4>Inspecting PostgreSQL</h4><p>Connect to PostgreSQL using pgAdmin:</p><pre>Host: localhost<br>Port: 5433<br>Database: inventory<br>Username: inventory<br>Password: inventory</pre><p>The database contains:</p><pre>inventory_records<br>processed_events</pre><p>The processed_events table records the first successfully processed delivery.</p><p>Later duplicate offsets do not create additional rows.</p><h4>Idempotent Consumer and Transactional Outbox</h4><p>The Idempotent Consumer Pattern is a natural continuation of the Transactional Outbox Pattern.</p><p>Transactional Outbox prevents a producer event from being lost:</p><pre>Business data + Outbox event<br>saved in one transaction</pre><p>However, an Outbox Publisher may produce duplicates:</p><pre>Publish event to Kafka<br>        ↓<br>Kafka acknowledges the message<br>        ↓<br>Publisher stops before marking<br>the Outbox row as PUBLISHED<br>        ↓<br>Event is published again</pre><p>The two patterns solve complementary problems:</p><pre>Transactional Outbox<br>    → Prevents lost producer events<br><br>Idempotent Consumer<br>    → Prevents duplicate consumer side effects</pre><p>A reliable event-driven system commonly needs both.</p><h4>Natural and Explicit Idempotency</h4><p>Some operations are naturally idempotent.</p><p>For example:</p><pre>Set status to COMPLETED</pre><p>Other operations require an explicit event record:</p><pre>Decrease stock<br>Charge payment<br>Issue refund<br>Send reward points</pre><p>Even a status update may require a guard to prevent an old event from overwriting a newer state.</p><p>Idempotency should be designed according to the business operation, not only the Kafka configuration.</p><h4>Production Considerations</h4><p>This project focuses on the core Idempotent Consumer Pattern.</p><p>A production implementation should also consider the following points.</p><h4>Consumer-Specific Event Identity</h4><p>If multiple logical consumers share the same table, use a composite unique key:</p><pre>consumer_name + event_id</pre><p>This allows each consumer to process the event once independently.</p><h4>Processed-Event Cleanup</h4><p>The processed_events table grows continuously.</p><p>A retention policy should define:</p><ul><li>How long Kafka messages may be replayed</li><li>How long event IDs must remain available</li><li>When rows can be archived</li><li>When rows can be safely deleted</li></ul><p>Deleting event IDs too early may allow an old replayed event to run again.</p><h4>Retry and Dead Letter Topics</h4><p>Technical failures should normally be retried.</p><p>Events that continue to fail should be sent to a Dead Letter Topic for investigation.</p><p>Duplicate events are not failures and should not be sent to a Dead Letter Topic.</p><p>They should be acknowledged and ignored.</p><h4>Event Schema Validation</h4><p>Consumers should validate required fields before applying business operations.</p><p>Invalid events should not be recorded as successfully processed unless the system intentionally decides to discard them.</p><h4>Observability</h4><p>Useful consumer metrics include:</p><ul><li>Successfully processed events</li><li>Ignored duplicate events</li><li>Failed events</li><li>Retry attempts</li><li>Processing duration</li><li>Consumer lag</li></ul><p>Logs and traces should contain:</p><ul><li>eventId</li><li>orderId</li><li>Topic</li><li>Partition</li><li>Offset</li><li>Consumer name</li></ul><h4>External Side Effects</h4><p>A PostgreSQL transaction cannot roll back an HTTP request sent to an external system.</p><p>For example:</p><pre>Call payment provider<br>        ↓<br>Provider charges payment<br>        ↓<br>Application stops before storing eventId</pre><p>The Kafka event may be redelivered and the external provider may be called again.</p><p>External operations should support their own idempotency key.</p><p>The event ID or another stable business key can be sent to the external system as that idempotency key.</p><h4>Database Migrations</h4><p>The demo uses Hibernate automatic schema updates.</p><p>Production systems should normally manage schema changes using Flyway or Liquibase.</p><h4>Conclusion</h4><p>Kafka may deliver the same event more than once.</p><p>This is expected behavior in an at-least-once delivery system.</p><p>The consumer must therefore ensure that repeated deliveries do not repeat business side effects.</p><p>In this example:</p><ul><li>Order Producer Service publishes duplicate events.</li><li>Every event contains a stable eventId.</li><li>Inventory Consumer Service atomically claims the event ID.</li><li>PostgreSQL prevents the same ID from being claimed twice.</li><li>The event marker and inventory operation share one transaction.</li><li>Duplicate messages are acknowledged and ignored.</li><li>Replaying the same event does not change the final database state.</li></ul><p>The important point is that idempotency is not achieved by assuming Kafka will deliver an event only once.</p><p>It is achieved by making repeated processing safe.</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-kafka-idempotent-consumer-pattern-demo">Spring Boot Kafka Idempotent Consumer Pattern Demo</a></p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7ce02516c25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transactional Outbox Pattern with Spring Boot, Kafka and PostgreSQL]]></title>
            <link>https://erkndmrl.medium.com/transactional-outbox-pattern-with-spring-boot-kafka-and-postgresql-b51ebd433d2f?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/b51ebd433d2f</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[outbox-pattern]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 13:32:00 GMT</pubDate>
            <atom:updated>2026-09-15T13:39:32.908Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M6I7nyPN_YFwGeHGpAtT4A.png" /></figure><p>In an event-driven microservice architecture, a service often needs to perform two operations:</p><ul><li>Save business data to a database</li><li>Publish an event to a message broker</li></ul><p>For example, Order Service may save a new order to PostgreSQL and then publish an OrderCreatedEvent to Apache Kafka.</p><p>These operations use two different systems.</p><p>A PostgreSQL transaction cannot automatically include a Kafka message, and a Kafka transaction cannot automatically control a PostgreSQL commit.</p><p>This creates a dual-write problem.</p><p>Consider the following flow:</p><pre>Save Order<br>    ↓<br>Commit Database Transaction<br>    ↓<br>Publish OrderCreatedEvent</pre><p>What happens if the database transaction succeeds but the application stops before publishing the event?</p><p>The order exists in the database, but downstream services never receive the event.</p><p>The system becomes inconsistent.</p><p>This is where the Transactional Outbox Pattern becomes useful.</p><p>In this article, we will implement a polling-based Transactional Outbox Pattern using Spring Boot, Apache Kafka, PostgreSQL, and Docker Compose.</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-kafka-transactional-outbox-pattern-demo">Spring Boot Kafka Transactional Outbox Pattern Demo</a></p><h4>What Is the Transactional Outbox Pattern?</h4><p>The Transactional Outbox Pattern stores business data and event data in the same database transaction.</p><p>Instead of publishing an event directly to Kafka, the application writes the event to an Outbox table.</p><pre>Database Transaction<br>    ├── Insert Order<br>    └── Insert Outbox Event</pre><p>Because both records are stored in the same PostgreSQL transaction, they either succeed together or fail together.</p><p>A separate component reads pending records from the Outbox table and publishes them to Kafka.</p><pre>Client<br>  |<br>  v<br>Order Service<br>  |<br>  | Single PostgreSQL Transaction<br>  |<br>  +-----------&gt; orders<br>  |<br>  +-----------&gt; outbox_events<br>                     |<br>                     v<br>              Outbox Publisher<br>                     |<br>                     v<br>                   Kafka<br>                     |<br>                     v<br>             Inventory Service</pre><p>If Kafka is unavailable, the Outbox event remains in PostgreSQL.</p><p>The publisher can retry it when Kafka becomes available again.</p><p>According to Spring’s explanation of the Outbox Pattern, the main objective is to coordinate database changes and message publication without using distributed two-phase commits.</p><h4>Project Scenario</h4><p>The project contains two Spring Boot services:</p><ul><li>Order Service</li><li>Inventory Service</li></ul><p>Order Service accepts an order request.</p><p>It stores:</p><ul><li>The new order in the orders table</li><li>An OrderCreatedEvent in the outbox_events table</li></ul><p>Both operations are executed inside the same database transaction.</p><p>The Outbox Publisher periodically checks the Outbox table.</p><p>When it finds a pending event, it publishes the serialized event to Kafka.</p><p>Inventory Service consumes the event and creates an inventory record in its own PostgreSQL database.</p><p>The complete flow is:</p><pre>POST /orders<br>     |<br>     v<br>Order Service<br>     |<br>     | One Local Transaction<br>     |<br>     +----&gt; orders<br>     |<br>     +----&gt; outbox_events: NEW<br>                    |<br>                    v<br>            Outbox Publisher<br>                    |<br>                    v<br>        Kafka: outbox.order-created<br>                    |<br>                    v<br>           Inventory Service<br>                    |<br>                    v<br>       inventory_records: RESERVED</pre><p>After Kafka acknowledges the message, the Outbox event status changes from NEW to PUBLISHED.</p><h4>Technologies</h4><p>The project uses:</p><ul><li>Java 17</li><li>Spring Boot 3.5</li><li>Spring Data JPA</li><li>Spring Kafka</li><li>Apache Kafka 4.1</li><li>PostgreSQL 17</li><li>Docker Compose</li><li>Kafka UI</li></ul><p>Kafka runs in KRaft mode, so ZooKeeper is not required.</p><h4>Project Structure</h4><p>The project is implemented as a Maven multi-module application.</p><pre>spring-boot-kafka-transactional-outbox-pattern-demo<br>├── outbox-messages<br>├── order-service<br>├── inventory-service<br>├── docker-compose.yml<br>└── pom.xml</pre><p>The root pom.xml defines the modules:</p><pre>&lt;modules&gt;<br>    &lt;module&gt;outbox-messages&lt;/module&gt;<br>    &lt;module&gt;order-service&lt;/module&gt;<br>    &lt;module&gt;inventory-service&lt;/module&gt;<br>&lt;/modules&gt;</pre><h4>outbox-messages</h4><p>This module contains the event contract and Kafka topic name shared by the services.</p><h4>order-service</h4><p>This service:</p><ul><li>Creates orders</li><li>Stores Outbox events</li><li>Publishes pending events to Kafka</li><li>Retries failed event publications</li></ul><h4>inventory-service</h4><p>This service:</p><ul><li>Consumes OrderCreatedEvent</li><li>Creates an inventory reservation record</li><li>Stores the result in a separate PostgreSQL database</li></ul><h4>Defining the Kafka Topic</h4><p>The project uses a single Kafka topic:</p><pre>public final class OutboxTopics {<br><br>    public static final String ORDER_CREATED =<br>            &quot;outbox.order-created&quot;;<br><br>    private OutboxTopics() {<br>    }<br>}</pre><p>Keeping topic names in a shared class prevents services from using different topic names accidentally.</p><h4>Creating the Event Contract</h4><p>The event contains an eventId in addition to the order information.</p><pre>public record OrderCreatedEvent(<br>        UUID eventId,<br>        UUID orderId,<br>        String productId,<br>        int quantity,<br>        BigDecimal amount,<br>        Instant occurredAt<br>) {<br>}</pre><p>The eventId uniquely identifies the event.</p><p>This identifier is important because the Transactional Outbox Pattern normally provides at-least-once delivery.</p><p>The same event may therefore be delivered more than once.</p><p>A production consumer can use eventId to detect duplicate messages.</p><h4>Order Entity</h4><p>The Order Service stores orders in its own PostgreSQL database.</p><pre>@Entity<br>@Table(name = &quot;orders&quot;)<br>public class OrderEntity {<br><br>    @Id<br>    private UUID id;<br><br>    private String productId;<br><br>    private int quantity;<br><br>    private BigDecimal amount;<br><br>    @Enumerated(EnumType.STRING)<br>    private OrderStatus status;<br><br>    private Instant createdAt;<br><br>    protected OrderEntity() {<br>    }<br><br>    public OrderEntity(<br>            UUID id,<br>            String productId,<br>            int quantity,<br>            BigDecimal amount<br>    ) {<br>        this.id = id;<br>        this.productId = productId;<br>        this.quantity = quantity;<br>        this.amount = amount;<br>        this.status = OrderStatus.CREATED;<br>        this.createdAt = Instant.now();<br>    }<br>}</pre><p>The demo uses a simple order status:</p><pre>public enum OrderStatus {<br>    CREATED<br>}</pre><h4>Outbox Event Entity</h4><p>The Outbox table stores the serialized event and the information required to publish it.</p><pre>@Entity<br>@Table(name = &quot;outbox_events&quot;)<br>public class OutboxEvent {<br><br>    @Id<br>    private UUID id;<br><br>    private String aggregateType;<br><br>    private UUID aggregateId;<br><br>    private String eventType;<br><br>    private String topic;<br><br>    @Lob<br>    @Column(columnDefinition = &quot;TEXT&quot;, nullable = false)<br>    private String payload;<br><br>    @Enumerated(EnumType.STRING)<br>    private OutboxStatus status;<br><br>    private int retryCount;<br><br>    private Instant createdAt;<br><br>    private Instant publishedAt;<br><br>    @Column(length = 1000)<br>    private String lastError;<br><br>    @Version<br>    private long version;<br>}</pre><p>The most important fields are:</p><ul><li>id: Unique event identifier</li><li>aggregateId: Order identifier</li><li>eventType: Type of the stored event</li><li>topic: Kafka destination topic</li><li>payload: Serialized JSON event</li><li>status: Current publication status</li><li>retryCount: Number of failed attempts</li><li>lastError: Latest publication error</li><li>publishedAt: Successful publication time</li></ul><p>The event statuses are:</p><pre>public enum OutboxStatus {<br>    NEW,<br>    PUBLISHED,<br>    FAILED<br>}</pre><p>Their meanings are:</p><pre>NEW        Event is waiting to be published<br>PUBLISHED  Kafka acknowledged the message<br>FAILED     The latest attempt failed and will be retried</pre><h4>Saving the Order and Outbox Event Together</h4><p>The most important part of the implementation is the transaction boundary.</p><pre>@Service<br>public class OrderApplicationService {<br><br>    private final OrderRepository orderRepository;<br>    private final OutboxEventRepository outboxRepository;<br>    private final ObjectMapper objectMapper;<br><br>    public OrderApplicationService(<br>            OrderRepository orderRepository,<br>            OutboxEventRepository outboxRepository,<br>            ObjectMapper objectMapper<br>    ) {<br>        this.orderRepository = orderRepository;<br>        this.outboxRepository = outboxRepository;<br>        this.objectMapper = objectMapper;<br>    }<br><br>    @Transactional<br>    public OrderEntity createOrder(<br>            String productId,<br>            int quantity,<br>            BigDecimal amount<br>    ) {<br>        UUID orderId = UUID.randomUUID();<br>        UUID eventId = UUID.randomUUID();<br>        Instant now = Instant.now();<br><br>        OrderEntity order = orderRepository.save(<br>                new OrderEntity(<br>                        orderId,<br>                        productId,<br>                        quantity,<br>                        amount<br>                )<br>        );<br><br>        OrderCreatedEvent event = new OrderCreatedEvent(<br>                eventId,<br>                orderId,<br>                productId,<br>                quantity,<br>                amount,<br>                now<br>        );<br><br>        try {<br>            String payload =<br>                    objectMapper.writeValueAsString(event);<br><br>            outboxRepository.save(<br>                    new OutboxEvent(<br>                            eventId,<br>                            orderId,<br>                            OrderCreatedEvent.class<br>                                    .getSimpleName(),<br>                            OutboxTopics.ORDER_CREATED,<br>                            payload<br>                    )<br>            );<br>        } catch (JsonProcessingException exception) {<br>            throw new IllegalStateException(<br>                    &quot;Could not serialize the order event&quot;,<br>                    exception<br>            );<br>        }<br><br>        return order;<br>    }<br>}</pre><p>The method does not call Kafka.</p><p>It only writes to PostgreSQL.</p><p>Because the method is annotated with @Transactional, inserting the order and inserting the Outbox event belong to the same local transaction.</p><p>If event serialization or the Outbox insert fails, the order transaction is also rolled back.</p><p>The database therefore cannot contain a successfully created order without its corresponding Outbox event.</p><h4>Creating the Order Endpoint</h4><p>The controller accepts a product ID, quantity, and amount.</p><pre>@RestController<br>@RequestMapping(&quot;/orders&quot;)<br>public class OrderController {<br><br>    private final OrderApplicationService service;<br>    private final OrderRepository repository;<br><br>    public OrderController(<br>            OrderApplicationService service,<br>            OrderRepository repository<br>    ) {<br>        this.service = service;<br>        this.repository = repository;<br>    }<br><br>    @PostMapping<br>    @ResponseStatus(HttpStatus.CREATED)<br>    public OrderEntity create(<br>            @Valid @RequestBody CreateOrderRequest request<br>    ) {<br>        return service.createOrder(<br>                request.productId(),<br>                request.quantity(),<br>                request.amount()<br>        );<br>    }<br><br>    @GetMapping<br>    public List&lt;OrderEntity&gt; list() {<br>        return repository.findAll();<br>    }<br><br>    public record CreateOrderRequest(<br>            @NotBlank String productId,<br>            @Min(1) int quantity,<br>            @NotNull @DecimalMin(&quot;0.01&quot;)<br>            BigDecimal amount<br>    ) {<br>    }<br>}</pre><p>The request is validated before the transaction begins.</p><h4>Finding Pending Outbox Events</h4><p>The Outbox repository retrieves both new and previously failed events.</p><pre>public interface OutboxEventRepository<br>        extends JpaRepository&lt;OutboxEvent, UUID&gt; {<br><br>    List&lt;OutboxEvent&gt;<br>    findTop50ByStatusInOrderByCreatedAtAsc(<br>            Collection&lt;OutboxStatus&gt; statuses<br>    );<br>}</pre><p>The publisher processes a maximum of 50 events during each polling cycle.</p><p>Both statuses are included:</p><pre>List.of(<br>    OutboxStatus.NEW,<br>    OutboxStatus.FAILED<br>)</pre><p>This allows previously failed events to be retried automatically.</p><h4>Publishing Events to Kafka</h4><p>The Outbox Publisher runs periodically using Spring’s @Scheduled annotation.</p><pre>@Component<br>public class OutboxPublisher {<br><br>    private static final Logger log =<br>            LoggerFactory.getLogger(<br>                    OutboxPublisher.class<br>            );<br><br>    private static final List&lt;OutboxStatus&gt;<br>            PENDING_STATUSES = List.of(<br>                    OutboxStatus.NEW,<br>                    OutboxStatus.FAILED<br>            );<br><br>    private final OutboxEventRepository repository;<br>    private final KafkaTemplate&lt;String, String&gt;<br>            kafkaTemplate;<br><br>    public OutboxPublisher(<br>            OutboxEventRepository repository,<br>            KafkaTemplate&lt;String, String&gt; kafkaTemplate<br>    ) {<br>        this.repository = repository;<br>        this.kafkaTemplate = kafkaTemplate;<br>    }<br><br>    @Scheduled(<br>        fixedDelayString =<br>            &quot;${outbox.publisher.fixed-delay:3000}&quot;<br>    )<br>    @Transactional<br>    public void publishPendingEvents() {<br>        repository<br>                .findTop50ByStatusInOrderByCreatedAtAsc(<br>                        PENDING_STATUSES<br>                )<br>                .forEach(this::publish);<br>    }<br>}</pre><p>The publisher sends the stored JSON payload to the topic recorded in the Outbox row.</p><pre>private void publish(OutboxEvent event) {<br>    try {<br>        kafkaTemplate.send(<br>                event.getTopic(),<br>                event.getAggregateId().toString(),<br>                event.getPayload()<br>        ).get(10, TimeUnit.SECONDS);<br><br>        event.markPublished();<br><br>        log.info(<br>                &quot;Published outbox event {} for order {}&quot;,<br>                event.getId(),<br>                event.getAggregateId()<br>        );<br>    } catch (Exception exception) {<br>        event.markFailed(exception);<br><br>        log.warn(<br>                &quot;Could not publish outbox event {}; &quot; +<br>                &quot;it will be retried&quot;,<br>                event.getId()<br>        );<br>    }<br>}</pre><p>The order ID is used as the Kafka message key.</p><p>When Kafka acknowledges the message, the event is marked as PUBLISHED.</p><pre>public void markPublished() {<br>    this.status = OutboxStatus.PUBLISHED;<br>    this.publishedAt = Instant.now();<br>    this.lastError = null;<br>}</pre><p>If publication fails, the event is not deleted.</p><pre>public void markFailed(Throwable exception) {<br>    this.status = OutboxStatus.FAILED;<br>    this.retryCount++;<br>    this.lastError = exception.getMessage();<br>}</pre><p>The next polling cycle selects the failed event again.</p><h4>Order Service Configuration</h4><p>The Order Service uses PostgreSQL and a String-based Kafka producer.</p><pre>server:<br>  port: 8080<br><br>spring:<br>  application:<br>    name: order-service<br><br>  datasource:<br>    url: ${DB_URL:jdbc:postgresql://localhost:5433/orders}<br>    username: ${DB_USERNAME:orders}<br>    password: ${DB_PASSWORD:orders}<br><br>  jpa:<br>    hibernate:<br>      ddl-auto: update<br>    open-in-view: false<br><br>  kafka:<br>    bootstrap-servers:<br>      ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}<br><br>    producer:<br>      key-serializer:<br>        org.apache.kafka.common.serialization.StringSerializer<br>      value-serializer:<br>        org.apache.kafka.common.serialization.StringSerializer<br><br>      properties:<br>        acks: all<br>        max.block.ms: 5000<br>        request.timeout.ms: 5000<br>        delivery.timeout.ms: 10000<br><br>outbox:<br>  publisher:<br>    fixed-delay:<br>      ${OUTBOX_PUBLISHER_DELAY:3000}</pre><p>The publisher runs approximately every three seconds.</p><p>The value is configurable using the OUTBOX_PUBLISHER_DELAY environment variable.</p><h4>Consuming the Event</h4><p>Inventory Service consumes the JSON payload as a String and converts it to OrderCreatedEvent.</p><pre>@Component<br>public class OrderCreatedListener {<br><br>    private static final Logger log =<br>            LoggerFactory.getLogger(<br>                    OrderCreatedListener.class<br>            );<br><br>    private final InventoryRepository repository;<br>    private final ObjectMapper objectMapper;<br><br>    public OrderCreatedListener(<br>            InventoryRepository repository,<br>            ObjectMapper objectMapper<br>    ) {<br>        this.repository = repository;<br>        this.objectMapper = objectMapper;<br>    }<br><br>    @KafkaListener(<br>            topics = OutboxTopics.ORDER_CREATED<br>    )<br>    @Transactional<br>    public void onOrderCreated(String payload)<br>            throws JsonProcessingException {<br><br>        OrderCreatedEvent event =<br>                objectMapper.readValue(<br>                        payload,<br>                        OrderCreatedEvent.class<br>                );<br><br>        if (repository.existsById(event.orderId())) {<br>            log.info(<br>                    &quot;Order {} was already processed&quot;,<br>                    event.orderId()<br>            );<br>            return;<br>        }<br><br>        repository.save(<br>                new InventoryRecord(<br>                        event.orderId(),<br>                        event.eventId(),<br>                        event.productId(),<br>                        event.quantity()<br>                )<br>        );<br><br>        log.info(<br>                &quot;Reserved inventory for order {} &quot; +<br>                &quot;from outbox event {}&quot;,<br>                event.orderId(),<br>                event.eventId()<br>        );<br>    }<br>}</pre><p>The demo includes a simple duplicate check based on orderId.</p><p>If the order was already processed, Inventory Service does not create another record.</p><p>For a production system, this should normally be implemented using a durable Idempotent Consumer or Inbox Pattern based on eventId.</p><h4>Docker Compose Environment</h4><p>The complete environment contains six containers:</p><ul><li>Order Service</li><li>Inventory Service</li><li>Order PostgreSQL</li><li>Inventory PostgreSQL</li><li>Apache Kafka</li><li>Kafka UI</li></ul><p>The services are available at:</p><pre>Order Service      http://localhost:8090<br>Inventory Service  http://localhost:8091<br>Kafka UI           http://localhost:8088<br>Kafka              localhost:9092<br>Order Database     localhost:5433<br>Inventory Database localhost:5434</pre><p>Order Service intentionally depends only on Order PostgreSQL.</p><p>It does not require Kafka to be available before starting.</p><p>This is important for the failure demonstration because orders should still be accepted while Kafka is temporarily unavailable.</p><h4>Running the Project</h4><p>First, clone the repository:</p><pre>git clone \<br>https://github.com/erkandemirel/spring-boot-kafka-transactional-outbox-pattern-demo.git</pre><p>Open the project directory:</p><pre>cd spring-boot-kafka-transactional-outbox-pattern-demo</pre><p>Build all Maven modules:</p><pre>mvn clean package</pre><p>Start the complete environment:</p><pre>docker compose up --build</pre><p>Check the running containers:</p><pre>docker compose ps</pre><h4>Testing the Normal Flow</h4><p>Create an order:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-100&quot;,<br>    &quot;quantity&quot;: 2,<br>    &quot;amount&quot;: 249.90<br>  }&#39;</pre><p>The response contains the created order:</p><pre>{<br>  &quot;id&quot;: &quot;02972e1b-e6d5-4178-8042-04703cd07e5c&quot;,<br>  &quot;productId&quot;: &quot;product-100&quot;,<br>  &quot;quantity&quot;: 2,<br>  &quot;amount&quot;: 249.90,<br>  &quot;status&quot;: &quot;CREATED&quot;<br>}</pre><p>Immediately after the request, the database contains:</p><pre>orders<br>└── Order status: CREATED<br><br>outbox_events<br>└── Outbox status: NEW</pre><p>Wait a few seconds for the publisher.</p><p>List the Outbox events:</p><pre>curl http://localhost:8090/outbox-events</pre><p>The status should become:</p><pre>PUBLISHED</pre><p>List the inventory records:</p><pre>curl http://localhost:8091/inventory-records</pre><p>The final state is:</p><pre>Order              CREATED<br>Outbox Event       PUBLISHED<br>Inventory Record   RESERVED</pre><p>The Kafka topic can be inspected using Kafka UI:</p><pre>http://localhost:8088</pre><p>The topic name is:</p><pre>outbox.order-created</pre><h4>Testing Kafka Failure</h4><p>The most important test is creating an order while Kafka is unavailable.</p><p>Stop only the Kafka container:</p><pre>docker compose stop kafka</pre><p>Order Service and PostgreSQL remain available.</p><p>Create another order:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-offline&quot;,<br>    &quot;quantity&quot;: 3,<br>    &quot;amount&quot;: 399.00<br>  }&#39;</pre><p>The request still succeeds.</p><p>Check the orders:</p><pre>curl http://localhost:8090/orders</pre><p>The order exists because creating an order does not require a direct Kafka call.</p><p>Now check the Outbox table:</p><pre>curl http://localhost:8090/outbox-events</pre><p>The latest event has the following state:</p><pre>status: FAILED<br>retryCount: 1<br>lastError: Kafka connection error</pre><p>The event has not been lost.</p><p>Its complete JSON payload remains in PostgreSQL.</p><h4>Testing Automatic Recovery</h4><p>Start Kafka again:</p><pre>docker compose start kafka</pre><p>After Kafka becomes available, the Outbox Publisher selects the failed event again.</p><p>The state changes from:</p><pre>FAILED → PUBLISHED</pre><p>Check the events:</p><pre>curl http://localhost:8090/outbox-events</pre><p>Then check Inventory Service:</p><pre>curl http://localhost:8091/inventory-records</pre><p>The inventory record should now exist:</p><pre>Inventory status: RESERVED</pre><p>The complete recovery flow is:</p><pre>Kafka Unavailable<br>       |<br>       v<br>Order + Outbox Event Saved<br>       |<br>       v<br>Outbox Status: FAILED<br>       |<br>       v<br>Kafka Started<br>       |<br>       v<br>Publisher Retries Event<br>       |<br>       v<br>Outbox Status: PUBLISHED<br>       |<br>       v<br>Inventory Status: RESERVED</pre><p>This demonstrates the main benefit of the Transactional Outbox Pattern.</p><p>The event does not need to be recreated from the order because the original event payload was stored safely during the original database transaction.</p><h4>Inspecting PostgreSQL</h4><p>The Order database can be opened from pgAdmin using:</p><pre>Host: localhost<br>Port: 5433<br>Database: orders<br>Username: orders<br>Password: orders</pre><p>The database contains:</p><pre>orders<br>outbox_events</pre><p>The Inventory database connection is:</p><pre>Host: localhost<br>Port: 5434<br>Database: inventory<br>Username: inventory<br>Password: inventory</pre><p>It contains:</p><pre>inventory_records</pre><p>The Outbox table allows us to inspect:</p><ul><li>Pending events</li><li>Successfully published events</li><li>Failed events</li><li>Retry counts</li><li>Publication timestamps</li><li>Latest errors</li><li>Serialized event payloads</li></ul><h4>Delivery Semantics</h4><p>This implementation provides at-least-once delivery.</p><p>Consider the following situation:</p><pre>Publish Event to Kafka<br>        ↓<br>Kafka Acknowledges Event<br>        ↓<br>Application Stops<br>        ↓<br>Outbox Status Was Not Updated</pre><p>Kafka contains the event, but the Outbox row may still have the NEW or FAILED status.</p><p>After restarting, the publisher sends the event again.</p><p>This means that the Transactional Outbox Pattern prevents lost events, but it does not automatically prevent duplicate events.</p><p>Consumers must be idempotent.</p><p>An idempotent consumer should produce the same result when it receives the same event multiple times.</p><p>For example, Inventory Service should not reserve the same inventory twice.</p><h4>Polling Publisher Advantages</h4><p>The polling approach is simple to understand and implement.</p><p>Its advantages include:</p><ul><li>No additional infrastructure is required</li><li>Events remain visible in PostgreSQL</li><li>Failed events can be retried</li><li>Operational state is easy to inspect</li><li>It works with standard Spring Data JPA</li><li>The polling interval can be configured</li></ul><p>For a demonstration project or a moderate event volume, polling provides a clear implementation of the pattern.</p><h4>Polling Publisher Challenges</h4><p>The polling approach also introduces several considerations:</p><ul><li>Frequent polling increases database activity</li><li>A long polling interval increases event latency</li><li>Multiple publisher instances may select the same rows</li><li>Published records must eventually be archived or deleted</li><li>Failed events need backoff and alerting</li><li>Duplicate delivery must be handled by consumers</li><li>Large event payloads increase Outbox table size</li></ul><p>Another implementation option is Change Data Capture.</p><p>A tool such as Debezium can read committed changes from the PostgreSQL transaction log and publish Outbox events to Kafka.</p><p>That approach can be explored in a separate project.</p><h4>Production Considerations</h4><p>This project intentionally focuses on the core Transactional Outbox flow.</p><p>A production-ready implementation should also consider the following patterns and features.</p><h4>Idempotent Consumer or Inbox Pattern</h4><p>Consumers should persist processed eventId values.</p><p>Before applying a business operation, the consumer should check whether the event was previously processed.</p><p>The event processing result and processed-event record should be committed in the same local database transaction.</p><h4>Safe Event Claiming</h4><p>When multiple Outbox Publisher instances run simultaneously, they may select the same event.</p><p>Rows should be claimed safely using a mechanism such as:</p><pre>SELECT ...<br>FOR UPDATE SKIP LOCKED</pre><p>Another option is assigning a temporary processing status and lease expiration time.</p><h4>Retry Backoff</h4><p>Failed events should not be retried continuously without delay.</p><p>A production implementation should include:</p><ul><li>Exponential backoff</li><li>Maximum retry limits</li><li>Next attempt timestamp</li><li>Failure metrics</li><li>Operational alerts</li></ul><h4>Outbox Cleanup</h4><p>Published events should eventually be archived or deleted.</p><p>Cleanup should be performed in batches to avoid creating large database transactions.</p><h4>Database Migrations</h4><p>The demo uses Hibernate automatic schema updates.</p><p>Production applications should normally manage database schemas with tools such as Flyway or Liquibase.</p><h4>Observability</h4><p>Useful Outbox metrics include:</p><ul><li>Number of NEW events</li><li>Number of FAILED events</li><li>Oldest unpublished event age</li><li>Publication success rate</li><li>Retry count</li><li>Publishing duration</li></ul><p>The eventId and aggregateId should also be included in logs and distributed traces.</p><h4>Event Schema Versioning</h4><p>Event structures change over time.</p><p>Events should include schema version information, and producers should avoid publishing internal JPA entities directly.</p><h4>Change Data Capture</h4><p>For systems with higher event volumes, Change Data Capture can publish Outbox rows without application-level polling.</p><p>Debezium is commonly used to capture committed PostgreSQL changes and send them to Kafka.</p><h4>Transactional Outbox and Saga Pattern</h4><p>The Transactional Outbox Pattern and Saga Pattern solve different problems.</p><p>Saga Pattern manages a distributed business transaction across multiple services.</p><p>Transactional Outbox Pattern ensures that a service’s committed database change has a corresponding event available for publication.</p><p>They are often used together.</p><p>For example:</p><pre>Order Service<br>    |<br>    | Save Order + OrderCreatedEvent<br>    | in one local transaction<br>    v<br>Transactional Outbox<br>    |<br>    v<br>Kafka<br>    |<br>    v<br>Saga Participants</pre><p>In the previous Saga project, Order Service saved an order and published an event directly.</p><p>Adding the Transactional Outbox Pattern closes the consistency gap between those two operations.</p><p>Saga manages business consistency across services.</p><p>Transactional Outbox manages reliable event publication inside each service.</p><h4>Conclusion</h4><p>The Transactional Outbox Pattern solves the dual-write problem that appears when a microservice needs to update its database and publish an event.</p><p>In this example:</p><ul><li>Order Service stores an order.</li><li>The same transaction stores an Outbox event.</li><li>A scheduled publisher reads pending events.</li><li>Kafka receives the serialized event.</li><li>Inventory Service consumes the event.</li><li>Failed publications remain in PostgreSQL.</li><li>Events are retried after Kafka recovers.</li></ul><p>The pattern does not create a distributed transaction between PostgreSQL and Kafka.</p><p>Instead, it guarantees that the information required to publish the event is committed together with the business data.</p><p>This prevents database changes from becoming invisible to downstream services.</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-kafka-transactional-outbox-pattern-demo">Spring Boot Kafka Transactional Outbox Pattern Demo</a></p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b51ebd433d2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Orchestration-Based Saga Pattern with Spring Boot and Kafka]]></title>
            <link>https://erkndmrl.medium.com/orchestration-based-saga-pattern-with-spring-boot-and-kafka-6a02f50a8d49?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a02f50a8d49</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[saga-pattern]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 11:41:56 GMT</pubDate>
            <atom:updated>2026-09-08T11:41:56.317Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H5j61YXuLdLla6_OuJFSqQ.png" /></figure><h4>Managing Distributed Transactions with a Central Saga Orchestrator</h4><p>Microservices usually own separate databases. This provides loose coupling and independent deployment, but it also introduces an important problem:</p><p><strong>How can we maintain data consistency when a business operation spans multiple microservices?</strong></p><p>A traditional database transaction cannot cover multiple independently deployed services and databases. The Saga Pattern solves this problem by dividing a distributed transaction into a sequence of local transactions.</p><p>In this article, we will implement an <strong>Orchestration-Based Saga Pattern</strong> using:</p><ul><li>Java 17</li><li>Spring Boot 3.5</li><li>Apache Kafka</li><li>Spring Kafka</li><li>Spring Data JPA</li><li>PostgreSQL</li><li>Docker Compose</li><li>Kafka UI</li></ul><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-kafka-orchestration-saga-pattern-demo">erkandemirel/spring-boot-kafka-orchestration-saga-pattern-demo</a></p><h4>What Is the Saga Pattern?</h4><p>A Saga represents a long-running business transaction consisting of multiple local transactions.</p><p>Each service:</p><ol><li>Performs its own local database transaction.</li><li>Publishes a result event.</li><li>Triggers or waits for the next step.</li><li>Executes a compensating transaction if the process fails.</li></ol><p>There are two common Saga implementations:</p><ul><li>Choreography-Based Saga</li><li>Orchestration-Based Saga</li></ul><p>In a choreography-based Saga, services react directly to each other’s events. There is no central coordinator.</p><p>In an orchestration-based Saga, a central component called the <strong>Saga Orchestrator</strong> controls the workflow. It decides which service should execute next and what should happen when a step fails.</p><p>This project focuses on the orchestration approach.</p><h4>Example Scenario</h4><p>Our demo contains four applications:</p><ul><li>Order Service</li><li>Payment Service</li><li>Inventory Service</li><li>Saga Orchestrator Service</li></ul><p>Each service owns a separate PostgreSQL database.</p><p>When a new order is created, the following process starts:</p><pre>Client<br>  |<br>  v<br>Order Service<br>  |<br>  | OrderCreatedEvent<br>  v<br>Saga Orchestrator<br>  |<br>  | ProcessPaymentCommand<br>  v<br>Payment Service<br>  |<br>  | PaymentCompletedEvent<br>  v<br>Saga Orchestrator<br>  |<br>  | ReserveInventoryCommand<br>  v<br>Inventory Service<br>  |<br>  | InventoryReservedEvent<br>  v<br>Saga Orchestrator<br>  |<br>  | CompleteOrderCommand<br>  v<br>Order Service</pre><p>The important difference from choreography is that the Payment and Inventory services do not decide what happens next.</p><p>They only:</p><ul><li>Receive a command.</li><li>Execute a local transaction.</li><li>Publish the result.</li></ul><p>The Saga Orchestrator receives that result and selects the next action.</p><h4>Project Structure</h4><p>The application is organized as a Maven multi-module project:</p><pre>spring-boot-kafka-orchestration-saga-pattern-demo<br>├── saga-messages<br>├── order-service<br>├── payment-service<br>├── inventory-service<br>├── saga-orchestrator-service<br>├── docker-compose.yml<br>└── pom.xml</pre><h4>saga-messages</h4><p>This module contains the commands, events, and Kafka topic names shared between services.</p><p>Example commands:</p><pre>ProcessPaymentCommand<br>RefundPaymentCommand<br>ReserveInventoryCommand<br>CompleteOrderCommand<br>CancelOrderCommand</pre><p>Example events:</p><pre>OrderCreatedEvent<br>PaymentCompletedEvent<br>PaymentFailedEvent<br>InventoryReservedEvent<br>InventoryFailedEvent<br>PaymentRefundedEvent<br>OrderCompletedEvent<br>OrderCancelledEvent</pre><p>Commands express an instruction:</p><p><em>Process this payment.</em></p><p>Events express something that has already happened:</p><p><em>The payment was completed.</em></p><p>This distinction makes the orchestration flow easier to understand.</p><h4>Creating an Order</h4><p>The Saga starts when the client sends a request to the Order Service:</p><pre>POST http://localhost:8090/orders<br>Content-Type: application/json<br><br>{<br>  &quot;productId&quot;: &quot;product-100&quot;,<br>  &quot;quantity&quot;: 2,<br>  &quot;amount&quot;: 250.00<br>}</pre><p>The Order Service:</p><ol><li>Generates an orderId.</li><li>Generates a sagaId.</li><li>Saves the order with the PENDING status.</li><li>Publishes an OrderCreatedEvent.</li></ol><p>A simplified version looks like this:</p><pre>@PostMapping<br>@ResponseStatus(HttpStatus.ACCEPTED)<br>@Transactional<br>public OrderEntity create(@Valid @RequestBody CreateOrderRequest request) {<br>    UUID orderId = UUID.randomUUID();<br>    UUID sagaId = UUID.randomUUID();<br><br>    OrderEntity order = repository.save(<br>        new OrderEntity(<br>            orderId,<br>            sagaId,<br>            request.productId(),<br>            request.quantity(),<br>            request.amount()<br>        )<br>    );<br><br>    kafkaTemplate.send(<br>        SagaTopics.ORDER_CREATED,<br>        sagaId.toString(),<br>        new OrderCreatedEvent(<br>            sagaId,<br>            orderId,<br>            request.productId(),<br>            request.quantity(),<br>            request.amount()<br>        )<br>    );<br><br>    return order;<br>}</pre><p>The sagaId is used as the Kafka message key and as the correlation identifier across the entire workflow.</p><h4>Starting the Saga</h4><p>The Saga Orchestrator listens for OrderCreatedEvent.</p><p>When it receives the event, it:</p><ol><li>Creates a Saga instance.</li><li>Stores the current Saga state.</li><li>Sends a ProcessPaymentCommand.</li></ol><pre>@KafkaListener(<br>    topics = SagaTopics.ORDER_CREATED,<br>    groupId = &quot;saga-orchestrator&quot;<br>)<br>@Transactional<br>public void onOrderCreated(OrderCreatedEvent event) {<br>    if (repository.existsById(event.sagaId())) {<br>        return;<br>    }<br><br>    repository.save(<br>        new SagaInstance(<br>            event.sagaId(),<br>            event.orderId(),<br>            event.productId(),<br>            event.quantity(),<br>            event.amount()<br>        )<br>    );<br><br>    kafkaTemplate.send(<br>        SagaTopics.PROCESS_PAYMENT,<br>        event.sagaId().toString(),<br>        new ProcessPaymentCommand(<br>            event.sagaId(),<br>            event.orderId(),<br>            event.productId(),<br>            event.quantity(),<br>            event.amount()<br>        )<br>    );<br>}</pre><p>The duplicate check provides basic protection against processing the same OrderCreatedEvent more than once.</p><h4>Processing the Payment</h4><p>The Payment Service listens for ProcessPaymentCommand.</p><p>For demonstration purposes, the project uses the following rule:</p><pre>amount &lt;= 1000  -&gt; payment succeeds<br>amount &gt; 1000   -&gt; payment fails</pre><p>When payment succeeds, the service saves the payment and publishes a PaymentCompletedEvent.</p><p>When it fails, it saves a failed payment record and publishes a PaymentFailedEvent.</p><p>The Payment Service does not send a command to the Inventory Service. That responsibility belongs to the orchestrator.</p><h4>Reserving Inventory</h4><p>After receiving PaymentCompletedEvent, the orchestrator changes the Saga status to INVENTORY_RESERVING and sends a ReserveInventoryCommand.</p><p>The Inventory Service uses this demo rule:</p><pre>quantity &lt;= 5  -&gt; inventory is reserved<br>quantity &gt; 5   -&gt; inventory reservation fails</pre><p>A successful reservation produces an InventoryReservedEvent.</p><p>The orchestrator then sends a CompleteOrderCommand to the Order Service.</p><p>After the order is completed, the Saga reaches its final successful state:</p><pre>Order       COMPLETED<br>Payment     COMPLETED<br>Reservation RESERVED<br>Saga        COMPLETED</pre><h4>What Happens When Payment Fails?</h4><p>If the payment amount is greater than 1000, the Payment Service publishes a PaymentFailedEvent.</p><p>The orchestrator receives this event and sends a CancelOrderCommand.</p><p>The final state becomes:</p><pre>Order   CANCELLED<br>Payment FAILED<br>Saga    FAILED</pre><p>The Saga is marked as FAILED because the payment was never completed. There is no completed payment transaction that needs to be reversed.</p><h4>Compensation After an Inventory Failure</h4><p>Now consider a more interesting scenario:</p><ol><li>The order is created.</li><li>The payment succeeds.</li><li>Inventory reservation fails.</li></ol><p>At this point, simply marking the Saga as failed would be incorrect because the payment has already been completed.</p><p>The orchestrator must execute a compensating transaction.</p><pre>InventoryFailedEvent<br>        |<br>        v<br>Saga Orchestrator<br>        |<br>        | RefundPaymentCommand<br>        v<br>Payment Service<br>        |<br>        | PaymentRefundedEvent<br>        v<br>Saga Orchestrator<br>        |<br>        | CancelOrderCommand<br>        v<br>Order Service<br>        |<br>        | OrderCancelledEvent<br>        v<br>Saga Orchestrator<br>        |<br>        v<br>Saga status: COMPENSATED</pre><p>When inventory fails, the orchestrator sends a refund command:</p><pre>@KafkaListener(<br>    topics = SagaTopics.INVENTORY_FAILED,<br>    groupId = &quot;saga-orchestrator&quot;<br>)<br>@Transactional<br>public void onInventoryFailed(InventoryFailedEvent event) {<br>    repository.findById(event.sagaId()).ifPresent(saga -&gt; {<br>        saga.startCompensation(event.reason());<br><br>        kafkaTemplate.send(<br>            SagaTopics.REFUND_PAYMENT,<br>            event.sagaId().toString(),<br>            new RefundPaymentCommand(<br>                event.sagaId(),<br>                event.orderId(),<br>                event.paymentId(),<br>                &quot;Compensation after inventory failure&quot;<br>            )<br>        );<br>    });<br>}</pre><p>After the payment is refunded, the orchestrator cancels the order.</p><p>The final state is:</p><pre>Order       CANCELLED<br>Payment     REFUNDED<br>Reservation FAILED<br>Saga        COMPENSATED</pre><p>COMPENSATED means that the Saga could not complete successfully, but the previously completed business operations were reversed using compensating transactions.</p><p>It is important to understand that compensation is not a database rollback.</p><p>A refund is a new business operation that compensates for the previously completed payment.</p><h4>Saga State Machine</h4><p>The orchestrator persists every Saga instance in its own PostgreSQL database.</p><p>The demo uses the following states:</p><pre>PAYMENT_PROCESSING<br>INVENTORY_RESERVING<br>ORDER_COMPLETING<br>COMPENSATING<br>ORDER_CANCELLING<br>COMPLETED<br>COMPENSATED<br>FAILED</pre><p>A successful Saga follows this path:</p><pre>PAYMENT_PROCESSING<br>        |<br>        v<br>INVENTORY_RESERVING<br>        |<br>        v<br>ORDER_COMPLETING<br>        |<br>        v<br>COMPLETED</pre><p>An inventory failure follows this path:</p><pre>PAYMENT_PROCESSING<br>        |<br>        v<br>INVENTORY_RESERVING<br>        |<br>        v<br>COMPENSATING<br>        |<br>        v<br>ORDER_CANCELLING<br>        |<br>        v<br>COMPENSATED</pre><p>Persisting the state makes the distributed workflow observable and provides a central place to inspect its progress.</p><h4>Running the Project</h4><p>First, clone the repository:</p><pre>git clone https://github.com/erkandemirel/spring-boot-kafka-orchestration-saga-pattern-demo.git<br>cd spring-boot-kafka-orchestration-saga-pattern-demo</pre><p>Build the project:</p><pre>mvn clean package</pre><p>Start all services:</p><pre>docker compose up --build</pre><p>The following applications will be available:</p><pre>Order Service        http://localhost:8090<br>Payment Service      http://localhost:8091<br>Inventory Service    http://localhost:8092<br>Saga Orchestrator    http://localhost:8093<br>Kafka UI             http://localhost:8088</pre><h4>Testing a Successful Saga</h4><p>Create an order with a valid payment amount and inventory quantity:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;productId&quot;:&quot;product-100&quot;,&quot;quantity&quot;:2,&quot;amount&quot;:250.00}&#39;</pre><p>Expected result:</p><pre>Order       COMPLETED<br>Payment     COMPLETED<br>Reservation RESERVED<br>Saga        COMPLETED</pre><h4>Testing a Payment Failure</h4><p>Create an order with an amount greater than 1000:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;productId&quot;:&quot;product-200&quot;,&quot;quantity&quot;:2,&quot;amount&quot;:1500.00}&#39;</pre><p>Expected result:</p><pre>Order   CANCELLED<br>Payment FAILED<br>Saga    FAILED</pre><h4>Testing Compensation</h4><p>Create an order with a quantity greater than 5:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;productId&quot;:&quot;product-300&quot;,&quot;quantity&quot;:10,&quot;amount&quot;:300.00}&#39;</pre><p>Payment succeeds, but inventory reservation fails. The orchestrator refunds the payment and cancels the order.</p><p>Expected result:</p><pre>Order       CANCELLED<br>Payment     REFUNDED<br>Reservation FAILED<br>Saga        COMPENSATED</pre><h4>Inspecting the Results</h4><p>The current states can be inspected through the REST endpoints:</p><pre>GET http://localhost:8090/orders<br>GET http://localhost:8091/payments<br>GET http://localhost:8092/reservations<br>GET http://localhost:8093/sagas</pre><p>Kafka UI is available at:</p><pre>http://localhost:8088</pre><p>Kafka UI can be used to inspect all commands and events exchanged during the Saga.</p><h4>Orchestration vs. Choreography</h4><p>The main advantage of orchestration is visibility.</p><p>The complete business workflow is controlled from one place. This makes it easier to understand:</p><ul><li>Which step is currently running</li><li>Which command should be sent next</li><li>Why a Saga failed</li><li>Which compensation should be executed</li><li>Whether a Saga completed or was compensated</li></ul><p>However, orchestration also has trade-offs.</p><p>The orchestrator contains workflow knowledge and may become a central point of complexity. It should coordinate the process without containing the internal business logic of participant services.</p><p>Payment logic should remain in the Payment Service, and inventory logic should remain in the Inventory Service.</p><h4>Production Considerations</h4><p>This project focuses on demonstrating the core orchestration flow. A production-ready implementation should also consider:</p><ul><li>Transactional Outbox Pattern</li><li>Idempotent command and event processing</li><li>Kafka retry and dead-letter topics</li><li>Optimistic locking for Saga state</li><li>Message schema versioning</li><li>Distributed tracing and metrics</li><li>Timeouts for unresponsive services</li><li>Secure handling of payment operations</li></ul><p>A Spring database transaction does not automatically make a database update and Kafka publication atomic. The Transactional Outbox Pattern can be used to close this consistency gap.</p><h4>Conclusion</h4><p>The Orchestration-Based Saga Pattern provides an explicit way to coordinate distributed transactions across microservices.</p><p>In this example:</p><ul><li>The Order Service starts the process.</li><li>The Saga Orchestrator controls the workflow.</li><li>The Payment and Inventory services execute local transactions.</li><li>Kafka carries commands and result events.</li><li>PostgreSQL stores service-owned data.</li><li>Compensating transactions restore business consistency after failures.</li></ul><p>The most important idea is that distributed transactions are not rolled back as a single database transaction. Instead, every service commits its own operation, and previously completed operations are reversed through explicit business actions when necessary.</p><p>Source code: <a href="https://github.com/erkandemirel/spring-boot-kafka-orchestration-saga-pattern-demo">Spring Boot Kafka Orchestration Saga Pattern Demo</a></p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a02f50a8d49" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choreography-Based Saga Pattern with Spring Boot and Kafka]]></title>
            <link>https://erkndmrl.medium.com/saga-pattern-with-spring-boot-and-kafka-e9607103b25a?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9607103b25a</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[saga-pattern]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Mon, 24 Aug 2026 19:04:03 GMT</pubDate>
            <atom:updated>2026-08-24T19:14:18.926Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jWNsnll69Wes_ZuqJu0Yfw.png" /></figure><p>In a microservice architecture, a single business operation may involve multiple services.</p><p>For example, creating an order may require:</p><ul><li>Creating the order</li><li>Processing the payment</li><li>Reserving inventory</li><li>Completing or cancelling the order</li></ul><p>When all these operations use the same database, a traditional database transaction can roll everything back if something fails.</p><p>However, in a microservice architecture, each service usually owns its database. A single database transaction cannot cover the entire operation.</p><p>This is where the Saga Pattern becomes useful.</p><p>In this article, we will build a simple choreography-based Saga implementation using Spring Boot, Apache Kafka, PostgreSQL, and Docker Compose.</p><p>The complete source code is available on <a href="https://github.com/erkandemirel/spring-boot-kafka-choreography-saga-pattern-demo">GitHub</a>.</p><h4>What Is the Saga Pattern?</h4><p>A Saga represents a distributed business transaction as a sequence of local transactions.</p><p>Each service:</p><ol><li>Performs a local database transaction.</li><li>Publishes an event.</li><li>Allows another service to continue the process.</li></ol><p>If one of the steps fails, previously completed operations are reversed using compensating transactions.</p><p>According to the <a href="https://microservices.io/patterns/data/saga.html">Saga Pattern description</a>, Saga coordination can be implemented in two main ways:</p><ul><li>Choreography</li><li>Orchestration</li></ul><h4>Choreography</h4><p>In choreography, there is no central coordinator.</p><p>Each service listens for events, performs its local operation, and publishes the next event.</p><pre>OrderCreated<br>    ↓<br>PaymentCompleted<br>    ↓<br>InventoryReserved<br>    ↓<br>OrderCompleted</pre><p>Services know which events they must consume and which events they must publish.</p><h4>Orchestration</h4><p>In orchestration, a central Saga orchestrator controls the workflow and tells each service what to do.</p><p>This project uses choreography because it provides a simple example of an event-driven Saga using Kafka.</p><h4>Project Scenario</h4><p>The project contains three microservices:</p><ul><li>Order Service</li><li>Payment Service</li><li>Inventory Service</li></ul><p>Each service has its own PostgreSQL database.</p><p>The complete successful flow is:</p><pre>Order Service<br>    |<br>    | OrderCreatedEvent<br>    v<br>Payment Service<br>    |<br>    | PaymentCompletedEvent<br>    v<br>Inventory Service<br>    |<br>    | InventoryReservedEvent<br>    v<br>Order Service</pre><p>The final order status becomes COMPLETED.</p><p>The failure flow is different:</p><pre>PaymentFailedEvent<br>    |<br>    v<br>Order Service → CANCELLED</pre><p>If payment succeeds but inventory reservation fails:</p><pre>InventoryFailedEvent<br>    |<br>    +----&gt; Order Service → CANCELLED<br>    |<br>    +----&gt; Payment Service → REFUNDED</pre><p>Refunding the completed payment is the compensating transaction in this example.</p><h4>Technologies</h4><p>The project uses:</p><ul><li>Java 17</li><li>Spring Boot 3.5</li><li>Spring Kafka</li><li>Spring Data JPA</li><li>Apache Kafka 4.1</li><li>PostgreSQL</li><li>Docker Compose</li><li>Kafka UI</li></ul><p>Kafka runs in KRaft mode, so ZooKeeper is not required. Apache Kafka documents combined broker and controller mode as a convenient configuration for development environments, although production environments should normally separate these roles. You can find more information in the <a href="https://kafka.apache.org/41/operations/kraft/">Apache Kafka KRaft documentation</a>.</p><h4>Project Structure</h4><p>The project is implemented as a Maven multi-module application.</p><pre>spring-boot-kafka-saga-pattern-demo<br>├── saga-events<br>├── order-service<br>├── payment-service<br>├── inventory-service<br>├── docker-compose.yml<br>└── pom.xml</pre><p>The saga-events module contains event definitions shared by the services.</p><p>The root pom.xml defines all modules:</p><pre>&lt;modules&gt;<br>    &lt;module&gt;saga-events&lt;/module&gt;<br>    &lt;module&gt;order-service&lt;/module&gt;<br>    &lt;module&gt;payment-service&lt;/module&gt;<br>    &lt;module&gt;inventory-service&lt;/module&gt;<br>&lt;/modules&gt;</pre><h4>Saga Events</h4><p>The services communicate through the following Kafka topics:</p><pre>saga.order-created<br>saga.payment-completed<br>saga.payment-failed<br>saga.inventory-reserved<br>saga.inventory-failed<br>saga.payment-refunded</pre><p>Topic names are defined in one class:</p><pre>public final class SagaTopics {<br><br>    public static final String ORDER_CREATED =<br>            &quot;saga.order-created&quot;;<br><br>    public static final String PAYMENT_COMPLETED =<br>            &quot;saga.payment-completed&quot;;<br><br>    public static final String PAYMENT_FAILED =<br>            &quot;saga.payment-failed&quot;;<br><br>    public static final String INVENTORY_RESERVED =<br>            &quot;saga.inventory-reserved&quot;;<br><br>    public static final String INVENTORY_FAILED =<br>            &quot;saga.inventory-failed&quot;;<br><br>    public static final String PAYMENT_REFUNDED =<br>            &quot;saga.payment-refunded&quot;;<br><br>    private SagaTopics() {<br>    }<br>}</pre><p>Each event carries a sagaId.</p><p>For example, the order-created event is defined as:</p><pre>public record OrderCreatedEvent(<br>        UUID sagaId,<br>        UUID orderId,<br>        String productId,<br>        int quantity,<br>        BigDecimal amount<br>) {<br>}</pre><p>The sagaId allows us to associate all events belonging to the same business transaction.</p><h4>Creating the Order</h4><p>The Saga begins when a client creates an order.</p><pre>@PostMapping<br>@ResponseStatus(HttpStatus.ACCEPTED)<br>@Transactional<br>public OrderEntity create(@Valid @RequestBody CreateOrderRequest request) {<br>    UUID orderId = UUID.randomUUID();<br>    UUID sagaId = UUID.randomUUID();<br><br>    OrderEntity order = repository.save(<br>            new OrderEntity(<br>                    orderId,<br>                    sagaId,<br>                    request.productId(),<br>                    request.quantity(),<br>                    request.amount()<br>            )<br>    );<br><br>    kafkaTemplate.send(<br>            SagaTopics.ORDER_CREATED,<br>            sagaId.toString(),<br>            new OrderCreatedEvent(<br>                    sagaId,<br>                    orderId,<br>                    request.productId(),<br>                    request.quantity(),<br>                    request.amount()<br>            )<br>    );<br><br>    return order;<br>}</pre><p>The initial order status is PENDING.</p><p>After saving the order, Order Service publishes an OrderCreatedEvent.</p><p>The sagaId is also used as the Kafka message key. This helps events belonging to the same Saga use consistent partitioning.</p><h4>Processing the Payment</h4><p>Payment Service listens for OrderCreatedEvent.</p><p>Spring Kafka supports consuming messages using the @KafkaListener annotation, as described in the <a href="https://docs.spring.io/spring-kafka/reference/kafka/receiving-messages.html">Spring Kafka documentation</a>.</p><pre>@KafkaListener(<br>        topics = SagaTopics.ORDER_CREATED,<br>        groupId = &quot;payment-service&quot;<br>)<br>@Transactional<br>public void onOrderCreated(OrderCreatedEvent event) {<br><br>    if (event.amount().compareTo(PAYMENT_LIMIT) &gt; 0) {<br><br>        repository.save(<br>                new PaymentEntity(<br>                        UUID.randomUUID(),<br>                        event.sagaId(),<br>                        event.orderId(),<br>                        event.amount(),<br>                        PaymentStatus.FAILED,<br>                        &quot;Payment amount exceeds demo limit&quot;<br>                )<br>        );<br><br>        kafkaTemplate.send(<br>                SagaTopics.PAYMENT_FAILED,<br>                event.sagaId().toString(),<br>                new PaymentFailedEvent(<br>                        event.sagaId(),<br>                        event.orderId(),<br>                        &quot;Payment was declined&quot;<br>                )<br>        );<br><br>        return;<br>    }<br><br>    UUID paymentId = UUID.randomUUID();<br><br>    repository.save(<br>            new PaymentEntity(<br>                    paymentId,<br>                    event.sagaId(),<br>                    event.orderId(),<br>                    event.amount(),<br>                    PaymentStatus.COMPLETED,<br>                    null<br>            )<br>    );<br><br>    kafkaTemplate.send(<br>            SagaTopics.PAYMENT_COMPLETED,<br>            event.sagaId().toString(),<br>            new PaymentCompletedEvent(<br>                    event.sagaId(),<br>                    event.orderId(),<br>                    paymentId,<br>                    event.productId(),<br>                    event.quantity(),<br>                    event.amount()<br>            )<br>    );<br>}</pre><p>For demonstration purposes, the payment rule is simple:</p><pre>amount &lt;= 1000 → COMPLETED<br>amount &gt; 1000  → FAILED</pre><p>If payment fails, Payment Service publishes PaymentFailedEvent.</p><p>Order Service consumes this event and changes the order status to CANCELLED.</p><p>If payment succeeds, Payment Service publishes PaymentCompletedEvent, and Inventory Service continues the Saga.</p><h4>Reserving Inventory</h4><p>Inventory Service listens for PaymentCompletedEvent.</p><pre>@KafkaListener(<br>        topics = SagaTopics.PAYMENT_COMPLETED,<br>        groupId = &quot;inventory-service&quot;<br>)<br>@Transactional<br>public void onPaymentCompleted(PaymentCompletedEvent event) {<br>    UUID reservationId = UUID.randomUUID();<br><br>    if (event.quantity() &gt; AVAILABLE_STOCK) {<br><br>        String reason =<br>                &quot;Insufficient stock; available quantity is &quot;<br>                        + AVAILABLE_STOCK;<br><br>        repository.save(<br>                new InventoryReservation(<br>                        reservationId,<br>                        event.sagaId(),<br>                        event.orderId(),<br>                        event.productId(),<br>                        event.quantity(),<br>                        ReservationStatus.FAILED,<br>                        reason<br>                )<br>        );<br><br>        kafkaTemplate.send(<br>                SagaTopics.INVENTORY_FAILED,<br>                event.sagaId().toString(),<br>                new InventoryFailedEvent(<br>                        event.sagaId(),<br>                        event.orderId(),<br>                        event.paymentId(),<br>                        reason<br>                )<br>        );<br><br>        return;<br>    }<br><br>    repository.save(<br>            new InventoryReservation(<br>                    reservationId,<br>                    event.sagaId(),<br>                    event.orderId(),<br>                    event.productId(),<br>                    event.quantity(),<br>                    ReservationStatus.RESERVED,<br>                    null<br>            )<br>    );<br><br>    kafkaTemplate.send(<br>            SagaTopics.INVENTORY_RESERVED,<br>            event.sagaId().toString(),<br>            new InventoryReservedEvent(<br>                    event.sagaId(),<br>                    event.orderId(),<br>                    event.paymentId(),<br>                    event.productId(),<br>                    event.quantity()<br>            )<br>    );<br>}</pre><p>The inventory rule is:</p><pre>quantity &lt;= 5 → RESERVED<br>quantity &gt; 5  → FAILED</pre><p>If inventory reservation succeeds, Inventory Service publishes InventoryReservedEvent.</p><p>Order Service consumes this event and completes the order.</p><pre>@KafkaListener(<br>        topics = SagaTopics.INVENTORY_RESERVED,<br>        groupId = &quot;order-service&quot;<br>)<br>@Transactional<br>public void onInventoryReserved(InventoryReservedEvent event) {<br>    repository.findById(event.orderId())<br>            .ifPresent(order -&gt; {<br>                order.complete();<br><br>                log.info(<br>                        &quot;Saga {} completed for order {}&quot;,<br>                        event.sagaId(),<br>                        event.orderId()<br>                );<br>            });<br>}</pre><h4>Compensating the Payment</h4><p>The most important failure scenario happens when payment succeeds but inventory reservation fails.</p><p>At this point, we cannot use a traditional rollback because Payment Service has already committed its local database transaction.</p><p>Instead, Inventory Service publishes InventoryFailedEvent.</p><p>Payment Service consumes this event and performs a compensating transaction.</p><pre>@KafkaListener(<br>        topics = SagaTopics.INVENTORY_FAILED,<br>        groupId = &quot;payment-service&quot;<br>)<br>@Transactional<br>public void onInventoryFailed(InventoryFailedEvent event) {<br>    repository.findById(event.paymentId())<br>            .ifPresent(payment -&gt; {<br><br>                if (payment.getStatus()<br>                        == PaymentStatus.COMPLETED) {<br><br>                    payment.refund(<br>                            &quot;Compensation after inventory failure&quot;<br>                    );<br><br>                    kafkaTemplate.send(<br>                            SagaTopics.PAYMENT_REFUNDED,<br>                            event.sagaId().toString(),<br>                            new PaymentRefundedEvent(<br>                                    event.sagaId(),<br>                                    event.orderId(),<br>                                    payment.getId(),<br>                                    event.reason()<br>                            )<br>                    );<br>                }<br>            });<br>}</pre><p>The payment status changes from:</p><pre>COMPLETED → REFUNDED</pre><p>This is not a technical rollback of the original database transaction.</p><p>It is a new business operation that compensates for the previously completed payment.</p><p>At the same time, Order Service consumes InventoryFailedEvent and changes the order status to CANCELLED.</p><h4>Running the Project</h4><p>First, build all Maven modules:</p><pre>mvn clean package</pre><p>Then start Kafka, PostgreSQL databases, Kafka UI, and all services:</p><pre>docker compose up --build</pre><p>The services are available at:</p><pre>Order Service     http://localhost:8090<br>Payment Service   http://localhost:8091<br>Inventory Service http://localhost:8092<br>Kafka UI          http://localhost:8088</pre><p><strong>Testing a Successful Saga</strong></p><p>Create an order with a valid payment amount and inventory quantity:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-100&quot;,<br>    &quot;quantity&quot;: 2,<br>    &quot;amount&quot;: 250.00<br>  }&#39;</pre><p>The event flow will be:</p><pre>OrderCreatedEvent<br>    ↓<br>PaymentCompletedEvent<br>    ↓<br>InventoryReservedEvent</pre><p>The final states will be:</p><pre>Order       COMPLETED<br>Payment     COMPLETED<br>Reservation RESERVED</pre><h4>Testing Payment Failure</h4><p>Create an order with an amount greater than 1000:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-200&quot;,<br>    &quot;quantity&quot;: 2,<br>    &quot;amount&quot;: 1500.00<br>  }&#39;</pre><p>Payment Service publishes PaymentFailedEvent.</p><p>The final states will be:</p><pre>Order   CANCELLED<br>Payment FAILED</pre><p>Inventory Service does not receive a payment-completed event, so inventory reservation is not attempted.</p><h4>Testing Inventory Failure and Compensation</h4><p>Create an order with a quantity greater than 5:</p><pre>curl -X POST http://localhost:8090/orders \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-300&quot;,<br>    &quot;quantity&quot;: 10,<br>    &quot;amount&quot;: 300.00<br>  }&#39;</pre><p>Payment succeeds, but inventory reservation fails.</p><p>The event flow will be:</p><pre>OrderCreatedEvent<br>    ↓<br>PaymentCompletedEvent<br>    ↓<br>InventoryFailedEvent<br>    ├── Order Service cancels the order<br>    └── Payment Service refunds the payment</pre><p>The final states will be:</p><pre>Order       CANCELLED<br>Payment     REFUNDED<br>Reservation FAILED</pre><p>This scenario demonstrates the compensating transaction.</p><h4>Inspecting the Final State</h4><p>All created orders can be listed with:</p><pre>curl http://localhost:8090/orders</pre><p>Payments can be listed with:</p><pre>curl http://localhost:8091/payments</pre><p>Inventory reservations can be listed with:</p><pre>curl http://localhost:8092/reservations</pre><p>Kafka topics and messages can also be inspected using Kafka UI:</p><pre>http://localhost:8088</pre><p>You should see the following topics:</p><pre>saga.order-created<br>saga.payment-completed<br>saga.payment-failed<br>saga.inventory-reserved<br>saga.inventory-failed<br>saga.payment-refunded</pre><h4>Database per Service</h4><p>Each service owns a separate PostgreSQL database:</p><pre>Order Service     → Order Database<br>Payment Service   → Payment Database<br>Inventory Service → Inventory Database</pre><p>Services do not access each other’s tables.</p><p>They communicate only through Kafka events.</p><p>This separation is important because sharing the same database would tightly couple the services and hide the distributed transaction problem that Saga is designed to solve.</p><h4>Choreography Advantages</h4><p>The choreography approach provides several advantages:</p><ul><li>There is no central Saga coordinator.</li><li>Services are loosely coupled through events.</li><li>New consumers can subscribe to existing events.</li><li>Individual services can be scaled independently.</li><li>The business workflow is naturally event-driven.</li></ul><h4>Choreography Challenges</h4><p>Choreography also introduces some challenges:</p><ul><li>The complete workflow is distributed across multiple services.</li><li>Understanding the event flow can become difficult.</li><li>Duplicate events must be handled safely.</li><li>Events may be delivered more than once.</li><li>Failed messages require retry and dead-letter handling.</li><li>Compensating transactions must be explicitly designed.</li><li>Monitoring requires correlation using identifiers such as sagaId.</li></ul><p>For small and medium workflows, choreography can be simple and effective.</p><p>For long or highly complex workflows, an orchestrator may make the process easier to understand and control.</p><h4>Production Considerations</h4><p>This project intentionally focuses on the core Saga flow.</p><p>A production-ready system should also consider the following patterns and features.</p><h4>Transactional Outbox Pattern</h4><p>Saving database data and publishing a Kafka event are two separate operations.</p><p>For example, the order may be committed to PostgreSQL while publishing OrderCreatedEvent fails.</p><p>The Transactional Outbox Pattern solves this problem by saving the business data and event data in the same local database transaction. A separate publisher then sends the event to Kafka.</p><h4>Idempotent Consumers</h4><p>Kafka consumers may receive the same event more than once.</p><p>Each consumer should detect previously processed events and avoid repeating operations such as:</p><ul><li>Charging a payment twice</li><li>Refunding the same payment twice</li><li>Reserving the same inventory twice</li></ul><h4>Retry and Dead Letter Topics</h4><p>Temporary errors should be retried.</p><p>Messages that continue to fail after multiple attempts should be sent to a Dead Letter Topic for investigation.</p><h4>Event Schema Versioning</h4><p>Events change over time.</p><p>Adding version information and using a schema management strategy helps services evolve without breaking existing consumers.</p><h4>Observability</h4><p>The sagaId should be included in:</p><ul><li>Application logs</li><li>Distributed traces</li><li>Metrics</li><li>Kafka message headers</li></ul><p>This makes it possible to follow one Saga across all participating services.</p><h4>Conclusion</h4><p>The Saga Pattern helps maintain business consistency when a transaction spans multiple microservices and databases.</p><p>In this example:</p><ul><li>Order Service starts the Saga.</li><li>Payment Service processes the payment.</li><li>Inventory Service reserves the product.</li><li>Kafka events connect the services.</li><li>Payment failure cancels the order.</li><li>Inventory failure cancels the order and refunds the payment.</li></ul><p>The most important point is that Saga does not provide an automatic distributed rollback.</p><p>Each service completes its own local transaction, and failures are handled through explicitly designed compensating actions.</p><p>This makes Saga more than a technical messaging pattern. It represents how the business should recover when only part of a distributed operation succeeds.</p><p>The complete source code can be found here:</p><p>Spring Boot Kafka Saga Pattern Demo — GitHub</p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9607103b25a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spring Boot Microservices Observability with OpenTelemetry, Zipkin, Prometheus, and Grafana]]></title>
            <link>https://erkndmrl.medium.com/spring-boot-microservices-observability-with-opentelemetry-zipkin-prometheus-and-grafana-b3d41d673910?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3d41d673910</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[zipkin]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Fri, 14 Aug 2026 08:04:40 GMT</pubDate>
            <atom:updated>2026-08-14T08:04:40.756Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dhd0QYnrG4exAHV2el9IvA.png" /></figure><p>Observability is an important part of a microservice architecture.</p><p>When an application consists of multiple services, checking a single log file is usually not enough. A request may enter one service, continue through another service, and fail somewhere in the middle of the distributed flow.</p><p>To understand what is happening inside the system, we need to answer several questions:</p><p>- Is the application running?<br>- How many requests are being processed?<br>- How much memory and CPU is the application using?<br>- Which services participated in a request?<br>- How long did each service operation take?<br>- Can we follow the same request across multiple services?<br>- Can we correlate application logs with distributed traces?</p><p>In this article, we will build a Spring Boot microservices observability example using:</p><p>- Spring Boot Actuator<br>- Micrometer<br>- OpenTelemetry<br>- Zipkin<br>- Prometheus<br>- Grafana<br>- Correlation ID<br>- Docker Compose</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-microservices-observability-demo">Spring Boot Microservices Observability Demo</a></p><p><strong>Project Architecture</strong></p><p>The project contains two Spring Boot services:</p><p>- Order Service<br>- Inventory Service</p><p>When a new order is created, Order Service sends an HTTP request to Inventory Service to reserve the requested quantity.</p><pre>Client<br> |<br> v<br>Order Service<br> |<br> v<br>Inventory Service</pre><p>Both services send distributed tracing information to Zipkin and expose application metrics for Prometheus.</p><p>Grafana uses Prometheus as its data source.</p><pre>Client<br> |<br> v<br>Order Service - - -&gt; Inventory Service<br>     |                     |<br>     + - - - Zipkin - - - -+<br>     |                     |<br>     + - - Prometheus - - -+<br>               |<br>            Grafana</pre><p><strong>What Is Observability?</strong></p><p>Observability helps us understand the internal state of a system by examining the data it produces.</p><p>The three main pillars of observability are:</p><p><strong>Metrics</strong></p><p>Metrics are numerical measurements collected over time.</p><p>Examples include:</p><p>- Request count<br>- Request duration<br>- Error rate<br>- JVM memory usage<br>- CPU usage<br>- Active threads</p><p>In this project, Spring Boot Actuator and Micrometer expose metrics, while Prometheus collects them.</p><p><strong>Traces</strong></p><p>A distributed trace represents the complete lifecycle of a request across multiple services.</p><p>Each trace contains one or more spans.</p><p>For example:</p><pre>POST /orders<br>└── POST /inventory/reserve</pre><p>The complete operation is a trace. Each individual service operation is represented by a span.</p><p>In this project, Micrometer Tracing uses the OpenTelemetry bridge to generate trace information, and Zipkin stores and visualizes it.</p><p><strong>Logs</strong></p><p>Logs describe events generated by an application.</p><p>By including trace IDs, span IDs, and correlation IDs in logs, we can connect a log entry to a specific distributed request.</p><p><strong>Project Requirements</strong></p><p>- Java 17 or later<br>- Maven 3.9+<br>- Docker Desktop<br>- Docker Compose<br>- Basic Spring Boot knowledge</p><p><strong>Multi-Module Project Structure</strong></p><p>The project is organized as a Maven multi-module application:</p><pre>spring-boot-microservices-observability-demo<br>├── order-service<br>├── inventory-service<br>├── prometheus<br>├── grafana<br>├── docker-compose.yml<br>└── pom.xml</pre><p>The root pom.xml defines both Spring Boot applications as Maven modules:</p><pre>&lt;modules&gt;<br>    &lt;module&gt;order-service&lt;/module&gt;<br>    &lt;module&gt;inventory-service&lt;/module&gt;<br>&lt;/modules&gt;</pre><p><strong>Required Dependencies</strong></p><p>Both microservices use the following dependencies:</p><pre>&lt;dependencies&gt;<br>    &lt;dependency&gt;<br>        &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>        &lt;artifactId&gt;spring-boot-starter-web&lt;/artifactId&gt;<br>    &lt;/dependency&gt;<br><br>    &lt;dependency&gt;<br>        &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>        &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;<br>    &lt;/dependency&gt;<br><br>    &lt;dependency&gt;<br>        &lt;groupId&gt;io.micrometer&lt;/groupId&gt;<br>        &lt;artifactId&gt;micrometer-registry-prometheus&lt;/artifactId&gt;<br>    &lt;/dependency&gt;<br><br>    &lt;dependency&gt;<br>        &lt;groupId&gt;io.micrometer&lt;/groupId&gt;<br>        &lt;artifactId&gt;micrometer-tracing-bridge-otel&lt;/artifactId&gt;<br>    &lt;/dependency&gt;<br><br>    &lt;dependency&gt;<br>        &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>        &lt;artifactId&gt;opentelemetry-exporter-zipkin&lt;/artifactId&gt;<br>    &lt;/dependency&gt;<br>&lt;/dependencies&gt;</pre><p>The dependencies have the following responsibilities:</p><p>- spring-boot-starter-actuator exposes health and metrics endpoints.<br>- micrometer-registry-prometheus converts Micrometer metrics into Prometheus format.<br>- micrometer-tracing-bridge-otel connects Micrometer Tracing to OpenTelemetry.<br>- opentelemetry-exporter-zipkin sends trace data to Zipkin.</p><p><strong>Creating the Inventory Service</strong></p><p>Inventory Service runs on port 8081.</p><p>It provides an endpoint for reserving inventory:</p><pre>@RestController<br>@RequestMapping(&quot;/inventory&quot;)<br>public class InventoryController {<br><br>    private static final Logger log =<br>            LoggerFactory.getLogger(InventoryController.class);<br><br>    private final Counter reservationCounter;<br><br>    public InventoryController(MeterRegistry registry) {<br>        this.reservationCounter = Counter<br>                .builder(&quot;demo.inventory.reservations&quot;)<br>                .description(&quot;Inventory reservations&quot;)<br>                .register(registry);<br>    }<br><br>    @PostMapping(&quot;/reserve&quot;)<br>    public InventoryResponse reserve(<br>            @RequestBody InventoryRequest request<br>    ) {<br>        log.info(<br>                &quot;Reserving inventory for product={} quantity={}&quot;,<br>                request.productId(),<br>                request.quantity()<br>        );<br><br>        reservationCounter.increment();<br><br>        return new InventoryResponse(<br>                request.productId(),<br>                request.quantity(),<br>                &quot;RESERVED&quot;<br>        );<br>    }<br><br>    public record InventoryRequest(<br>            String productId,<br>            int quantity<br>    ) {<br>    }<br><br>    public record InventoryResponse(<br>            String productId,<br>            int reserved,<br>            String status<br>    ) {<br>    }<br>}</pre><p>The controller defines a custom Micrometer counter:</p><pre>demo.inventory.reservations</pre><p>Prometheus exposes this counter as:</p><pre>demo_inventory_reservations_total</pre><p>The counter increases whenever an inventory reservation is created.</p><p><strong>Inventory Service Configuration</strong></p><p>The application.yml configuration is:</p><pre>server:<br>  port: 8081<br><br>spring:<br>  application:<br>    name: inventory-service<br><br>management:<br>  endpoints:<br>    web:<br>      exposure:<br>        include: health,info,prometheus,metrics<br><br>  endpoint:<br>    health:<br>      show-details: always<br><br>  tracing:<br>    sampling:<br>      probability: 1.0<br><br>  zipkin:<br>    tracing:<br>      endpoint: ${ZIPKIN_ENDPOINT:http://localhost:9411/api/v2/spans}<br><br>logging:<br>  pattern:<br>    level: &quot;%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-},%X{correlationId:-}]&quot;</pre><p>The tracing sampling probability is set to 1.0.</p><pre>management:<br>  tracing:<br>    sampling:<br>      probability: 1.0</pre><p>This means every request is traced.</p><p>A 100% sampling rate is useful for demonstration and development environments. In a production environment, a lower sampling rate may be more appropriate because tracing every request can create additional storage and processing costs.</p><p><strong>Creating the Order Service</strong></p><p>Order Service runs on port 8080.</p><p>It receives an order request and calls Inventory Service:</p><pre>@RestController<br>@RequestMapping(&quot;/orders&quot;)<br>public class OrderController {<br><br>    private static final Logger log =<br>            LoggerFactory.getLogger(OrderController.class);<br><br>    private final RestClient inventoryRestClient;<br>    private final Counter orderCounter;<br><br>    public OrderController(<br>            RestClient inventoryRestClient,<br>            MeterRegistry registry<br>    ) {<br>        this.inventoryRestClient = inventoryRestClient;<br><br>        this.orderCounter = Counter<br>                .builder(&quot;demo.orders.created&quot;)<br>                .description(&quot;Created demo orders&quot;)<br>                .register(registry);<br>    }<br><br>    @PostMapping<br>    public OrderResponse create(<br>            @RequestBody OrderRequest request<br>    ) {<br>        log.info(<br>                &quot;Creating order for product={} quantity={}&quot;,<br>                request.productId(),<br>                request.quantity()<br>        );<br><br>        InventoryResponse inventory =<br>                inventoryRestClient<br>                        .post()<br>                        .uri(&quot;/inventory/reserve&quot;)<br>                        .body(<br>                                new InventoryRequest(<br>                                        request.productId(),<br>                                        request.quantity()<br>                                )<br>                        )<br>                        .retrieve()<br>                        .body(InventoryResponse.class);<br><br>        orderCounter.increment();<br><br>        log.info(&quot;Order created and inventory reserved&quot;);<br><br>        return new OrderResponse(<br>                &quot;ORDER-&quot; + System.currentTimeMillis(),<br>                &quot;CREATED&quot;,<br>                inventory<br>        );<br>    }<br><br>    public record OrderRequest(<br>            String productId,<br>            int quantity<br>    ) {<br>    }<br><br>    public record InventoryRequest(<br>            String productId,<br>            int quantity<br>    ) {<br>    }<br><br>    public record InventoryResponse(<br>            String productId,<br>            int reserved,<br>            String status<br>    ) {<br>    }<br><br>    public record OrderResponse(<br>            String orderId,<br>            String status,<br>            InventoryResponse inventory<br>    ) {<br>    }<br>}</pre><p>The Order Service counter is defined as:</p><pre>demo.orders.created</pre><p>Prometheus publishes this counter as:</p><pre>demo_orders_total</pre><p>Prometheus reserves the _created suffix for creation-time metadata. For this reason, Micrometer normalizes the metric name and exposes it as demo_orders_total.</p><p><strong>Creating the HTTP Client</strong></p><p>We use Spring’s RestClient to communicate with Inventory Service:</p><pre>@Configuration<br>public class HttpClientConfig {<br><br>    @Bean<br>    RestClient inventoryRestClient(<br>            RestClient.Builder builder,<br>            @Value(&quot;${inventory-service.url}&quot;) String baseUrl<br>    ) {<br>        return builder<br>                .baseUrl(baseUrl)<br>                .requestInterceptor(<br>                        (request, body, execution) -&gt; {<br>                            String correlationId =<br>                                    MDC.get(&quot;correlationId&quot;);<br><br>                            if (correlationId != null) {<br>                                request.getHeaders().set(<br>                                        &quot;X-Correlation-ID&quot;,<br>                                        correlationId<br>                                );<br>                            }<br><br>                            return execution.execute(<br>                                    request,<br>                                    body<br>                            );<br>                        }<br>                )<br>                .build();<br>    }<br>}</pre><p>It is important to use the Spring-managed RestClient.Builder.</p><p>Spring Boot automatically adds observation and tracing instrumentation to this builder. When Order Service calls Inventory Service, OpenTelemetry tracing headers are automatically added to the outgoing HTTP request.</p><p>As a result, both services participate in the same distributed trace.</p><p><strong>Correlation ID Filter</strong></p><p>Trace IDs are generated by the tracing system. A correlation ID can be supplied by a client or generated by the application.</p><p>The filter checks the X-Correlation-ID request header:</p><pre>@Component<br>public class CorrelationIdFilter<br>        extends OncePerRequestFilter {<br><br>    public static final String HEADER =<br>            &quot;X-Correlation-ID&quot;;<br><br>    public static final String MDC_KEY =<br>            &quot;correlationId&quot;;<br><br>    @Override<br>    protected void doFilterInternal(<br>            HttpServletRequest request,<br>            HttpServletResponse response,<br>            FilterChain filterChain<br>    ) throws ServletException, IOException {<br><br>        String correlationId =<br>                Optional.ofNullable(<br>                                request.getHeader(HEADER)<br>                        )<br>                        .filter(value -&gt; !value.isBlank())<br>                        .orElseGet(<br>                                () -&gt; UUID<br>                                        .randomUUID()<br>                                        .toString()<br>                        );<br><br>        MDC.put(MDC_KEY, correlationId);<br>        response.setHeader(HEADER, correlationId);<br><br>        try {<br>            filterChain.doFilter(request, response);<br>        } finally {<br>            MDC.remove(MDC_KEY);<br>        }<br>    }<br>}</pre><p>If a client sends an X-Correlation-ID header, the application uses the supplied value.</p><p>If the header is missing, the filter generates a new UUID.</p><p>The same correlation ID is:</p><p>- Added to the logging MDC<br>- Returned in the HTTP response<br>- Forwarded from Order Service to Inventory Service</p><p><strong>Order Service Configuration</strong></p><pre>server:<br>  port: 8080<br><br>spring:<br>  application:<br>    name: order-service<br><br>inventory-service:<br>  url: ${INVENTORY_SERVICE_URL:http://localhost:8081}<br><br>management:<br>  endpoints:<br>    web:<br>      exposure:<br>        include: health,info,prometheus,metrics<br><br>  endpoint:<br>    health:<br>      show-details: always<br><br>  tracing:<br>    sampling:<br>      probability: 1.0<br><br>  zipkin:<br>    tracing:<br>      endpoint: ${ZIPKIN_ENDPOINT:http://localhost:9411/api/v2/spans}<br><br>logging:<br>  pattern:<br>    level: &quot;%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-},%X{correlationId:-}]&quot;</pre><p>The log pattern includes:</p><pre>application name<br>trace ID<br>span ID<br>correlation ID</pre><p>An Order Service log entry looks similar to:</p><pre>INFO [order-service,<br>0521482af89f4a205cb439345f8f9aef,<br>aaf921a6e73c72f4,<br>medium-demo-001]<br>Creating order for product=product-100 quantity=2</pre><p>The corresponding Inventory Service log contains the same trace ID and correlation ID but has a different span ID:</p><pre>INFO [inventory-service,<br>0521482af89f4a205cb439345f8f9aef,<br>b84835d470e1a958,<br>medium-demo-001]<br>Reserving inventory for product=product-100 quantity=2</pre><p>This demonstrates that both log entries belong to the same distributed request.</p><p><strong>Prometheus Configuration</strong></p><p>Prometheus collects metrics from both applications every five seconds.</p><p>Create prometheus/prometheus.yml:</p><pre>global:<br>  scrape_interval: 5s<br><br>scrape_configs:<br>  - job_name: order-service<br>    metrics_path: /actuator/prometheus<br>    static_configs:<br>      - targets:<br>          - order-service:8080<br><br>  - job_name: inventory-service<br>    metrics_path: /actuator/prometheus<br>    static_configs:<br>      - targets:<br>          - inventory-service:8081</pre><p>Because Prometheus and the Spring Boot applications run inside the same Docker Compose network, service names can be used as hostnames.</p><p><strong>Grafana Data Source</strong></p><p>Grafana is automatically configured to use Prometheus as its default data source:</p><pre>apiVersion: 1<br><br>datasources:<br>  - name: Prometheus<br>    uid: prometheus<br>    type: prometheus<br>    access: proxy<br>    url: http://prometheus:9090<br>    isDefault: true<br>    editable: true</pre><p>The project also contains a pre-provisioned Grafana dashboard.</p><p>The dashboard displays:</p><p>- HTTP request rate<br>- JVM heap usage<br>- Created order count<br>- Inventory reservation count</p><p>The Order panel uses:</p><pre>demo_orders_total</pre><p>The Inventory panel uses:</p><pre>demo_inventory_reservations_total</pre><p><strong>Dockerfiles</strong></p><p>Both services use a simple Java 17 runtime image.</p><p>Order Service Dockerfile:</p><pre>FROM eclipse-temurin:17-jre<br><br>WORKDIR /app<br><br>COPY target/order-service-0.0.1-SNAPSHOT.jar app.jar<br><br>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;app.jar&quot;]</pre><p>Inventory Service follows the same structure:</p><pre>FROM eclipse-temurin:17-jre<br><br>WORKDIR /app<br><br>COPY target/inventory-service-0.0.1-SNAPSHOT.jar app.jar<br><br>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;app.jar&quot;]</pre><p><strong>Docker Compose Configuration</strong></p><p>The complete environment contains five containers:</p><p>- Order Service<br>- Inventory Service<br>- Zipkin<br>- Prometheus<br>- Grafana</p><pre>services:<br>  zipkin:<br>    image: openzipkin/zipkin:3<br>    ports:<br>      - &quot;9411:9411&quot;<br><br>  inventory-service:<br>    build: ./inventory-service<br>    environment:<br>      ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans<br>    ports:<br>      - &quot;8081:8081&quot;<br>    depends_on:<br>      - zipkin<br><br>  order-service:<br>    build: ./order-service<br>    environment:<br>      INVENTORY_SERVICE_URL: http://inventory-service:8081<br>      ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans<br>    ports:<br>      - &quot;8080:8080&quot;<br>    depends_on:<br>      - inventory-service<br>      - zipkin<br><br>  prometheus:<br>    image: prom/prometheus:v2.54.1<br>    command:<br>      - &quot;--config.file=/etc/prometheus/prometheus.yml&quot;<br>    volumes:<br>      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro<br>    ports:<br>      - &quot;9090:9090&quot;<br>    depends_on:<br>      - order-service<br>      - inventory-service<br><br>  grafana:<br>    image: grafana/grafana:11.2.2<br>    environment:<br>      GF_SECURITY_ADMIN_USER: admin<br>      GF_SECURITY_ADMIN_PASSWORD: admin<br>    volumes:<br>      - ./grafana/provisioning:/etc/grafana/provisioning:ro<br>      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro<br>    ports:<br>      - &quot;3000:3000&quot;<br>    depends_on:<br>      - prometheus</pre><p>Docker Compose creates a common network automatically.</p><p>The containers can therefore communicate using their service names:</p><pre>order-service<br>inventory-service<br>zipkin<br>prometheus<br>grafana</pre><p><strong>Building the Project</strong></p><p>Build both Spring Boot applications from the root directory:</p><pre>mvn clean package</pre><p>Start the complete environment:</p><pre>docker compose up - build</pre><p>Check the running containers:</p><pre>docker compose ps</pre><p>The following services should be running:</p><pre>order-service<br>inventory-service<br>zipkin<br>prometheus<br>grafana</pre><p><strong>Checking Application Health</strong></p><p>Order Service:</p><pre>curl http://localhost:8080/actuator/health</pre><p>Inventory Service:</p><pre>curl http://localhost:8081/actuator/health</pre><p>Expected response:</p><pre>{<br> &quot;status&quot;: &quot;UP&quot;<br>}</pre><p><strong>Creating an Order</strong></p><p>Send a request to Order Service:</p><pre>curl -i -X POST &quot;http://localhost:8080/orders&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;X-Correlation-ID: medium-demo-001&quot; \<br>  -d &#39;{<br>    &quot;productId&quot;: &quot;product-100&quot;,<br>    &quot;quantity&quot;: 2<br>  }&#39;</pre><p>Expected response:</p><pre>{<br>  &quot;orderId&quot;: &quot;ORDER-1783878917975&quot;,<br>  &quot;status&quot;: &quot;CREATED&quot;,<br>  &quot;inventory&quot;: {<br>    &quot;productId&quot;: &quot;product-100&quot;,<br>    &quot;reserved&quot;: 2,<br>    &quot;status&quot;: &quot;RESERVED&quot;<br>  }<br>}</pre><p>The HTTP response also contains:</p><pre>X-Correlation-ID: medium-demo-001</pre><p><strong>Checking Distributed Logs</strong></p><p>Check Order Service logs:</p><pre>docker compose logs order-service</pre><p>Check Inventory Service logs:</p><pre>docker compose logs inventory-service</pre><p>Follow both services:</p><pre>docker compose logs -f \<br>  order-service \<br>  inventory-service</pre><p>The log entries should contain:</p><p>- The same trace ID<br>- The same correlation ID<br>- Different span IDs</p><p><strong>Viewing the Distributed Trace in Zipkin</strong></p><p>Open Zipkin: http://localhost:9411</p><p>Use the following search options:</p><pre>Service Name: order-service<br>Span Name: http post /orders<br>Lookback: Last Hour<br>Limit: 100</pre><p>Click Run Query.</p><p>Open the returned trace.</p><p>The trace contains three spans:</p><pre>order-service:<br>  http post /orders<br><br>order-service:<br>  http post<br><br>inventory-service:<br>  http post /inventory/reserve</pre><p>The first span represents the incoming request to Order Service. The second span represents the outgoing HTTP client operation. The third span represents the incoming request handled by Inventory Service.</p><p>Prometheus also calls /actuator/prometheus every five seconds. These calls generate additional traces. Selecting http post /orders as the span name prevents Prometheus scrape traces from hiding the application trace in the search results.</p><p><strong>Checking Prometheus Targets</strong></p><p>Open: http://localhost:9090/targets</p><p>Both targets should be UP:</p><pre>order-service<br>inventory-service</pre><p>If a target is down, check its container status and Actuator endpoint.</p><p><strong>Querying Custom Metrics</strong></p><p>Open Prometheus: http://localhost:9090</p><p>Query the number of created orders:</p><pre>demo_orders_total</pre><p>Query the number of inventory reservations:</p><pre>demo_inventory_reservations_total</pre><p>Display both metrics:</p><pre>demo_orders_total<br>or<br>demo_inventory_reservations_total</pre><p>The counters increase after every successful order operation.</p><p><strong>Querying HTTP Metrics</strong></p><p>Display all recorded HTTP requests:</p><pre>http_server_requests_seconds_count</pre><p>Display only Order Service requests:</p><pre>http_server_requests_seconds_count{<br>  job=&quot;order-service&quot;<br>}</pre><p>Calculate the HTTP request rate:</p><pre>sum by (job, uri) (<br>  rate(<br>    http_server_requests_seconds_count[1m]<br>  )<br>)</pre><p><strong>Opening the Grafana Dashboard</strong></p><p>Open Grafana: http://localhost:3000</p><p>Login credentials:</p><pre>Username: admin<br>Password: admin</pre><p>Navigate to:</p><pre>Dashboards<br>→ Spring Boot Demo<br>→ Spring Boot Microservices Observability</pre><p>The dashboard is automatically provisioned when the Grafana container starts.</p><p>It contains the following panels:</p><p>- HTTP Request Rate<br>- JVM Heap Used<br>- Orders Created<br>- Inventory Reservations</p><p><strong>Generating More Traffic</strong></p><p>Send multiple order requests:</p><pre>for i in {1..10}<br>do<br>  curl -X POST &quot;http://localhost:8080/orders&quot; \<br>    -H &quot;Content-Type: application/json&quot; \<br>    -H &quot;X-Correlation-ID: order-test-$i&quot; \<br>    -d &#39;{<br>      &quot;productId&quot;: &quot;product-100&quot;,<br>      &quot;quantity&quot;: 2<br>    }&#39;<br><br>  echo<br>done</pre><p>After generating traffic:</p><p>- Search for the new traces in Zipkin.<br>- Check the counters in Prometheus.<br>- Watch the panels change in Grafana.<br>- Compare trace IDs and correlation IDs in the service logs.</p><p><strong>Stopping the Environment</strong></p><p>Stop all containers:</p><pre>docker compose down</pre><p>Remove the containers and associated volumes:</p><pre>docker compose down -v</pre><p>Start the environment again:</p><pre>docker compose up - build</pre><p><strong>Metrics, Traces, and Correlation IDs</strong></p><p>These mechanisms have different responsibilities.</p><p>Metrics Answer Aggregate Questions</p><pre>How many requests were processed?<br>What is the current error rate?<br>How much memory is being used?</pre><p>Traces Explain Individual Requests</p><pre>Which services participated?<br>Which operation was slow?<br>Where did the request fail?</pre><p><strong>Correlation IDs Support Business-Level Tracking</strong></p><pre>Which logs belong to a specific client request?<br>Can the client provide its own tracking identifier?<br>Can support teams search logs using a known request ID?</pre><p>Using all three mechanisms provides a more complete view of the application.</p><p><strong>Production Considerations</strong></p><p>This project is designed as an observability demonstration. Before using a similar setup in production, consider the following points:</p><p>- Reduce the tracing sampling probability if traffic is high.<br>- Do not use default Grafana credentials.<br>- Protect Actuator endpoints.<br>- Add persistent volumes for Prometheus, Grafana, and Zipkin.<br>- Configure data retention policies.<br>- Avoid exposing internal monitoring services publicly.<br>- Add alert rules to Prometheus.<br>- Configure Grafana alerts.<br>- Use structured JSON logging.<br>- Consider a centralized logging platform.<br>- Add resource limits to Docker or Kubernetes deployments.<br>- Use TLS and authentication between observability components.</p><p><strong>Conclusion</strong></p><p>In this article, we created a complete Spring Boot microservices observability example.</p><p>We used:</p><p>- Spring Boot Actuator to expose health and application information<br>- Micrometer to generate application metrics<br>- Prometheus to collect and query metrics<br>- Grafana to visualize application behavior<br>- OpenTelemetry to instrument distributed operations<br>- Zipkin to store and display distributed traces<br>- Correlation IDs to follow requests through application logs<br>- Docker Compose to run the complete environment</p><p>When a request enters Order Service and continues to Inventory Service, both operations appear under the same trace ID.</p><p>At the same time, Prometheus collects application and JVM metrics, while Grafana displays them using a preconfigured dashboard.</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/spring-boot-microservices-observability-demo">Spring Boot Microservices Observability Demo</a></p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3d41d673910" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combining Resilience4j Patterns in Spring Boot]]></title>
            <link>https://erkndmrl.medium.com/combining-resilience4j-patterns-in-spring-boot-a3a392b199f2?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3a392b199f2</guid>
            <category><![CDATA[circuit-breaker]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[resilience4j]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 10:38:36 GMT</pubDate>
            <atom:updated>2026-07-17T10:38:36.239Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HHUXTknkHVBHXjnJd_OHsQ.png" /></figure><p>In a previous article, we explored how to use the Circuit Breaker pattern with Resilience4j in a Spring Boot application.</p><p>Circuit Breaker is an important fault-tolerance pattern, but it does not solve every problem that may occur while communicating with an external service.</p><p>A downstream service may:</p><p>- Receive too many requests<br>- Process too many concurrent operations<br>- Respond too slowly<br>- Fail temporarily<br>- Become completely unavailable</p><p>In this article, we will combine the following Resilience4j patterns in a single Spring Boot application:</p><pre>Rate Limiter<br>→ Bulkhead<br>→ TimeLimiter<br>→ Circuit Breaker<br>→ Retry<br>→ Payment Provider<br>→ Fallback</pre><p>The Payment Provider in this project is only a simulated downstream service. The main purpose of the project is to demonstrate how different fault-tolerance patterns can work together.</p><h4>What Does Each Pattern Do?</h4><p>Before starting the implementation, let’s briefly look at the responsibility of each pattern.</p><p><strong>Rate Limiter</strong></p><p>Rate Limiter controls how many requests can be processed during a specific period.</p><p>It protects the downstream service from receiving more traffic than it can handle.</p><p><strong>Bulkhead</strong></p><p>Bulkhead limits the number of concurrent calls.</p><p>It prevents one slow or overloaded downstream service from consuming all application resources.</p><p><strong>TimeLimiter</strong></p><p>TimeLimiter defines how long the application should wait for an asynchronous operation.</p><p>If the operation does not finish within the configured duration, a TimeoutException is generated.</p><p><strong>Circuit Breaker</strong></p><p>Circuit Breaker monitors successful and failed calls. When the failure rate exceeds the configured threshold, it opens the circuit and rejects new calls without executing the downstream operation.</p><p><strong>Retry</strong></p><p>Retry repeats a failed operation for a configured number of attempts. It is useful for temporary network or service errors that may disappear after a short period.</p><p><strong>Fallback</strong></p><p>Fallback returns an alternative response when the protected operation cannot be completed.</p><p><strong>Prerequisites</strong></p><p>- Java 17 or later<br>- Maven 3.9+<br>- Spring Boot 3<br>- Basic knowledge of Spring Boot<br>- Basic knowledge of Resilience4j</p><p><strong>Project Dependencies</strong></p><p>We will use Spring Web, Spring Boot Actuator, Spring AOP and the Resilience4j Spring Boot starter.</p><p>Add the following dependencies to pom.xml:</p><pre>&lt;dependencies&gt;<br> &lt;dependency&gt;<br> &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br> &lt;artifactId&gt;spring-boot-starter-web&lt;/artifactId&gt;<br> &lt;/dependency&gt;<br>&lt;dependency&gt;<br> &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br> &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;<br> &lt;/dependency&gt;<br>&lt;dependency&gt;<br> &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br> &lt;artifactId&gt;spring-boot-starter-aop&lt;/artifactId&gt;<br> &lt;/dependency&gt;<br>&lt;dependency&gt;<br> &lt;groupId&gt;io.github.resilience4j&lt;/groupId&gt;<br> &lt;artifactId&gt;resilience4j-spring-boot3&lt;/artifactId&gt;<br> &lt;version&gt;2.2.0&lt;/version&gt;<br> &lt;/dependency&gt;<br>&lt;/dependencies&gt;</pre><p>Spring AOP is required because Resilience4j annotations are applied through aspects.</p><p><strong>Payment Response</strong></p><p>First, create a response model:</p><pre>public record PaymentResponse(<br> String orderId,<br> String status,<br> String message,<br> String source<br>) {<br>}</pre><p>This response will be returned for both successful provider calls and fallback operations.</p><p><strong>Simulating the Payment Provider</strong></p><p>The PaymentProviderClient simulates an external service.</p><p>It supports four different scenarios:</p><p>- success<br>- failure<br>- slow<br>- random</p><pre>@Component<br>public class PaymentProviderClient {<br><br>    public PaymentResponse charge(String orderId, String scenario) {<br>        return switch (scenario.toLowerCase()) {<br>            case &quot;success&quot; -&gt; success(orderId);<br>            case &quot;failure&quot; -&gt;<br>                    throw new IllegalStateException(<br>                            &quot;Payment provider returned an error&quot;<br>                    );<br>            case &quot;slow&quot; -&gt; slowResponse(orderId);<br>            case &quot;random&quot; -&gt; randomResponse(orderId);<br>            default -&gt; throw new IllegalArgumentException(<br>                    &quot;Unknown scenario. Supported values: &quot; +<br>                    &quot;success, failure, slow, random&quot;<br>            );<br>        };<br>    }<br><br>    private PaymentResponse success(String orderId) {<br>        return new PaymentResponse(<br>                orderId,<br>                &quot;PAID&quot;,<br>                &quot;Payment completed&quot;,<br>                &quot;payment-provider&quot;<br>        );<br>    }<br><br>    private PaymentResponse slowResponse(String orderId) {<br>        sleep(Duration.ofSeconds(3));<br>        return success(orderId);<br>    }<br><br>    private PaymentResponse randomResponse(String orderId) {<br>        if (ThreadLocalRandom.current().nextBoolean()) {<br>            throw new IllegalStateException(<br>                    &quot;Random payment provider failure&quot;<br>            );<br>        }<br><br>        return success(orderId);<br>    }<br><br>    private void sleep(Duration duration) {<br>        try {<br>            Thread.sleep(duration.toMillis());<br>        } catch (InterruptedException exception) {<br>            Thread.currentThread().interrupt();<br><br>            throw new IllegalStateException(<br>                    &quot;Payment provider call was interrupted&quot;,<br>                    exception<br>            );<br>        }<br>    }<br>}</pre><p>The slow scenario waits for three seconds. Later, we will configure TimeLimiter with a two-second timeout so that this call produces a fallback response.</p><p><strong>Combining Resilience4j Annotations</strong></p><p>The service method returns a CompletableFuture because TimeLimiter works with asynchronous return types.</p><pre>@Service<br>public class PaymentService {<br><br>    private static final String BACKEND = &quot;paymentProvider&quot;;<br><br>    private final PaymentProviderClient paymentProviderClient;<br><br>    public PaymentService(<br>            PaymentProviderClient paymentProviderClient<br>    ) {<br>        this.paymentProviderClient = paymentProviderClient;<br>    }<br><br>    @RateLimiter(<br>            name = BACKEND,<br>            fallbackMethod = &quot;fallbackPayment&quot;<br>    )<br>    @Bulkhead(<br>            name = BACKEND,<br>            type = Bulkhead.Type.SEMAPHORE,<br>            fallbackMethod = &quot;fallbackPayment&quot;<br>    )<br>    @TimeLimiter(<br>            name = BACKEND,<br>            fallbackMethod = &quot;fallbackPayment&quot;<br>    )<br>    @CircuitBreaker(<br>            name = BACKEND,<br>            fallbackMethod = &quot;fallbackPayment&quot;<br>    )<br>    @Retry(<br>            name = BACKEND,<br>            fallbackMethod = &quot;fallbackPayment&quot;<br>    )<br>    public CompletableFuture&lt;PaymentResponse&gt; processPayment(<br>            String orderId,<br>            String scenario<br>    ) {<br>        return CompletableFuture.supplyAsync(<br>                () -&gt; paymentProviderClient.charge(orderId, scenario)<br>        );<br>    }<br><br>    private CompletableFuture&lt;PaymentResponse&gt; fallbackPayment(<br>            String orderId,<br>            String scenario,<br>            Throwable throwable<br>    ) {<br>        PaymentResponse response = new PaymentResponse(<br>                orderId,<br>                &quot;PENDING_PAYMENT&quot;,<br>                &quot;Fallback response: &quot;<br>                        + throwable.getClass().getSimpleName(),<br>                &quot;fallback&quot;<br>        );<br><br>        return CompletableFuture.completedFuture(response);<br>    }<br>}</pre><p>All annotations use the same instance name:</p><pre>paymentProvider</pre><p>This name connects the annotations to their corresponding configuration in application.yml.</p><p>The fallback method must:</p><p>- Accept the original method parameters<br>- Accept a Throwable as the final parameter<br>- Return a compatible return type</p><p>Because processPayment returns CompletableFuture&lt;PaymentResponse&gt;, the fallback method must return the same type.</p><p><strong>Resilience4j Configuration</strong></p><p>The complete configuration is shown below:</p><pre>server:<br>  port: 8080<br><br>management:<br>  endpoints:<br>    web:<br>      exposure:<br>        include: health,metrics,circuitbreakers,circuitbreakerevents,retries,retryevents,ratelimiters,ratelimiterevents,bulkheads,bulkheadevents<br>  endpoint:<br>    health:<br>      show-details: always<br>  health:<br>    circuitbreakers:<br>      enabled: true<br>    ratelimiters:<br>      enabled: true<br><br>resilience4j:<br>  ratelimiter:<br>    instances:<br>      paymentProvider:<br>        limitForPeriod: 5<br>        limitRefreshPeriod: 10s<br>        timeoutDuration: 0<br>        registerHealthIndicator: true<br><br>  bulkhead:<br>    instances:<br>      paymentProvider:<br>        maxConcurrentCalls: 3<br>        maxWaitDuration: 0<br><br>  timelimiter:<br>    instances:<br>      paymentProvider:<br>        timeoutDuration: 2s<br>        cancelRunningFuture: true<br><br>  circuitbreaker:<br>    instances:<br>      paymentProvider:<br>        registerHealthIndicator: true<br>        slidingWindowType: COUNT_BASED<br>        slidingWindowSize: 5<br>        minimumNumberOfCalls: 5<br>        failureRateThreshold: 50<br>        slowCallDurationThreshold: 1s<br>        slowCallRateThreshold: 50<br>        waitDurationInOpenState: 10s<br>        permittedNumberOfCallsInHalfOpenState: 2<br>        automaticTransitionFromOpenToHalfOpenEnabled: true<br><br>  retry:<br>    instances:<br>      paymentProvider:<br>        maxAttempts: 3<br>        waitDuration: 500ms<br>        enableExponentialBackoff: true<br>        exponentialBackoffMultiplier: 2</pre><p>Let’s examine each configuration separately.</p><p><strong>Rate Limiter Configuration</strong></p><pre>limitForPeriod: 5<br>limitRefreshPeriod: 10s<br>timeoutDuration: 0</pre><p>The application permits five calls every ten seconds.</p><p>When the limit is exceeded, the request is rejected immediately because timeoutDuration is set to zero.</p><p><strong>Bulkhead Configuration</strong></p><pre>maxConcurrentCalls: 3<br>maxWaitDuration: 0</pre><p>Only three payment operations can run concurrently.</p><p>Additional calls are rejected immediately instead of waiting for an available slot.</p><p>This protects the application from resource exhaustion when the downstream service becomes slow.</p><p><strong>TimeLimiter Configuration</strong></p><pre>timeoutDuration: 2s<br>cancelRunningFuture: true</pre><p>The application waits for the asynchronous operation for a maximum of two seconds.</p><p>The simulated slow provider waits for three seconds, so TimeLimiter generates a TimeoutException.</p><p><strong>Circuit Breaker Configuration</strong></p><pre>slidingWindowType: COUNT_BASED<br>slidingWindowSize: 5<br>minimumNumberOfCalls: 5<br>failureRateThreshold: 50</pre><p>Circuit Breaker evaluates the latest five calls.</p><p>After at least five calls are recorded, the circuit opens if 50 percent or more of those calls fail.</p><pre>waitDurationInOpenState: 10s<br>permittedNumberOfCallsInHalfOpenState: 2<br>automaticTransitionFromOpenToHalfOpenEnabled: true</pre><p>After remaining open for ten seconds, Circuit Breaker moves to the HALF_OPEN state.</p><p>It permits two test calls to determine whether the downstream service has recovered.</p><p><strong>Retry Configuration</strong></p><pre>maxAttempts: 3<br>waitDuration: 500ms<br>enableExponentialBackoff: true<br>exponentialBackoffMultiplier: 2</pre><p>A failed operation can be attempted up to three times.</p><p>The waiting duration increases using exponential backoff:</p><pre>First retry → 500 ms<br>Second retry → 1000 ms</pre><p>Retry should be used carefully for operations that may create side effects. In real payment or order systems, retry operations should normally be combined with an idempotency mechanism.</p><p><strong>Creating the Controller</strong></p><p>The controller accepts the order ID as a path variable and the simulation scenario as a request parameter.</p><pre>@RestController<br>@RequestMapping(&quot;/orders&quot;)<br>public class PaymentController {<br><br>    private final PaymentService paymentService;<br><br>    public PaymentController(PaymentService paymentService) {<br>        this.paymentService = paymentService;<br>    }<br><br>    @PostMapping(&quot;/{orderId}/payment&quot;)<br>    public CompletableFuture&lt;PaymentResponse&gt; processPayment(<br>            @PathVariable String orderId,<br>            @RequestParam(defaultValue = &quot;success&quot;) String scenario<br>    ) {<br>        return paymentService.processPayment(<br>                orderId,<br>                scenario<br>        );<br>    }<br>}</pre><p><strong>Running the Application</strong></p><p>Start the application:</p><pre>mvn spring-boot:run</pre><p>The application will run on port 8080.</p><p><strong>Successful Call</strong></p><p>Send a successful request:</p><pre>curl -X POST \<br>&quot;http://localhost:8080/orders/order-100/payment?scenario=success&quot;</pre><p>Response:</p><pre>{<br>  &quot;orderId&quot;: &quot;order-100&quot;,<br>  &quot;status&quot;: &quot;PAID&quot;,<br>  &quot;message&quot;: &quot;Payment completed&quot;,<br>  &quot;source&quot;: &quot;payment-provider&quot;<br>}</pre><p><strong>Provider Failure</strong></p><p>Send a request that generates a provider error:</p><pre>curl -X POST \<br>&quot;http://localhost:8080/orders/order-101/payment?scenario=failure&quot;</pre><p>After the configured retry attempts are completed, the fallback response is returned:</p><pre>{<br>  &quot;orderId&quot;: &quot;order-101&quot;,<br>  &quot;status&quot;: &quot;PENDING_PAYMENT&quot;,<br>  &quot;message&quot;: &quot;Fallback response: IllegalStateException&quot;,<br>  &quot;source&quot;: &quot;fallback&quot;<br>}</pre><p><strong>Slow Provider and TimeLimiter</strong></p><p>Send a slow request:</p><pre>curl -X POST \<br>&quot;http://localhost:8080/orders/order-102/payment?scenario=slow&quot;</pre><p>The provider waits for three seconds, but TimeLimiter allows only two seconds.</p><p>Response:</p><pre>{<br> &quot;orderId&quot;: &quot;order-102&quot;,<br> &quot;status&quot;: &quot;PENDING_PAYMENT&quot;,<br> &quot;message&quot;: &quot;Fallback response: TimeoutException&quot;,<br> &quot;source&quot;: &quot;fallback&quot;<br>}</pre><p><strong>Random Failure Scenario</strong></p><p>The random scenario succeeds or fails with an approximately equal probability:</p><pre>curl -X POST \<br>&quot;http://localhost:8080/orders/order-103/payment?scenario=random&quot;</pre><p>Calling this endpoint several times helps demonstrate Retry and Circuit Breaker transitions.</p><p><strong>Testing the Rate Limiter</strong></p><p>Run the request more than five times within ten seconds:</p><pre>for i in {1..7}<br>do<br> curl -X POST \<br> &quot;http://localhost:8080/orders/order-$i/payment?scenario=success&quot;<br> echo<br>done</pre><p>After the configured limit is reached, Rate Limiter rejects additional requests and fallback handles the result.</p><p><strong>Testing the Bulkhead</strong></p><p>The slow scenario can be used to keep concurrent operations busy.</p><p>Send more than three slow requests simultaneously:</p><pre>for i in {1..5}<br>do<br>  curl -X POST \<br>  &quot;http://localhost:8080/orders/order-$i/payment?scenario=slow&quot; &amp;<br>done<br><br>wait</pre><p>Because maxConcurrentCalls is three, additional operations are rejected by Bulkhead.</p><p><strong>Monitoring with Spring Boot Actuator</strong></p><p>The project exposes several Actuator endpoints.</p><p>Check the overall health:</p><pre>curl &quot;http://localhost:8080/actuator/health&quot;</pre><p>Check the Circuit Breaker instances:</p><pre>curl &quot;http://localhost:8080/actuator/circuitbreakers&quot;</pre><p>Check Circuit Breaker events:</p><pre>curl &quot;http://localhost:8080/actuator/circuitbreakerevents&quot;</pre><p>Check Retry events:</p><pre>curl &quot;http://localhost:8080/actuator/retryevents&quot;</pre><p>Check Rate Limiter events:</p><pre>curl &quot;http://localhost:8080/actuator/ratelimiterevents&quot;</pre><p>Check Bulkhead events:</p><pre>curl &quot;http://localhost:8080/actuator/bulkheadevents&quot;</pre><p>These endpoints make it possible to observe successful calls, failed calls, rejected requests, retry attempts and Circuit Breaker state transitions.</p><p><strong>Why Do We Need Multiple Patterns?</strong></p><p>Each pattern protects the application from a different type of failure.</p><pre>Rate Limiter → Protects against excessive request rates<br>Bulkhead → Protects application resources<br>TimeLimiter → Protects against slow operations<br>Retry → Handles temporary failures<br>Circuit Breaker → Prevents repeated calls to an unhealthy service<br>Fallback → Provides an alternative response</pre><p>Circuit Breaker alone cannot restrict traffic, limit concurrent operations or stop an individual slow call.</p><p>Similarly, Retry alone may make an overloaded service even less stable by sending additional requests.</p><p>The patterns should therefore be selected and configured according to the behavior of the downstream service.</p><p><strong>Important Considerations</strong></p><p>Using every resilience pattern does not automatically make an application resilient.</p><p>Some important points should be considered:</p><p>- Do not retry validation or business errors.<br>- Use timeout values based on the expected service latency.<br>- Avoid aggressive retry configurations.<br>- Use idempotency for operations with side effects.<br>- Configure separate Resilience4j instances for separate downstream services.<br>- Monitor fallback and rejected call rates.<br>- Test the configuration under concurrent load.<br>- Keep fallback operations simple and reliable.</p><p><strong>Conclusion</strong></p><p>In this article, we combined Rate Limiter, Bulkhead, TimeLimiter, Circuit Breaker, Retry and Fallback in a Spring Boot application.</p><p>Together, these patterns provide protection against:</p><p>- Excessive traffic<br>- Too many concurrent operations<br>- Slow downstream services<br>- Temporary failures<br>- Repeated calls to unavailable services</p><p>Resilience4j provides these patterns as independent modules, allowing us to select and combine only the mechanisms required by our application.</p><p>The complete source code is available on GitHub:</p><p><a href="https://github.com/erkandemirel/resilience4j-fault-tolerance-patterns-demo">resilience4j-fault-tolerance-patterns-demo</a></p><p>You can also read the previous Circuit Breaker article:</p><p><a href="https://medium.com/@erkndmrl/using-resilience4j-circuit-breaker-in-spring-boot-microservices-c2a45198ca1c">Using Resilience4j Circuit Breaker in Spring Boot Microservices</a></p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3a392b199f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Resilience4j Circuit Breaker in Spring Boot Microservices]]></title>
            <link>https://erkndmrl.medium.com/using-resilience4j-circuit-breaker-in-spring-boot-microservices-c2a45198ca1c?source=rss-6349117db536------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2a45198ca1c</guid>
            <category><![CDATA[circuit-breaker]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[resilience4j]]></category>
            <dc:creator><![CDATA[Erkan Demirel]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 12:48:24 GMT</pubDate>
            <atom:updated>2026-06-30T12:48:24.683Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lx3m8D3oJ6-RemfhRvhdqw.png" /></figure><p>In a microservice architecture, one service usually depends on another service. An order service may call a payment provider. A notification service may call an SMS provider. A reporting service may call many internal APIs.</p><p>This is normal, but it also creates a risk. If one downstream service becomes slow or unavailable, the caller can also become slow. Then requests start waiting, threads are consumed, and a small failure can turn into a bigger system problem.</p><p>This is where the Circuit Breaker pattern helps.</p><p>In this article, we will build a simple Spring Boot example with Resilience4j. The example is intentionally small and easy to understand. We will create an order payment API that calls a fake payment provider. When the payment provider fails, the circuit breaker will return a fallback response instead of letting the error break the request.</p><h4>What is a Circuit Breaker?</h4><p>A circuit breaker protects your application from repeatedly calling a failing dependency.</p><p>The idea is similar to an electrical circuit breaker:</p><ul><li>When calls are successful, the circuit is closed.</li><li>When too many calls fail, the circuit opens.</li><li>While the circuit is open, calls do not go to the failing service.</li><li>After a waiting time, the circuit becomes half-open and tests a few calls.</li><li>If those calls are successful, the circuit closes again.</li></ul><p>This gives the failing service time to recover and keeps your own service responsive.</p><h4>Prerequisites</h4><p>For this example, we will use:</p><ul><li>Java 21</li><li>Spring Boot 3</li><li>Maven</li><li>Resilience4j</li><li>Spring Boot Actuator</li></ul><h4>Project Dependencies</h4><p>First, create a Spring Boot project and add the following dependencies.</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>    &lt;artifactId&gt;spring-boot-starter-web&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>    &lt;artifactId&gt;spring-boot-starter-aop&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>    &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.github.resilience4j&lt;/groupId&gt;<br>    &lt;artifactId&gt;resilience4j-spring-boot3&lt;/artifactId&gt;<br>    &lt;version&gt;2.2.0&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>The spring-boot-starter-aop dependency is important because Resilience4j uses Spring AOP to apply annotations such as @CircuitBreaker.</p><h4>Application Configuration</h4><p>Now we can configure the circuit breaker in application.yml.</p><pre>server:<br>  port: 8080<br><br>management:<br>  endpoints:<br>    web:<br>      exposure:<br>        include: health,metrics,circuitbreakers,circuitbreakerevents<br>  endpoint:<br>    health:<br>      show-details: always<br>  health:<br>    circuitbreakers:<br>      enabled: true<br>resilience4j:<br>  circuitbreaker:<br>    instances:<br>      paymentProvider:<br>        registerHealthIndicator: true<br>        slidingWindowType: COUNT_BASED<br>        slidingWindowSize: 5<br>        minimumNumberOfCalls: 5<br>        failureRateThreshold: 50<br>        waitDurationInOpenState: 10s<br>        permittedNumberOfCallsInHalfOpenState: 2<br>        automaticTransitionFromOpenToHalfOpenEnabled: true</pre><p>Let us quickly explain the important parts:</p><ul><li>slidingWindowSize: 5 means Resilience4j will evaluate the last 5 calls.</li><li>minimumNumberOfCalls: 5 means it needs at least 5 calls before calculating the failure rate.</li><li>failureRateThreshold: 50 means the circuit opens if 50 percent or more calls fail.</li><li>waitDurationInOpenState: 10s means it waits 10 seconds before trying again.</li><li>permittedNumberOfCallsInHalfOpenState: 2 means it allows 2 test calls in half-open state.</li></ul><p>The actuator configuration helps us observe circuit breaker health and events while developing.</p><h4>Creating the Response Models</h4><p>Our API will return a payment response.</p><pre>package com.example.resilience4jdemo.model;<br><br>public record PaymentProviderResponse(<br>        String orderId,<br>        boolean successful,<br>        String providerReference,<br>        String source<br>) {<br>}</pre><pre>package com.example.resilience4jdemo.model;<br><br>public record PaymentResponse(<br>        String orderId,<br>        String status,<br>        String message,<br>        String source<br>) {<br>}</pre><p>The source field is useful for this demo because it shows whether the result came from the payment provider or from the fallback method.</p><h4>Simulating a Downstream Service</h4><p>In a real project, this class could be a Feign client, WebClient call, RestClient call, or another integration layer.</p><p>For this article, we will keep it simple and simulate failure with a request parameter.</p><pre>package com.example.resilience4jdemo.client;<br><br>import com.example.resilience4jdemo.model.PaymentProviderResponse;<br>import org.springframework.stereotype.Component;<br>import java.util.UUID;<br><br>@Component<br>public class PaymentProviderClient {<br>    public PaymentProviderResponse charge(String orderId, boolean fail) {<br>        if (fail) {<br>            throw new IllegalStateException(&quot;Payment provider is temporarily unavailable&quot;);<br>        }<br>        return new PaymentProviderResponse(<br>                orderId,<br>                true,<br>                UUID.randomUUID().toString(),<br>                &quot;payment-provider&quot;<br>        );<br>    }<br>}</pre><p>If fail is true, the client throws an exception. This allows us to test the circuit breaker easily.</p><h4>Adding the Circuit Breaker</h4><p>Now we can create our main service.</p><pre>package com.example.resilience4jdemo.service;<br><br>import com.example.resilience4jdemo.client.PaymentProviderClient;<br>import com.example.resilience4jdemo.model.PaymentProviderResponse;<br>import com.example.resilience4jdemo.model.PaymentResponse;<br>import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;<br>import org.springframework.stereotype.Service;<br><br>@Service<br>public class PaymentService {<br>    private final PaymentProviderClient paymentProviderClient;<br>    public PaymentService(PaymentProviderClient paymentProviderClient) {<br>        this.paymentProviderClient = paymentProviderClient;<br>    }<br>    @CircuitBreaker(name = &quot;paymentProvider&quot;, fallbackMethod = &quot;fallbackPayment&quot;)<br>    public PaymentResponse processPayment(String orderId, boolean failPaymentProvider) {<br>        PaymentProviderResponse providerResponse =<br>                paymentProviderClient.charge(orderId, failPaymentProvider);<br>        return new PaymentResponse(<br>                orderId,<br>                &quot;PAID&quot;,<br>                &quot;Payment completed with reference: &quot; + providerResponse.providerReference(),<br>                providerResponse.source()<br>        );<br>    }<br>    private PaymentResponse fallbackPayment(<br>            String orderId,<br>            boolean failPaymentProvider,<br>            Throwable throwable<br>    ) {<br>        return new PaymentResponse(<br>                orderId,<br>                &quot;PENDING_PAYMENT&quot;,<br>                &quot;Payment provider is unavailable. The order will be retried later.&quot;,<br>                &quot;fallback: &quot; + throwable.getClass().getSimpleName()<br>        );<br>    }<br>}</pre><p>The most important line is:</p><pre>@CircuitBreaker(name = &quot;paymentProvider&quot;, fallbackMethod = &quot;fallbackPayment&quot;)</pre><p>The name must match the instance name in application.yml.</p><p>The fallback method must have the same parameters as the original method, plus a Throwable parameter at the end.</p><p>In this example, when the payment provider fails, we do not mark the order as paid. We return PENDING_PAYMENT. This is safer than making a final business decision when the payment provider is unavailable.</p><h4>Creating the REST Controller</h4><p>Now we expose a simple endpoint.</p><pre>package com.example.resilience4jdemo.controller;<br><br>import com.example.resilience4jdemo.model.PaymentResponse;<br>import com.example.resilience4jdemo.service.PaymentService;<br>import org.springframework.web.bind.annotation.PathVariable;<br>import org.springframework.web.bind.annotation.PostMapping;<br>import org.springframework.web.bind.annotation.RequestMapping;<br>import org.springframework.web.bind.annotation.RequestParam;<br>import org.springframework.web.bind.annotation.RestController;<br><br>@RestController<br>@RequestMapping(&quot;/orders&quot;)<br>public class PaymentController {<br>    private final PaymentService paymentService;<br>    public PaymentController(PaymentService paymentService) {<br>        this.paymentService = paymentService;<br>    }<br>    @PostMapping(&quot;/{orderId}/payment&quot;)<br>    public PaymentResponse processPayment(<br>            @PathVariable String orderId,<br>            @RequestParam(defaultValue = &quot;false&quot;) boolean failPaymentProvider<br>    ) {<br>        return paymentService.processPayment(orderId, failPaymentProvider);<br>    }<br>}</pre><h4>Running the Application</h4><p>Start the application:</p><pre>mvn spring-boot:run</pre><p>Call the endpoint with a successful downstream service:</p><pre>curl -X POST &quot;http://localhost:8080/orders/order-100/payment&quot;</pre><p>Example response:</p><pre>{<br>  &quot;orderId&quot;: &quot;order-100&quot;,<br>  &quot;status&quot;: &quot;PAID&quot;,<br>  &quot;message&quot;: &quot;Payment completed with reference: 7d53c6a4-0f90-4c23-b87e-3c4f2e01c31f&quot;,<br>  &quot;source&quot;: &quot;payment-provider&quot;<br>}</pre><p>Now simulate a failing downstream service:</p><pre>curl -X POST &quot;http://localhost:8080/orders/order-100/payment?failPaymentProvider=true&quot;</pre><p>Example fallback response:</p><pre>{<br>  &quot;orderId&quot;: &quot;order-100&quot;,<br>  &quot;status&quot;: &quot;PENDING_PAYMENT&quot;,<br>  &quot;message&quot;: &quot;Payment provider is unavailable. The order will be retried later.&quot;,<br>  &quot;source&quot;: &quot;fallback: IllegalStateException&quot;<br>}</pre><p>If you repeat the failed request several times, the circuit breaker will open.</p><p>While it is open, Resilience4j will stop calling the failing method and go to the fallback directly.</p><p>You can check events with:</p><pre>curl &quot;http://localhost:8080/actuator/circuitbreakerevents&quot;</pre><p>You can also check health:</p><pre>curl &quot;http://localhost:8080/actuator/health&quot;</pre><h4>When Should You Use Circuit Breaker?</h4><p>Circuit breaker is useful when your service calls something that can fail or become slow, for example:</p><ul><li>Another internal microservice</li><li>A third-party API</li><li>A payment provider</li><li>An identity verification service</li><li>A notification provider</li><li>A document processing service</li></ul><p>But do not use it blindly everywhere. First define what the fallback should mean for your business.</p><p>For example:</p><ul><li>A product recommendation service can return an empty list.</li><li>A reporting service can return cached data.</li><li>A payment service should not mark an order as paid if the provider is unavailable.</li></ul><p>The fallback is not just technical code. It is also a business decision.</p><h4>Conclusion</h4><p>Resilience4j Circuit Breaker is a practical way to make Spring Boot microservices more resilient.</p><p>In this example, we created a small payment processing service. When the downstream payment provider failed, our application stayed responsive and returned a controlled fallback response.</p><p>The key points are:</p><ul><li>Add resilience4j-spring-boot3 and spring-boot-starter-aop.</li><li>Configure a circuit breaker instance in application.yml.</li><li>Use @CircuitBreaker on the method that calls the risky dependency.</li><li>Create a fallback method with the correct signature.</li><li>Monitor circuit breaker behavior with Spring Boot Actuator.</li></ul><p>This pattern is simple, but it can prevent many production problems when used with the right fallback strategy.</p><p>The GitHub repo for the above article can be found <a href="https://github.com/erkandemirel/resilience4j-circuit-breaker-demo">here</a>.</p><p>Thanks for reading.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2a45198ca1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>