<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jaskaran  on Medium]]></title>
        <description><![CDATA[Stories by Jaskaran  on Medium]]></description>
        <link>https://medium.com/@jaskaran.1405?source=rss-f24a0a658256------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yEST-_cfVBthex5Vpf7qqg.jpeg</url>
            <title>Stories by Jaskaran  on Medium</title>
            <link>https://medium.com/@jaskaran.1405?source=rss-f24a0a658256------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:18:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jaskaran.1405/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Code to Cloud: A Complete Guide to Deploying a Java Microservice with Docker and Kubernetes]]></title>
            <link>https://medium.com/@jaskaran.1405/from-code-to-cloud-a-complete-guide-to-deploying-a-java-microservice-with-docker-and-kubernetes-ad770976aa01?source=rss-f24a0a658256------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad770976aa01</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Jaskaran ]]></dc:creator>
            <pubDate>Sat, 23 Aug 2025 14:34:44 GMT</pubDate>
            <atom:updated>2025-08-23T14:34:44.283Z</atom:updated>
            <content:encoded><![CDATA[<p>This article explains the entire workflow for taking a Java microservice, built with Spring Boot, and deploying it as a scalable, resilient application in a Kubernetes cluster. We will cover three main stages:</p><ol><li><strong>Building the Microservice</strong>: Creating a simple RESTful web service using Java and Spring Boot.</li><li><strong>Containerizing with Docker</strong>: Packaging the application and its dependencies into a lightweight, portable Docker image.</li><li><strong>Deploying to Kubernetes</strong>: Defining and running the application in a Kubernetes cluster, making it accessible and manageable.</li></ol><h3>Step 1: Building the Java Microservice</h3><p>The foundation of our deployment is a simple web application built using the Spring Boot framework, which makes it easy to create stand-alone, production-grade applications.</p><h4>The Application Code</h4><p>The entry point of our application is the MicroserviceApplication.java file.</p><pre>package com.example.microservice;<br><br>import org.springframework.boot.SpringApplication;<br>import org.springframework.boot.autoconfigure.SpringBootApplication;<br>import org.springframework.web.bind.annotation.GetMapping;<br>import org.springframework.web.bind.annotation.RestController;<br><br>@SpringBootApplication<br>@RestController<br>public class MicroserviceApplication {<br><br>    public static void main(String[] args) {<br>        SpringApplication.run(MicroserviceApplication.class, args);<br>    }<br><br>    @GetMapping(&quot;/&quot;)<br>    public String hello() {<br>        return &quot;Hello from the microservice!&quot;;<br>    }<br>}</pre><ul><li>@SpringBootApplication: A convenience annotation that enables auto-configuration and component scanning.</li><li>@RestController &amp; @GetMapping(&quot;/&quot;): These annotations define a simple REST endpoint. When you access the root URL (/) of the application, it will return the string &quot;Hello from the microservice!&quot;.</li></ul><h4>Project Dependencies and Configuration</h4><p>The project’s build and dependencies are managed by Maven in the pom.xml file. The key dependency is spring-boot-starter-web, which provides all the necessary components for building a web application.</p><p>For a production-ready service, it’s crucial for Kubernetes to understand the application’s health. We can achieve this by adding the <strong>Spring Boot Actuator</strong> dependency, which exposes health-check endpoints.</p><p>Here is the recommended change to your pom.xml:</p><pre>  &lt;dependency&gt;<br>   &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>   &lt;artifactId&gt;spring-boot-starter-web&lt;/artifactId&gt;<br>  &lt;/dependency&gt;<br>  &lt;dependency&gt;<br>   &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;<br>   &lt;artifactId&gt;spring-boot-starter-actuator&lt;/artifactId&gt;<br>  &lt;/dependency&gt;<br>  &lt;dependency&gt;</pre><p>Finally, src/main/resources/application.properties explicitly sets the application to run on port 8080, which is a common convention.</p><pre>server.port=8080</pre><p>Do a maven build; It will compiles the code, runs tests, and packages it into an executable JAR file located at target/microservice-0.0.1-SNAPSHOT.jar.</p><h3>Step 2: Containerizing with Docker</h3><p>Now that we have our application JAR, we need to package it into a container image. A container includes the application and all its runtime dependencies. We define how to build this image using a Dockerfile.</p><pre># Use an official OpenJDK runtime as a parent image<br>FROM eclipse-temurin:17-jre-alpine<br><br># Set the working directory<br>WORKDIR /app<br><br># Copy the built jar file<br>COPY target/microservice-0.0.1-SNAPSHOT.jar app.jar<br><br># Expose port 8080<br>EXPOSE 8080<br><br># Run the application<br>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;app.jar&quot;]</pre><p><strong>Dockerfile Breakdown:</strong></p><ol><li>FROM eclipse-temurin:17-jre-alpine: Starts with a minimal base image containing Java 17, which keeps our final image small and secure.</li><li>WORKDIR /app: Sets the working directory inside the container to /app.</li><li>COPY target/microservice-0.0.1-SNAPSHOT.jar app.jar: Copies the JAR file from your local target directory into the container and renames it to app.jar.</li><li>EXPOSE 8080: Informs Docker that the application inside the container will listen on port 8080.</li><li>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;app.jar&quot;]: Specifies the command to run when the container starts. This executes our Java application.</li></ol><p>To build the Docker image, run this command from the root directory:</p><pre># Replace &#39;your-dockerhub-username&#39; with your actual username or registry<br>docker build -t your-dockerhub-username/java-microservice:latest .</pre><p>After building, you would push it to a container registry so Kubernetes can access it:</p><pre>docker push your-dockerhub-username/java-microservice:latest</pre><h3>Step 3: Deploying to Kubernetes</h3><p>With our container image ready, we can now deploy it to Kubernetes. We will define two key objects: a Deployment to manage our application Pods and a Service to expose it to the internet.</p><h4>The Kubernetes Deployment</h4><p>A Deployment tells Kubernetes how to create and update instances of your application. It ensures that a specified number of application replicas (Pods) are running at all times.</p><p>Here is the deployment.enterprise.yaml file, enhanced with <strong>liveness and readiness probes</strong>. These probes are critical for robust deployments, as they help Kubernetes know if your application is running correctly or is ready to receive traffic.</p><p>The yaml file configuration ensures that 3 identical copies of my java-microservice application are always running. Each copy should be created from the java-microservice-sample:local Docker image, be given specific CPU and memory resources, and be monitored to ensure it&#39;s both alive and ready to receive traffic.</p><pre>   apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: java-microservice<br>  labels:<br>    app: java-microservice<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: java-microservice<br>  template:<br>    metadata:<br>      labels:<br>        app: java-microservice<br>    spec:<br>      containers:<br>        - name: java-microservice<br>          image: java-microservice-sample:local<br>          ports:<br>            - containerPort: 8080<br>          env:<br>            - name: SPRING_PROFILES_ACTIVE<br>              value: prod<br>          resources:<br>            requests:<br>              memory: &quot;256Mi&quot;<br>              cpu: &quot;250m&quot;<br>            limits:<br>              memory: &quot;512Mi&quot;<br>              cpu: &quot;500m&quot;<br>          livenessProbe:<br>            httpGet:<br>              path: /hello<br>              port: 8080<br>            initialDelaySeconds: 30<br>            periodSeconds: 10<br>          readinessProbe:<br>            httpGet:<br>              path: /hello<br>              port: 8080<br>            initialDelaySeconds: 10<br>            periodSeconds: 5</pre><p><strong>Deployment Breakdown:</strong></p><ul><li>replicas: 3: Tells the Deployment to run three identical Pods of our application for high availability.</li><li>selector: Defines how the Deployment finds which Pods to manage. The app: java-microservice label is key.</li><li>template: Contains the specification for the Pods that will be created.</li><li>metadata.labels: We label our Pods with app: java-microservice so the Deployment and Service can find them.</li><li>spec.containers.image: Specifies the Docker image to use. <strong>You must replace </strong><strong>java-microservice:latest with the image you pushed.</strong></li><li>livenessProbe &amp; readinessProbe: These use the Actuator endpoints we added. Kubernetes will periodically check these HTTP endpoints to ensure the application is healthy and ready to serve requests.</li></ul><h4>The Kubernetes Service</h4><p>A Service exposes your application as a network service. We use a LoadBalancer type to expose it to the internet via a cloud provider&#39;s load balancer. Here is the <strong>service.enterprise.yaml:</strong></p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: java-microservice<br>spec:<br>  type: LoadBalancer<br>  selector:<br>    app: java-microservice<br>  ports:<br>    - protocol: TCP<br>      port: 80<br>      targetPort: 8080</pre><p><strong>Service Breakdown:</strong></p><ul><li>type: LoadBalancer: Creates an external load balancer in your cloud environment to route external traffic to the service.</li><li>selector: app: java-microservice: This is the crucial link. The Service routes traffic to any Pod that has this label, which are the Pods managed by our Deployment.</li><li>ports: This defines the port mapping. Traffic coming into the load balancer on port: 80 (standard HTTP) will be forwarded to targetPort: 8080 on the Pods, which is where our Spring Boot application is listening.</li></ul><h4>Applying the Manifests</h4><p>To deploy the application, navigate to the project’s root directory and apply both manifest files using kubectl:</p><pre>kubectl apply -f k8s/</pre><p>This command tells Kubernetes to create or update the resources defined in the deployment.enterprise.yaml and service.enterprise.yaml files.</p><p>You can check the status of your deployment with these commands:</p><pre># Check if the deployment was successful and replicas are available<br>kubectl get deployment java-microservice<br><br># Check if the Pods are running<br>kubectl get pods -l app=java-microservice<br><br># Get the service details, including the external IP of the load balancer<br>kubectl get service java-microservice</pre><p>After a few minutes, the EXTERNAL-IP for the service will be assigned. You can then access your application by navigating to that IP address in your web browser, and you should see <strong><em>&quot;Hello from the microservice!&quot;.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad770976aa01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Chaos to Consistency: A Guide to the Saga Pattern with Spring Boot and Kafka]]></title>
            <link>https://medium.com/@jaskaran.1405/from-chaos-to-consistency-a-guide-to-the-saga-pattern-with-spring-boot-and-kafka-a54691d29775?source=rss-f24a0a658256------2</link>
            <guid isPermaLink="false">https://medium.com/p/a54691d29775</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[saga-pattern]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <dc:creator><![CDATA[Jaskaran ]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 18:47:43 GMT</pubDate>
            <atom:updated>2025-08-19T07:21:42.388Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of microservices, ensuring data consistency across distributed services can feel like orchestrating a chaotic symphony. How do you guarantee that a customer’s order, payment, and notification all succeed together, or fail gracefully, without bringing the whole system to a halt?</p><p>Enter the <strong>Saga pattern</strong>.</p><p>This article isn’t just a theoretical overview. We’ll walk you through building a resilient, event-driven e-commerce system using Spring Boot and Apache Kafka. You’ll learn how to manage complex workflows, handle failures with compensating actions, and ensure your microservices work together in perfect harmony.</p><h3>The Challenge: Why Traditional Transactions Fail in Microservices</h3><p>In a monolithic application, you could wrap a multi-step business process in a single ACID transaction. If one step failed, the database would roll everything back automatically. Simple.</p><p>But in a microservices architecture, services have their own private databases. A single transaction can’t span across them. The classic solution, a Two-Phase Commit (2PC), introduces a central coordinator that locks resources in each service until the entire transaction completes. This creates tight coupling and bottlenecks, undermining the very reason you chose microservices: scalability and resilience.</p><h3>The Solution: The Saga Pattern</h3><p>The Saga pattern offers a different approach. It breaks down a global transaction into a sequence of local transactions that are managed by each individual microservice.</p><ul><li>Each service completes its own transaction and publishes an event.</li><li>The next service in the chain listens for that event and executes its local transaction.</li><li>If any step fails, the saga executes <strong>compensating transactions</strong> to undo the work of the preceding steps.</li></ul><p>This ensures “eventual consistency” without the need for resource-locking, leading to a more robust and scalable system.</p><h3>Our Blueprint: An E-Commerce Order Saga</h3><p>To make this concrete, let’s design a simple order processing system.</p><p>Our system consists of three microservices communicating asynchronously via Apache Kafka:</p><ul><li><strong>Order Service:</strong> Initiates the saga when a customer places an order.</li><li><strong>Payment Service:</strong> Processes the payment for the order.</li><li><strong>Notification Service:</strong> Notifies the user about the outcome.</li></ul><p>Here’s a visual overview of the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q51OsERg-en8lEwNcJbVQg.png" /><figcaption>System Design</figcaption></figure><h3>Let’s Build It: A Step-by-Step Implementation</h3><p>Now, let’s get our hands dirty with some code. We’ll focus on the key parts of the implementation.</p><h3>1. The Shared Language: Defining Events</h3><p>To ensure all services speak the same language, we define our events in a shared Maven module. This prevents inconsistencies and makes the system easier to maintain.</p><p><strong>shared-events/src/main/java/com/example/events/OrderCreatedEvent.java</strong></p><pre>public class OrderCreatedEvent {<br>    private String orderId;<br>    private String userId;<br>    private java.math.BigDecimal amount;<br>    // Constructors, Getters, and Setters<br>}</pre><h3>2. Kicking Off the Saga: The Order Service</h3><p>The Order Service starts the process. After validating a new order, it saves it with a PENDING status and publishes an OrderCreatedEvent.</p><p><strong>order-service/src/main/java/com/example/order/OrderService.java</strong></p><pre>@Service<br>public class OrderService {<br><br>    @Autowired<br>    private KafkaTemplate&lt;String, OrderCreatedEvent&gt; kafkaTemplate;<br><br>    private final Map&lt;String, Order&gt; pendingOrders = new ConcurrentHashMap&lt;&gt;();<br><br>    public void createOrder(CreateOrderRequest request) {<br>        // 1. Validate and save the order with PENDING status<br>        String orderId = &quot;order-&quot; + System.currentTimeMillis();<br>        Order order = new Order(orderId, request.getUserId(), request.getAmount(), &quot;PENDING&quot;);<br>        pendingOrders.put(orderId, order);<br>        <br>        // 2. Create the event<br>        OrderCreatedEvent event = new OrderCreatedEvent(orderId, order.getUserId(), order.getAmount());<br><br>        // 3. Publish to Kafka to trigger the next step<br>        kafkaTemplate.send(&quot;order-created-topic&quot;, event);<br>        log.info(&quot;Published OrderCreatedEvent for orderId: {}&quot;, orderId);<br>    }<br>    // ... methods to handle success and failure events<br>}</pre><h3>3. The Safety Net: Compensation and Timeouts</h3><p>What if the Payment Service never responds? We can’t leave the order in a PENDING state forever. The Order Service runs a scheduled task to check for timed-out orders.</p><p>If an order remains pending for too long (e.g., 15 seconds), the watcher triggers a compensating action by publishing an OrderFailedEvent.</p><p><strong>order-service/src/main/java/com/example/order/OrderTimeoutWatcher.java</strong></p><pre>@Component<br>public class OrderTimeoutWatcher {<br><br>    private static final long TIMEOUT_MS = 15000;<br><br>    @Autowired<br>    private OrderService orderService; // Assuming this holds the state<br><br>    @Autowired<br>    private KafkaTemplate&lt;String, OrderFailedEvent&gt; kafkaTemplate;<br><br>    @Scheduled(fixedRate = 5000)<br>    public void checkForTimedOutOrders() {<br>        log.debug(&quot;Checking for timed-out orders...&quot;);<br>        orderService.getPendingOrders().forEach((orderId, order) -&gt; {<br>            if (System.currentTimeMillis() - order.getCreatedAt() &gt; TIMEOUT_MS) {<br>                log.warn(&quot;Order {} timed out. Publishing compensation event.&quot;, orderId);<br>                <br>                // Publish a compensation event<br>                OrderFailedEvent event = new OrderFailedEvent(orderId, &quot;Order timed out.&quot;);<br>                kafkaTemplate.send(&quot;order-failed-topic&quot;, event);<br><br>                // Update local status to FAILED<br>                orderService.handleOrderFailure(orderId);<br>            }<br>        });<br>    }<br>}</pre><h3>4. Processing the Payment: The Payment Service</h3><p>The Payment Service listens for OrderCreatedEvent, processes the payment, and then publishes the result.</p><p><strong>payment-service/src/main/java/com/example/payment/PaymentListener.java</strong></p><pre>@Service<br>public class PaymentListener {<br><br>    @KafkaListener(topics = &quot;order-created-topic&quot;, groupId = &quot;payment-group&quot;)<br>    public void handleOrderCreated(OrderCreatedEvent event) {<br>        log.info(&quot;Received OrderCreatedEvent: {}&quot;, event.getOrderId());<br>        <br>        try {<br>            // Simulate payment processing<br>            processPayment(event);<br>            <br>            // Publish success event<br>            PaymentProcessedEvent successEvent = new PaymentProcessedEvent(event.getOrderId());<br>            kafkaTemplate.send(&quot;payment-processed-topic&quot;, successEvent);<br><br>        } catch (Exception e) {<br>            // Publish failure event (part of compensation)<br>            PaymentFailedEvent failedEvent = new PaymentFailedEvent(event.getOrderId(), e.getMessage());<br>            kafkaTemplate.send(&quot;payment-failed-topic&quot;, failedEvent);<br>        }<br>    }<br>}</pre><h3>5. Closing the Loop: The Notification Service</h3><p>The final step in our saga is to inform the user about the outcome of their order. The <strong>Notification Service</strong> is a terminal service in this workflow; it listens for final state events (PaymentProcessedEvent and OrderFailedEvent) and acts on them, but it doesn&#39;t publish new events to continue the saga.</p><p>Its role is crucial for user experience, providing clear communication and building trust in the system.</p><p>The service listens to two different topics to handle both success and failure scenarios.</p><h4>Implementation: NotificationListener.java</h4><p>Here is the core logic for the Notification Service. It contains two listener methods, each subscribed to a different Kafka topic. This allows it to react appropriately to different outcomes for the same business process.</p><pre>package com.example.notification;<br><br>import com.example.events.OrderFailedEvent;<br>import com.example.events.PaymentProcessedEvent;<br>import org.slf4j.Logger;<br>import org.slf4j.LoggerFactory;<br>import org.springframework.kafka.annotation.KafkaListener;<br>import org.springframework.stereotype.Service;<br><br>@Service<br>public class NotificationListener {<br><br>    private static final Logger log = LoggerFactory.getLogger(NotificationListener.class);<br><br>    /**<br>     * Listens for successful payment events to notify the user.<br>     * @param event The event indicating a payment was successfully processed.<br>     */<br>    @KafkaListener(topics = &quot;payment-processed-topic&quot;, groupId = &quot;notification-group&quot;)<br>    public void handlePaymentSuccess(PaymentProcessedEvent event) {<br>        log.info(&quot;Received PaymentProcessedEvent for orderId: {}. Sending success notification.&quot;, event.getOrderId());<br>        // In a real application, you would integrate with an email or SMS service here.<br>        // For example: emailService.sendOrderConfirmation(event.getOrderId());<br>    }<br><br>    /**<br>     * Listens for failed order events (from payment failure or timeout) to notify the user.<br>     * @param event The event indicating the order has failed.<br>     */<br>    @KafkaListener(topics = &quot;order-failed-topic&quot;, groupId = &quot;notification-group&quot;)<br>    public void handleOrderFailure(OrderFailedEvent event) {<br>        log.info(&quot;Received OrderFailedEvent for orderId: {}. Reason: {}. Sending failure notification.&quot;, <br>                 event.getOrderId(), event.getReason());<br>        // In a real application, you would explain the failure and suggest next steps.<br>        // For example: emailService.sendOrderFailedNotification(event.getOrderId(), event.getReason());<br>    }<br>}</pre><p>By handling both success and failure events, the Notification Service ensures that the user is never left in the dark, making the entire system more robust and user-friendly.</p><h3>Error Handling and Best Practices</h3><p>A distributed system must be designed for failure. Here are some best practices we followed:</p><ul><li><strong>Idempotent Consumers:</strong> Ensure that processing the same event multiple times doesn&#39;t cause duplicate actions.</li><li><strong>Centralized Logging:</strong> Use SLF4J and a structured logging format to trace a saga&#39;s flow across services using a unique correlation ID (like orderId).</li><li><strong>Configuration Management:</strong> Externalize Kafka broker addresses, topic names, and group IDs in application.properties for flexibility across environments.</li><li><strong>Monitoring:</strong> Set up dashboards to monitor Kafka topic lag and the state of active sagas.</li></ul><h3>Conclusion</h3><p>The Saga pattern is a powerful tool for managing complex, long-running processes in a microservices architecture. By embracing event-driven communication and planning for failure with compensation logic, you can build systems that are not only scalable but also remarkably resilient. With frameworks like Spring Boot and message brokers like Kafka, implementing this pattern has never been more accessible.</p><p>## Code Repository</p><p>You can find the complete source code for this project on <a href="https://github.com/jaskaran14/saga-pattern-springboot-kafka">GitHub</a>:</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a54691d29775" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Developer’s Guide to Centralized Logging with the ELK Stack and Docker Compose]]></title>
            <link>https://medium.com/@jaskaran.1405/a-developers-guide-to-centralized-logging-with-the-elk-stack-and-docker-compose-cee8acd4cbbc?source=rss-f24a0a658256------2</link>
            <guid isPermaLink="false">https://medium.com/p/cee8acd4cbbc</guid>
            <dc:creator><![CDATA[Jaskaran ]]></dc:creator>
            <pubDate>Sun, 10 Aug 2025 13:39:17 GMT</pubDate>
            <atom:updated>2025-08-10T18:39:41.728Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s complex application landscape, understanding what’s happening inside your services is not just a luxury — it’s a necessity. Centralized logging is the key to effective debugging, monitoring, and security analysis. The ELK Stack (Elasticsearch, Logstash, and Kibana) is the industry-standard open-source solution for this, and with Docker, setting it up has never been easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*8Q1OdqcRe-ekG751QLfRfA.png" /></figure><p>In this guide, we’ll build a complete, production-ready logging pipeline from scratch using Docker Compose. You’ll learn how to ship logs from your applications, process them, store them, and create alerts for critical events like exceptions.</p><h3>What We’ll Build</h3><p>Our final architecture will look like this:</p><ul><li><strong>A Sample App</strong>: A simple container that generates both regular [INFO] logs and periodic [ERROR] logs.</li><li><strong>Logstash</strong>: The workhorse that collects, parses, and transforms logs before sending them on.</li><li><strong>Elasticsearch</strong>: A powerful search engine that will store and index our logs for fast retrieval.</li><li><strong>Kibana</strong>: A beautiful web interface for searching, visualizing, and creating dashboards and alerts from our log data.</li></ul><p>Let’s get started!</p><h3>Step 1: The docker-compose.yml File</h3><p>First, create a docker-compose.yml file. This single file defines our entire stack. It includes our sample app, the three ELK components, and the networking required for them to communicate.</p><pre>version: &#39;3.8&#39;<br><br>services:<br>  # Sample application to generate logs<br>  app:<br>    image: alpine<br>    container_name: my-app<br>    # This command now uses a counter to generate an [ERROR] log with a fake<br>    # exception every 4th iteration (every 20 seconds).<br>    command: &gt;<br>      sh -c &#39;i=0; while true; do<br>        i=$((i+1));<br>        if [ $$(($$i % 4)) -eq 0 ]; then echo &quot;[ERROR] Something went wrong! Exception: NullPointerException at com.example.UserService:42&quot;;<br>        else echo &quot;[INFO] Hello from the app service at `date`&quot;; fi;<br>        sleep 5; done&#39;<br>    logging:<br>      driver: &quot;gelf&quot;<br>      options:<br>        gelf-address: &quot;udp://host.docker.internal:12201&quot;<br>        tag: &quot;my-app&quot; # This tag helps identify the log source in Kibana<br><br>  elasticsearch:<br>    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.1<br>    container_name: elasticsearch<br>    environment:<br>      - discovery.type=single-node<br>      - ES_JAVA_OPTS=-Xms512m -Xmx512m<br>      # Disable security for development. DO NOT use in production.<br>      - xpack.security.enabled=false<br>    volumes:<br>      - esdata:/usr/share/elasticsearch/data # Persist Elasticsearch data<br>    ports:<br>      - &quot;9200:9200&quot;<br>      - &quot;9300:9300&quot;<br>    networks:<br>      - elk<br>    healthcheck:<br>      # Check that Elasticsearch is healthy before starting other services<br>      test: [&quot;CMD-SHELL&quot;, &quot;curl -s http://localhost:9200/_cluster/health | grep -vq &#39;\&quot;status\&quot;:\&quot;red\&quot;&#39;&quot;]<br>      interval: 10s<br>      timeout: 10s<br>      retries: 10<br><br>  logstash:<br>    image: docker.elastic.co/logstash/logstash:8.11.1<br>    container_name: logstash<br>    volumes:<br>      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro<br>    ports:<br>      - &quot;5044:5044&quot;<br>      - &quot;12201:12201/udp&quot; # GELF input<br>    depends_on:<br>      elasticsearch:<br>        condition: service_healthy<br>    networks:<br>      - elk<br><br>  kibana:<br>    image: docker.elastic.co/kibana/kibana:8.11.1<br>    container_name: kibana<br>    environment:<br>      ELASTICSEARCH_HOSTS: http://elasticsearch:9200<br>    ports:<br>      - &quot;5601:5601&quot;<br>    depends_on:<br>      elasticsearch:<br>        condition: service_healthy<br>    networks:<br>      - elk<br><br>networks:<br>  elk:<br>    driver: bridge<br><br>volumes:<br>  esdata:<br>    driver: local</pre><p><strong>Key Configurations:</strong></p><ul><li><strong>app.logging</strong>: We configure our app to use the gelf (Graylog Extended Log Format) driver. This tells Docker to ship the container&#39;s logs directly to Logstash over UDP.</li><li><strong>elasticsearch.healthcheck</strong>: This is crucial. It prevents Logstash and Kibana from starting until Elasticsearch is fully initialized and healthy, avoiding common startup race conditions.</li><li><strong>logstash.volumes</strong>: We mount a local configuration file into the Logstash container, which we&#39;ll create next.</li></ul><h3>Step 2: The Logstash Pipeline</h3><p>Logstash needs to be told where to listen for logs and where to send them. Create a logstash/pipeline directory and add the following logstash.conf file inside it.</p><pre># This configuration tells Logstash to listen for logs from Docker&#39;s<br># gelf logging driver and send them to Elasticsearch.<br><br>input {<br>  gelf {<br>    port =&gt; 12201<br>    type =&gt; &quot;docker&quot;<br>  }<br>}<br><br># The filter section is where you can parse and transform logs.<br># It&#39;s empty for now but is a powerful feature of Logstash.<br><br>output {<br>  elasticsearch {<br>    hosts =&gt; [&quot;http://elasticsearch:9200&quot;]<br>    index =&gt; &quot;logstash-%{+YYYY.MM.dd}&quot;<br>  }<br>}</pre><p>This simple configuration does two things:</p><ol><li><strong>input</strong>: Listens for incoming GELF traffic on UDP port 12201.</li><li><strong>output</strong>: Forwards any received logs to our elasticsearch service, placing them in a new, daily index (e.g., logstash-2025.08.10).</li></ol><h3>Step 3: <strong>The Role of Elasticsearch</strong></h3><p>In this logging pipeline, <strong>Elasticsearch</strong> serves as the central data store and analytics engine. After Logstash processes the incoming logs from your application, it sends them to Elasticsearch. Elasticsearch’s primary job is to efficiently store these logs and, more importantly, to <strong>index</strong> them. This indexing process is like creating a highly sophisticated, searchable catalog of every piece of data in your logs.</p><p>When you use Kibana to search for an error, create a visualization, or trigger an alert, Kibana sends a query to Elasticsearch, which uses its powerful indexes to find and return the relevant log data with incredible speed, even across millions or billions of log entries.</p><h3>Step 4: Launch the Stack</h3><p>With your files in place, open a terminal and run:</p><pre>docker-compose up -d</pre><p>This will download the images and start all four containers in the background. It may take a minute or two for everything to initialize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qgy_m-pkMX_tuFP9Gzd7Lg.png" /><figcaption>Command execution results</figcaption></figure><p><strong>🚨 A Note for Windows Users:</strong> The Windows Firewall often blocks the UDP traffic from Docker to Logstash. You must create an <strong>Inbound Rule</strong> to <strong>allow UDP traffic on port 12201</strong> for this setup to work.</p><h3>Step 5: Visualize Logs in Kibana</h3><p>Once the stack is running, open your browser to <a href="http://localhost:5601."><strong>http://localhost:5601</strong>.</a></p><p>Before you can see logs, you need to tell Kibana where to find them.</p><h4>Click the main menu (☰) and go to <strong>Stack Management &gt; Data Views</strong>.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*rIw4PqThQr34mJ4jITuUCA.png" /></figure><p>2. Click <strong>Create data view</strong>.</p><p>3. Use logstash-* as the <strong>Index pattern</strong> and select @timestamp as the <strong>Timestamp field</strong>.</p><p>3. Click <strong>Create data view</strong>.</p><p>Now, head back to the main menu and click <strong>Discover</strong>. You should see a live stream of logs from your app, including the [INFO] and [ERROR] messages!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_v20hkmYbsOBv1MxX35s5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1OMvXUUgF8igOkuB8ja7ew.png" /></figure><h3>Step 6: Create Alerts for Exceptions</h3><p>Seeing logs is great, but being proactively notified of problems is even better. Let’s create a rule to alert us whenever an exception occurs.</p><ol><li>In Kibana, go to <strong>Observability &gt; Alerts</strong>.</li><li>Click <strong>Create rule</strong> and select the <strong>Log threshold</strong> type.</li><li>Configure the rule:</li></ol><ul><li><strong>Name</strong>: Application Exception Detected</li><li><strong>Index pattern</strong>: logstash-*</li><li><strong>Condition</strong>: Use the KQL query message: *Exception*</li><li><strong>Threshold</strong>: Set it to trigger when the count is at least 1 over the last 1 minute.</li></ul><p>4. Under <strong>Actions</strong>, select <strong>Server log</strong> to have alerts written to the Kibana logs (you can easily configure email or Slack connectors later).</p><p>5. Click <strong>Create rule</strong>.</p><p>That’s it! Within a minute of your app logging an error, the alert will become active, demonstrating the power of your new monitoring pipeline.</p><h3>Conclusion</h3><p>You now have a fully functional, scalable logging infrastructure. This setup provides a solid foundation that you can expand upon. You can start shipping logs from your real applications, build rich visualization dashboards, and create more sophisticated alerting rules to gain deep insights into your systems’ health and performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cee8acd4cbbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Database to Data Stream: A Practical Guide to Real-Time CDC with Debezium, Kafka, and Spring…]]></title>
            <link>https://medium.com/@jaskaran.1405/from-database-to-data-stream-a-practical-guide-to-real-time-cdc-with-debezium-kafka-and-spring-70d6d1d9ae8d?source=rss-f24a0a658256------2</link>
            <guid isPermaLink="false">https://medium.com/p/70d6d1d9ae8d</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cdc]]></category>
            <dc:creator><![CDATA[Jaskaran ]]></dc:creator>
            <pubDate>Wed, 06 Aug 2025 18:25:24 GMT</pubDate>
            <atom:updated>2025-08-06T18:25:24.414Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Database to Data Stream: A Practical Guide to Real-Time CDC with Debezium, Kafka, and Spring Boot</h3><p>In modern software architecture, the need to react to data changes <em>as they happen</em> is more critical than ever. Whether it’s for updating search indexes, invalidating caches, feeding analytics pipelines, or enabling event-driven microservices, the days of slow, nightly batch jobs are numbered. The challenge? Tapping into a traditional database, the source of truth for so many applications, without putting an undue load on it or modifying the application’s core logic.</p><p>This is where Change Data Capture (CDC) comes in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*_n8mRP20hc9OAr78c3atIw.jpeg" /></figure><p>In this article, we’ll walk through a complete, practical implementation of a high-throughput, fault-tolerant CDC pipeline. We will build a system that captures every single INSERT, UPDATE, and DELETE from a PostgreSQL table and streams them in real-time to a powerful Spring Boot consumer.</p><p><strong>Our Technology Stack:</strong></p><ul><li><strong>PostgreSQL</strong>: Our transactional database of record.</li><li><strong>Debezium</strong>: The CDC engine that magically turns database changes into a stream of events.</li><li><strong>Apache Kafka</strong>: The scalable, durable streaming backbone that decouples our system.</li><li><strong>Spring Boot</strong>: The framework for our robust, high-performance consumer microservice.</li><li><strong>Docker</strong>: To containerize and orchestrate all our components seamlessly.</li></ul><p>By the end, you’ll have a blueprint for building your own real-time data pipelines.</p><h3>The Architecture: A Bird’s-Eye View</h3><p>Before we dive into code, let’s understand the flow of data.</p><ol><li><strong>The Change</strong>: An application performs a standard operation (e.g., INSERT INTO events ...) on our PostgreSQL database.</li><li><strong>The Capture</strong>: Debezium, running as a Kafka Connect source connector, is tailing PostgreSQL’s Write-Ahead Log (WAL). It sees the INSERT and creates a detailed JSON event describing the change.</li><li><strong>The Stream</strong>: Debezium publishes this event to a specific Kafka topic.</li><li><strong>The Consumption</strong>: Our Spring Boot microservice, subscribed to this topic, consumes the event.</li><li><strong>The Reaction</strong>: The microservice can now perform any action: update a cache, call another API, save to a data warehouse, etc. If it fails to process a message, it’s safely routed to a Dead-Letter Topic (DLT) for later analysis.</li></ol><h3>Step 1: Laying the Foundation with Docker Compose</h3><p>The easiest way to run our stack is with Docker. Our docker-compose.yml file defines all the necessary services.</p><pre>version: &#39;3.8&#39;<br>services:<br>  zookeeper:<br>    image: confluentinc/cp-zookeeper:7.5.0<br>    environment:<br>      ZOOKEEPER_CLIENT_PORT: 2181<br><br>  kafka:<br>    image: confluentinc/cp-kafka:7.5.0<br>    depends_on: [zookeeper]<br>    ports:<br>      - &quot;9092:9092&quot;       # For internal Docker network traffic<br>      - &quot;29092:29092&quot;     # For external traffic from your host machine<br>    environment:<br>      KAFKA_BROKER_ID: 1<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:29092<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092<br>      # ... other Kafka settings<br><br>  postgres:<br>    image: postgres:15<br>    ports: [&quot;5432:5432&quot;]<br>    environment:<br>      POSTGRES_USER: postgres<br>      POSTGRES_PASSWORD: postgres<br>      POSTGRES_DB: postgres<br>    # IMPORTANT: Debezium requires the WAL level to be &#39;logical&#39;<br>    command: postgres -c wal_level=logical<br><br>  connect: # This is our Debezium service<br>    image: debezium/connect:2.4<br>    depends_on: [kafka, postgres]<br>    ports: [&quot;8083:8083&quot;]<br>    environment:<br>      BOOTSTRAP_SERVERS: kafka:9092<br>      GROUP_ID: 1<br>      # ... other Kafka Connect settings</pre><p>The key takeaways here are the KAFKA_ADVERTISED_LISTENERS which allow our local Spring Boot app to connect to Kafka inside Docker, and the wal_level=logical command for PostgreSQL, which is a mandatory prerequisite for Debezium.</p><h3>Step 2: Configuring the Debezium Connector</h3><p>With our services running, we need to tell Debezium what to watch. We do this by POSTing a JSON configuration to the Kafka Connect REST API.</p><p><strong>register-postgres.json</strong></p><pre>{<br>  &quot;name&quot;: &quot;postgres-connector&quot;,<br>  &quot;config&quot;: {<br>    &quot;connector.class&quot;: &quot;io.debezium.connector.postgresql.PostgresConnector&quot;,<br>    &quot;database.hostname&quot;: &quot;postgres&quot;,<br>    &quot;database.port&quot;: &quot;5432&quot;,<br>    &quot;database.user&quot;: &quot;postgres&quot;,<br>    &quot;database.password&quot;: &quot;postgres&quot;,<br>    &quot;database.dbname&quot;: &quot;postgres&quot;,<br>    &quot;database.server.name&quot;: &quot;postgrescdc&quot;,<br>    &quot;table.include.list&quot;: &quot;events.events&quot;,<br>    &quot;topic.prefix&quot;: &quot;postgrescdc&quot;,<br>    &quot;plugin.name&quot;: &quot;pgoutput&quot;<br>  }<br>}</pre><p>This configuration tells Debezium to connect to our postgres container and watch the events.events(schema.table) table. It will publish change events to a topic named &lt;database.server.name&gt;.&lt;table.include.list&gt;, which in our case is postgrescdc.events.events.</p><h3>Step 3: Building a High-Performance, Resilient Consumer</h3><p>Now for the fun part: our Spring Boot application. Our goal isn’t just to consume messages, but to do so at scale and with robust error handling.</p><h4>Performance Tuning</h4><p>To handle a high volume of messages (e.g., 1000 records/second), we need to process in batches and with multiple threads. We configure this in application.properties:</p><pre><br># Kafka Broker connection<br>spring.kafka.bootstrap-servers=localhost:29092<br><br># Performance Tuning: Enable batch listening and concurrency<br>spring.kafka.listener.type=batch<br>spring.kafka.listener.concurrency=3<br>spring.kafka.consumer.max-poll-records=500</pre><ul><li>listener.type=batch: Tells Spring to deliver messages in lists, not one by one.</li><li>listener.concurrency=3: Runs three consumer threads in parallel (ensure your topic has at least 3 partitions for this to be effective!).</li><li>max-poll-records=500: Fetches up to 500 records at a time per poll.</li></ul><h4>The Batch Consumer</h4><p>Our consumer service is now designed to handle a List of messages.</p><p><strong>EventKafkaConsumerService.java</strong></p><pre>@Service<br>public class EventKafkaConsumerService {<br><br>    private static final Logger LOGGER = LoggerFactory.getLogger(EventKafkaConsumerService.class);<br>    private final ObjectMapper objectMapper;<br>    // ... constructor<br><br>    @KafkaListener(topics = &quot;postgrescdc.events.events&quot;, groupId = &quot;event-processor-group&quot;)<br>    public void consume(List&lt;String&gt; messages) {<br>        LOGGER.info(&quot;Received a batch of {} events.&quot;, messages.size());<br>        for (int i = 0; i &lt; messages.size(); i++) {<br>            String message = messages.get(i);<br>            try {<br>                // ... parsing logic ...<br>                String operation = payload.get(&quot;op&quot;).asText(); // &#39;c&#39;, &#39;u&#39;, &#39;d&#39;, &#39;r&#39;<br>                switch (operation) {<br>                    // ... handle different operations ...<br>                }<br>            } catch (JsonProcessingException e) {<br>                // ... error handling ...<br>            }<br>        }<br>    }<br>    // ... handler methods ...<br>}</pre><h4>Bulletproof Error Handling with a DLT</h4><p>What happens if we receive a malformed message (a “poison pill”)? We don’t want it to block our entire pipeline. The solution is a Dead-Letter Topic (DLT). We can configure a global error handler to automatically catch exceptions and route the failing message to a DLT.</p><p><strong>KafkaConsumerConfig.java</strong></p><pre>@Configuration<br>public class KafkaConsumerConfig {<br><br>    @Bean<br>    public DefaultErrorHandler errorHandler(KafkaTemplate&lt;String, String&gt; template) {<br>        // Publish the failed record to a topic with a &quot;.DLT&quot; suffix.<br>        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template,<br>                (record, exception) -&gt; new TopicPartition(record.topic() + &quot;.DLT&quot;, record.partition()));<br><br>        // Send failing messages straight to the DLT with no retries.<br>        return new DefaultErrorHandler(recoverer, new FixedBackOff(0L, 1L));<br>    }<br>}</pre><p>This configuration is incredibly powerful. Now, if our consume method throws an exception, Spring Kafka will automatically catch it and publish the problematic message to postgrescdc.events.events.DLT.</p><p>We can then have a separate, simple consumer to monitor this DLT for failures.</p><p><strong>DeadLetterTopicConsumerService.java</strong></p><pre>@Service<br>public class DeadLetterTopicConsumerService {<br>    private static final Logger LOGGER = LoggerFactory.getLogger(DeadLetterTopicConsumerService.class);<br><br>    @KafkaListener(topics = &quot;postgrescdc.events.events.DLT&quot;, groupId = &quot;dlt-processor-group&quot;)<br>    public void consumeDltMessage(String message,<br>                                  @Header(KafkaHeaders.EXCEPTION_MESSAGE) String exceptionMessage) {<br>        LOGGER.error(&quot;Received message from DLT:&quot;);<br>        LOGGER.error(&quot;  Message: {}&quot;, message);<br>        LOGGER.error(&quot;  Exception: {}&quot;, exceptionMessage);<br>        // TODO: Add alerting logic here!<br>    }<br>}</pre><h3>Conclusion</h3><p>We’ve successfully built a complete, end-to-end Change Data Capture pipeline. This architecture is not just a theoretical exercise; it’s a production-ready pattern that provides immense value. It decouples your systems, enables real-time reactivity, scales horizontally, and is resilient to failure.</p><p>By leveraging the power of Debezium to tap into the database’s own source of truth, Kafka to provide a durable event backbone, and Spring Boot to build intelligent consumers, you can unlock the data in your monolithic systems and propel your architecture into the world of real-time events.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70d6d1d9ae8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>