<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[CloudCraft - Medium]]></title>
        <description><![CDATA[CloudCraft is your place for deep-dive content on cloud computing, containerization, K8s, Docker, infrastructure automation, and cloud-native architecture. - Medium]]></description>
        <link>https://medium.com/cloudnativepub?source=rss----4b146f5391bb---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>CloudCraft - Medium</title>
            <link>https://medium.com/cloudnativepub?source=rss----4b146f5391bb---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:18:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cloudnativepub" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Private Supabase Bucket Can Still Leak Files Through Your App]]></title>
            <link>https://medium.com/cloudnativepub/a-private-supabase-bucket-can-still-leak-files-through-your-app-2ae083ce5354?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/2ae083ce5354</guid>
            <category><![CDATA[supabase]]></category>
            <category><![CDATA[cloud-security]]></category>
            <dc:creator><![CDATA[jacob perks]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 22:06:46 GMT</pubDate>
            <atom:updated>2026-10-25T22:06:45.433Z</atom:updated>
            <content:encoded><![CDATA[<h4>Follow the document through the server. Storage privacy cannot make the application’s ownership decision for it.</h4><figure><img alt="Engraved locked filing cabinet with a red side opening through which a hand can still take a document." src="https://cdn-images-1.medium.com/proxy/1*zusND51juqw17r4k_waq-g.png" /></figure><p>During an application review, I found a document-processing path that checked whether the caller was logged in. It did not establish whether the requested document belonged to that caller.</p><p>The server could retrieve the file. The missing decision was whether it should retrieve that file for this person.</p><p>That is worth checking even when your Supabase bucket is private. A backend acting with elevated privileges can become a second route to information that the browser cannot access directly.</p><h3>Follow one document request</h3><p>Consider this illustrative document-summary feature:</p><ol><li>Customer A signs in and requests a summary of a document.</li><li>The endpoint checks that A has a valid session.</li><li>It accepts the requested file reference.</li><li>The server reads the file using privileged storage access.</li><li>It returns a summary to A.</li></ol><p>Now make the requested document belong to customer B. If no step establishes A’s permission to that document, the valid session answers the wrong question.</p><p>The original finding established a missing ownership check. This Supabase example shows the same kind of application-level decision; it is not a claim that every private bucket has this problem.</p><h3>Put authorization before the privileged read</h3><p>The corrected sequence needs a deliberate relationship check:</p><p><strong>Stage:</strong> Identify the caller<br><strong>Decision:</strong> Is there a valid authenticated customer?</p><p><strong>Stage:</strong> Resolve the document<br><strong>Decision:</strong> Which trusted document record is being requested?</p><p><strong>Stage:</strong> Authorize the action<br><strong>Decision:</strong> May this customer process this document under the product’s rules?</p><p><strong>Stage:</strong> Read and process<br><strong>Decision:</strong> Retrieve only the authorized object, then return the permitted result</p><p>A shared-workspace product may permit several members to use a document. A personal-document product may permit only its owner. The relationship must match the actual product rather than a convenient assumption about folder names.</p><p>The server should derive the storage reference from trusted records after establishing access, rather than treat an arbitrary supplied path as permission.</p><p><a href="https://supabase.com/docs/guides/storage/security/access-control">Supabase documents storage access control and privileged access</a>. The application still has to decide when its privileged operation is justified for the caller.</p><h3>A signed link moves the decision earlier</h3><p>An application may give the customer a temporary signed URL instead of returning the file directly. The important question becomes who can obtain that link.</p><p>If the endpoint gives customer A a valid link to customer B’s private document, the authorization failure has already happened. A short expiry limits duration; it does not repair the decision to issue it.</p><p>Check where links travel afterwards too: emails, logs and shared messages can have different audiences. Some products intentionally allow anyone holding a link to view a file. That should be an explicit sharing feature with understood limits.</p><h3>Test every operation that touches the file</h3><p>For an owned or authorized test environment, create synthetic documents under two ordinary accounts. Exercise the permitted owner’s actions first, then repeat the relevant requests as the unrelated account.</p><p>Include preview, text extraction, download-link creation, replacement and deletion wherever those features exist. A secure upload route does not demonstrate that the parser uses the same rule.</p><p>For shared workspaces, add a permitted colleague and a removed member. The results should follow the current membership rules. Inspect both the response and the resulting file state; hiding an item in the sidebar is not sufficient.</p><p>Keep an evidence record that names the operation and account relationship. “Storage passed” hides which paths were actually tested.</p><h3>A summary can be a data leak too</h3><p>File security is not limited to whether someone can download the original PDF. A preview, extracted text, generated summary or search result can reveal its contents just as effectively.</p><p>In the illustrative summary feature, refusing a direct download would not help if the processing endpoint still returned the important paragraphs. The authorization boundary needs to cover the information derived from the file, not only the storage object.</p><p>List where those derivatives are stored. Does the text enter a search index? Is the summary cached by document identifier? Can a background job write its result to a record another workspace can read? Each step should preserve the intended audience.</p><p>A cache keyed only by a document name, for example, deserves investigation in a multi-customer application. That is a design question, not proof of a specific vulnerability. The review needs to establish the actual key, ownership model and retrieval path before making a claim.</p><p>The useful unit of review is therefore the document and its outputs. Looking only at the bucket misses much of the feature customers are paying for.</p><h3>Membership can change while work is queued</h3><p>Suppose a workspace member requests a long-running analysis and then loses access before it finishes. Should the result still be available to that person? Should it remain available to the workspace? The product needs an explicit answer.</p><p>The same issue appears when a document is deleted or moved while a job is waiting. A queued task may contain a reference that was valid at request time but no longer represents an authorized operation.</p><p>For sensitive workflows, inspect both acceptance of the task and delivery of its result. Decide which permissions must be checked again, and ensure failures do not expose document content in logs or error messages. Do not assume a queue makes the earlier authorization decision permanently valid.</p><p>Test this with synthetic records: request the operation, change the relevant membership or document state and observe the result. The expected behaviour should come from the product’s rules, not whichever path happens to win the race.</p><p>This is a useful place for an auditor to distinguish an implementation defect from an unresolved product decision. Both need attention, but they need different fixes.</p><h3>Make sharing intentional and revocation understandable</h3><p>Some products are supposed to share files outside the owner’s account. That does not make their storage design wrong. It means the sharing rule needs to be reviewed separately from private access.</p><p>Can the owner choose who receives access? Does a shared link expose just one object or a larger folder? What happens when sharing is disabled? If an already-issued link remains usable until expiry, the interface should not promise instant revocation unless the implementation actually provides it.</p><p>The security review should follow the same questions for replacements. Uploading a new version may change the contents available through an old link. A customer who intended to share version one might not expect later material to appear there.</p><p>None of this requires banning useful sharing features. It requires making the audience, duration and version clear, then testing that the underlying requests enforce that agreement. A private bucket is a useful default; the app still decides every exception it creates.</p><h3>Fix the shared decision when the paths share it</h3><p>If several endpoints repeat the same incomplete check, patching only the first report leaves the others uncertain. Look for a consistent authorization routine that each sensitive path can use, while preserving operation-specific rules.</p><p>Then retest the refused request and normal use. Do not publish real file paths, customer contents or signed URLs as proof.</p><p>For an app that handles customer documents, the useful scope follows the file from storage through every feature that can return or modify it. The bucket setting is one stop along that path.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ae083ce5354" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/a-private-supabase-bucket-can-still-leak-files-through-your-app-2ae083ce5354">A Private Supabase Bucket Can Still Leak Files Through Your App</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Bomb in Memory: Anatomy of a Hazelcast Production Outage]]></title>
            <link>https://medium.com/cloudnativepub/the-hidden-bomb-in-memory-anatomy-of-a-hazelcast-production-outage-832244ed1700?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/832244ed1700</guid>
            <category><![CDATA[hazelcast]]></category>
            <category><![CDATA[outage]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[memory-management]]></category>
            <dc:creator><![CDATA[Alireza Alimohammadi]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 12:32:13 GMT</pubDate>
            <atom:updated>2026-10-20T12:32:12.657Z</atom:updated>
            <content:encoded><![CDATA[<p>How an unapplied <strong>eviction policy</strong> and a 5GB heap froze 10 critical microservices after 365 days</p><p>It was around 10:00 AM on an ordinary weekday. Our team was focused on routine daily tasks, monitoring dashboards looked calm, and our microservices ecosystem — consisting of roughly 40 Dockerized services running smoothly — seemed healthy.</p><p>Then, out of nowhere, alerts started flashing red.</p><h3>The Mystery: “Running” Services That Did Absolutely Nothing</h3><p>Reports flooded in that core business operations were failing. When we checked our containers, we faced a puzzling symptom:</p><p>About 10 out of our 40 microservices were completely unresponsive. Yet, running docker ps showed them in a green <strong>“Up”</strong> state. They hadn’t crashed, they hadn’t exited, and they weren’t restarting. They were behaving like “zombies”—technically alive, but completely frozen, unable to process any incoming traffic.</p><p>We immediately jumped into the container logs. Across multiple services, we found:</p><pre>nested exception is com.hazelcast.client.HazelcastClientOfflineException:<br>No connection found to cluster</pre><h3>Suspect #1: Was It Docker Networking?</h3><p>In distributed environments, your first instinct is usually the network layer. Did a Docker bridge network drop? Was internal DNS failing to resolve service hostnames?</p><p>We tested inter-container communication and ran ping and connection checks. The network was rock solid. The network layer was innocent.</p><p>The problem wasn’t getting to the host — the problem was the cluster itself: <strong>Hazelcast</strong>.</p><p>Hazelcast serves as our shared in-memory data grid across these microservices. The frozen services were heavily reliant on Hazelcast for distributed caching. When the cluster stopped responding, threads got blocked waiting for connections, completely starving the application thread pools.</p><h3>30 Minutes of High Pressure</h3><p>During a production outage affecting critical services, every minute feels like an hour.</p><p>We attempted our first recovery action: <strong>restarting the Hazelcast container</strong>.</p><p>After the restart, a few microservices managed to reconnect and recovered, but several key services remained stuck offline. The cluster still felt unstable.</p><p>We quickly performed a clean, synchronized second restart. This time, client connections re-established properly, the Spring beans initialized without throwing HazelcastClientOfflineExceptionand the services breathed again.</p><p>After 30 intense minutes, production was back online. But as any engineer knows: <strong>restarting a server only buys you time — it doesn’t fix the bug.</strong></p><h3>The Post-Mortem: A 1-Year-Old Hidden Bomb</h3><p>Once the fire was out, we dug deep into Hazelcast’s historical logs and memory metrics. That’s when we saw the real horror story:</p><ol><li><strong>Allocated JVM Heap:</strong> Exactly <strong>5 GB</strong>.</li><li><strong>Current Memory Usage:</strong> <strong>100% full (OutOfMemory state)</strong>.</li><li><strong>Container Uptime:</strong> <strong>Over 1 year of continuous running without a single restart.</strong></li><li><strong>Memory Metrics Graph:</strong> A steady, unrelenting <strong>upward</strong> slope spanning 12 months.</li></ol><p>Hazelcast had not freed a single byte of memory in an entire year! It had dutifully accepted every single cache entry since day one, never evicting anything, until the final remaining megabytes were consumed at 10:00 AM yesterday. Once the heap choked completely, the JVM froze, GC paused threads indefinitely, and the Hazelcast node stopped responding to client heartbeats.</p><h3>The Mystery: “Wait… Didn’t We Configure a 2-Minute Eviction?”</h3><p>This discovery surprised us. When we checked our configuration files, we clearly had an eviction policy defined:</p><blockquote>Keep cache entries for 2 minutes, evict expired keys, and retain only recent data.</blockquote><p>Why did Hazelcast ignore this policy for a whole year?</p><h4>Technical Root Causes: Why Text Configs Silently Fail in Docker</h4><p>Digging into Hazelcast’s internals and containerized deployments revealed common pitfalls that cause eviction configs to be silently skipped:</p><ol><li><strong>Silent Fallback on Config File Loading:</strong> In Dockerized Hazelcast deployments, if the configuration file path is not explicitly set via environment variables (like HZ_CONFIG) or system properties (-Dhazelcast.config=/path/to/hazelcast.xml), the instance silently falls back to its bundled default configuration inside the JAR. It starts up cleanly without throwing an error, but your custom eviction rules are never loaded!</li><li><strong>Map Name Mismatch &amp; Wildcard Rules:</strong> In Spring Boot / Hazelcast clients, if addMapConfig (as seen in our stack trace at HazelcastConfig.java:54) or the XML configuration targets a map name that doesn’t match the runtime map name exactly (or misses the wildcard *), Hazelcast defaults to its global default map configuration—which has a Time-To-Live (TTL) of <strong>0 (infinite)</strong>.</li><li><strong>Missing Eviction Policy or Max Size Threshold:</strong> Specifying TTL without properly defining the eviction percentage or policy (LRU - Least Recently Used / LFU) can prevent the background eviction sweeper thread from aggressively purging old entries under sudden memory spikes.</li></ol><h3>The Immediate Live Fix: Taking Control via Management Center</h3><p>Rather than modifying config files and risking another restart in an active production environment, we connected directly to the <strong>Hazelcast Management Center (UI)</strong>.</p><p>Through the web interface, we dynamically updated the live map configurations:</p><ul><li>Configured a strict <strong>TTL of 2 minutes</strong>.</li><li>Applied <strong>LRU (Least Recently Used)</strong> eviction policy.</li><li>Restricted maximum memory per entry.</li></ul><p>Within seconds, the background cleaner kicked in, accumulated stale data was cleared from RAM, and memory consumption plummeted back down to healthy, safe levels.</p><h3>Key Takeaways for Production Engineering</h3><ol><li><strong>A </strong><strong>Running Container is Not Necessarily a Healthy Service:</strong>docker ps Only tells you the OS process is alive. If application threads are starved waiting on distributed caches or databases, your service is effectively dead. Always configure meaningful Health &amp; Readiness probes./actuator/health).</li><li><strong>Beware the “Endless Linear Climb”: </strong>If a cache memory graph only goes up over months and never shows a “sawtooth” pattern (periodic drops from GC and eviction), you don’t have a stable cache — you have a slow memory leak waiting to burst.</li><li><strong>Always Verify Config Loading in Bootstrap Logs: </strong>Never assume your mounted config file was applied. Look for the exact line in Hazelcast startup logs:Config location: /opt/hazelcast/hazelcast.xml</li><li><strong>In-Memory Grids Need Strict Eviction Defaults: </strong>Never leave map configurations to default values in production. Always set an explicit TTL and a fallback Max-Size-Policy so memory can never exceed its safety threshold.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=832244ed1700" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/the-hidden-bomb-in-memory-anatomy-of-a-hazelcast-production-outage-832244ed1700">The Hidden Bomb in Memory: Anatomy of a Hazelcast Production Outage</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taming the Chaos: Why Your Microservices Need an Outbound API Gateway]]></title>
            <link>https://medium.com/cloudnativepub/taming-the-chaos-why-your-microservices-need-an-outbound-api-gateway-58a803841aed?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/58a803841aed</guid>
            <category><![CDATA[api-integration]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[api-gateway]]></category>
            <dc:creator><![CDATA[Alireza Alimohammadi]]></dc:creator>
            <pubDate>Fri, 28 Aug 2026 06:42:09 GMT</pubDate>
            <atom:updated>2026-08-28T06:42:08.246Z</atom:updated>
            <content:encoded><![CDATA[<p>In modern microservices architecture, we spend a lot of time talking about <strong>Inbound API Gateways</strong> — the front doors that handle traffic coming <em>from</em> the outside world <em>into</em> our systems. But what happens when our internal microservices need to talk to the outside world?</p><p>In highly regulated, large-scale enterprise environments — like stock markets, insurance companies, or core banking ecosystems — this is a massive challenge. Imagine an ecosystem of 20+ microservices. Almost all of them need to call external systems: the legacy Core Engine (like a Stock Matching Engine or Core Banking ledger), third-party KYC (Know Your Customer) providers, or SMS gateways.</p><p>If every microservice manages its own external connections, you end up with spaghetti architecture, decentralized security risks, and zero unified observability. The solution? <strong>The Outbound API Gateway (Egress Gateway).</strong></p><p>In this article, we’ll explore the High-Level Design (HLD) of an Outbound API Gateway, how to migrate to it in a multi-phase approach, and which technology stack fits best — especially for ecosystems heavily relying on Java/Spring and .NET.</p><h3>1. Why Do You Need an Outbound API Gateway?</h3><p>Before diving into the design, let’s clarify why letting microservices call external APIs directly is a bad idea in an enterprise environment:</p><ul><li><strong>Credential Sprawl:</strong> If 15 services need to call an external Legacy Core, you have to inject those credentials into 15 different places.</li><li><strong>DDoS-ing Your Own Core:</strong> Without centralized rate limiting, a sudden spike in your microservices could overwhelm a fragile legacy system, bringing operations to a halt.</li><li><strong>Blind Spots:</strong> Debugging a failing external API is a nightmare when logs and metrics are scattered across dozens of different services.</li><li><strong>Redundant Code:</strong> Every team has to write their own Circuit Breakers, retry logic, and error-handling mechanisms.</li></ul><p>An Outbound API Gateway acts as a <strong>centralized reverse proxy for outgoing traffic</strong>, solving all these problems at the infrastructure level.</p><h3>2. High-Level Design (HLD) &amp; C4 Architecture</h3><p>Let’s visualize the architecture. In a standard setup, the Outbound Gateway sits at the edge of your internal network, facing outward.</p><p><strong>AS-IS:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*Mdm2VE-HUyOi-u9LLXlGAw.png" /></figure><p><strong>TO-BE:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*DMXSGjkSTJ4oRld-aseoSQ.png" /></figure><h3>The Architecture Flow</h3><p><strong>The Caller:</strong> Your internal microservice (e.g., Order Service or Policy Service) needs external data.</p><p><strong>The Request:</strong> Instead of calling https://external-core.system.local/api/v1/data, the microservice calls the Outbound Gateway: <a href="http://outbound-gateway/core/v1/data.">http://outbound-gateway/core/v1/data.</a></p><p><strong>The Gateway:</strong></p><ul><li><strong>Intercepts</strong> the request.</li><li><strong>Injects</strong> the necessary authentication headers (e.g., an API Key fetched from a secure Vault).</li><li><strong>Enforces</strong> rate limits and circuit breakers.</li><li><strong>Forwards</strong> the request to the actual external system via a secure connection (often utilizing Mutual TLS or mTLS).</li></ul><p><strong>The Response:</strong> The response flows back through the gateway, where metrics (latency, error rates) are recorded before returning to the microservice.</p><h3>Key Components to Include in the Gateway:</h3><ul><li><strong>Security Module:</strong> Token management, header enrichment, and mTLS termination.</li><li><strong>Resilience Module:</strong> Circuit Breakers (e.g., Resilience4j) and Retry logic.</li><li><strong>Observability Module:</strong> Distributed tracing (OpenTelemetry) and centralized logging.</li><li><strong>Traffic Management:</strong> Rate limiting and bulkheading.</li></ul><h3>Option A: Spring Cloud Gateway (The Developer-Friendly Choice)</h3><p>If your technology is based on Java, Spring Cloud Gateway (SCG) is a highly attractive option.</p><ul><li><strong>Pros:</strong> It’s built on Spring Boot and Project Reactor (Non-blocking). Your teams already know how to write custom filters in Java. It integrates seamlessly with Resilience4j and Micrometer.</li><li><strong>Cons:</strong> Runs on the JVM, meaning a higher memory footprint compared to native proxies written in C++ or Go.</li><li><strong>Best for:</strong> Teams that want full programmatic control over their gateway using the language they already know.</li></ul><h3>Option B: Kong (The High-Performance API Gateway)</h3><p>Kong is built on top of NGINX and uses Lua (or Go/Javascript for custom plugins).</p><ul><li><strong>Pros:</strong> Incredibly fast, highly scalable, and language-agnostic. It has a massive ecosystem of out-of-the-box plugins. It treats .NET, Quarkus, and Spring Boot the same.</li><li><strong>Cons:</strong> You have to manage a separate infrastructure piece (and usually a PostgreSQL database for state, unless running in DB-less mode).</li><li><strong>Best for:</strong> Teams that want an enterprise-grade infrastructure layer and don’t want developers writing custom Java code for routing.</li></ul><h3>The Verdict</h3><p>If you are managing teams heavily invested in Java, <strong>Spring Cloud Gateway</strong> is the most pragmatic choice for quick adoption. However, if your external API traffic is extraordinarily high (tens of thousands of requests per second), <strong>Kong (in DB-less declarative mode)</strong> is the superior architectural choice.</p><h4><strong>Mitigating the Single Point of Failure (SPOF) Risk</strong></h4><p>While an Outbound API Gateway solves the “spaghetti architecture” problem, it introduces a new architectural risk: it becomes a Single Point of Failure (SPOF). If the gateway goes down, all microservices lose access to external third-party providers. In a high-stakes financial environment, this is unacceptable.</p><p>To ensure the Outbound Gateway remains highly available and resilient, we must implement the following strategies:</p><ol><li><strong>Horizontal Scaling and Multi-AZ Deployment:</strong> Deploy the Gateway as a cluster of multiple instances distributed across different Availability Zones (AZs) or data centers. If one zone experiences an outage, the surviving instances will continue processing egress traffic seamlessly.</li><li><strong>Strict Statelessness:</strong> The Gateway instances must be 100% stateless. They should not store any local session data. Secrets (like mTLS certificates and API keys) are fetched dynamically from a Secret Manager (e.g., HashiCorp Vault) and cached in memory. This allows Kubernetes (or your orchestrator) to kill and spin up new Gateway pods on demand without data loss.</li><li><strong>Internal Load Balancing with Active Health Checks:</strong> Microservices should never route traffic to a specific Gateway instance. Instead, traffic is routed through an Internal Load Balancer (or a Kubernetes Service). The load balancer must rely on deep health checks (e.g., /actuator/health in Spring) to instantly remove unhealthy Gateway nodes from the routing pool.</li><li><strong>Bulkheading (Connection Pool Isolation):</strong> We must protect the Gateway from its own downstream dependencies. If one external API becomes extremely slow, it could exhaust all the Gateway’s HTTP worker threads, causing a system-wide outage. By implementing the <strong>Bulkhead Pattern</strong>, we allocate dedicated connection pools to specific external domains. If “Provider A” fails, it exhausts only its thread pool, leaving the rest of the Gateway fully operational for Providers B and C.</li><li><strong>Horizontal Pod Autoscaling (HPA):</strong> During market opening hours or end-of-month batch processing, egress traffic can spike significantly. The Gateway must be configured with auto-scaling rules based on CPU utilization and request throughput, ensuring it scales out <em>before</em> resource exhaustion occurs.</li></ol><h3>4. The Multi-Phase Migration Plan</h3><p>You cannot migrate 20+ services to a new gateway overnight. Doing so in a live enterprise environment is a recipe for disaster. Here is a battle-tested 4-phase migration plan:</p><h3>Phase 1: Foundation and Infrastructure Setup</h3><ul><li>Deploy the Outbound Gateway in a staging environment.</li><li>Configure the core modules: connect it to your secret manager (e.g., HashiCorp Vault) for external API credentials.</li><li>Set up Distributed Tracing (ensure traceId and spanId are logged properly).</li><li>Implement baseline Circuit Breakers and Rate Limits.</li></ul><h3>Phase 2: The Pilot (Low-Risk Migration)</h3><ul><li>Select 1 or 2 low-risk microservices (e.g., an SMS Notification service or a non-critical read-only data service).</li><li><strong>Start with safe HTTP methods:</strong> In the very first migration step, route only GET requests through the gateway. This validates routing, observability, and latency without the risk of mutating data. Once GET requests prove stable, proceed to migrate POST, PUTand other state-changing methods.</li><li>Update their application properties to point to the Outbound Gateway instead of the external URLs.</li><li>Remove local retry/circuit-breaking logic from these specific microservices.</li><li>Monitor logs and metrics for 1–2 sprints to ensure latency and error rates are stable.</li></ul><h3>Phase 3: Incremental Mass Rollout</h3><ul><li>Group the remaining microservices by domain (e.g., User Management, Order Processing, Financial Ledgers).</li><li>Migrate one domain group at a time.</li><li>Crucial Step: Ensure developers are passing standard tracing headers (X-B3-TraceId or W3C traceparent) in their requests so end-to-end observability is maintained.</li></ul><h3>Phase 4: Enforce and Deprecate (Lockdown)</h3><ul><li>Once all services are migrated, update your network security policies (Kubernetes NetworkPolicies or firewall rules).</li><li><strong>Rule:</strong> Block all direct outbound internet/external traffic from the internal microservices. The <em>only</em> application allowed to talk to the outside world should be the Outbound Gateway.</li><li>This ensures future microservices cannot bypass the architecture.</li></ul><h3>5. Crucial Architectural Notes for Enterprise Systems</h3><p>When designing this in a regulated context, keep these non-negotiable rules in mind:</p><ol><li><strong>Data Masking and PII:</strong> Do not blindly log payloads! Enterprise systems pass PANs (Card Numbers), trade amounts, and personal data. The Gateway must mask sensitive fields in the logs.</li><li><strong>mTLS (Mutual TLS):</strong> High-security external systems (like a Central Bank or Stock Exchange) often require mTLS. This means both the client and server verify each other’s cryptographic certificates. The Gateway should hold your client certificate and handle this handshake automatically, completely hiding the complexity from your microservices.</li><li><strong>Timeouts are King:</strong> Legacy systems can sometimes hang. If your Outbound Gateway doesn’t have strict timeout policies, a hanging external API will exhaust your Gateway’s connection pool, causing a cascading failure across all 20+ of your microservices.</li><li><strong>Granular Circuit Breaking:</strong> Don’t use one giant circuit breaker for an entire external ecosystem. Configure circuit breakers <em>per endpoint</em>. If the KYC-API is down, it shouldn’t trip the circuit for the Order-Matching-API.</li></ol><h3>Conclusion</h3><p>An Outbound API Gateway is not just a routing tool; it is a shield. It protects your external partners from your internal traffic spikes, and it protects your internal microservices from external system failures. By centralizing security, retries, and observability, your development teams can stop writing redundant infrastructure code and focus on what actually matters: delivering business value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58a803841aed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/taming-the-chaos-why-your-microservices-need-an-outbound-api-gateway-58a803841aed">Taming the Chaos: Why Your Microservices Need an Outbound API Gateway</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JSON Schema vs Avro: Designing Kafka Data Boundaries]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/cloudnativepub/json-schema-vs-avro-designing-kafka-data-boundaries-6e551bf21a83?source=rss----4b146f5391bb---4"><img src="https://cdn-images-1.medium.com/max/1536/1*-TCNwHDfPRr9YeJolbeAsw.png" width="1536"></a></p><p class="medium-feed-snippet">How to place JSON Schema and Avro in the right layers of your Kafka architecture.</p><p class="medium-feed-link"><a href="https://medium.com/cloudnativepub/json-schema-vs-avro-designing-kafka-data-boundaries-6e551bf21a83?source=rss----4b146f5391bb---4">Continue reading on CloudCraft »</a></p></div>]]></description>
            <link>https://medium.com/cloudnativepub/json-schema-vs-avro-designing-kafka-data-boundaries-6e551bf21a83?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/6e551bf21a83</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[json-schema]]></category>
            <category><![CDATA[avro-schema]]></category>
            <category><![CDATA[schemaregistry]]></category>
            <dc:creator><![CDATA[Zeinab Dashti]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 17:05:22 GMT</pubDate>
            <atom:updated>2026-04-22T17:05:18.516Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Ubuntu: When an Operating System Starts with Being Human]]></title>
            <link>https://medium.com/cloudnativepub/ubuntu-when-an-operating-system-starts-with-being-human-42a4d92c7c3b?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/42a4d92c7c3b</guid>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[long-term-support]]></category>
            <category><![CDATA[trust]]></category>
            <category><![CDATA[community]]></category>
            <category><![CDATA[operating-systems]]></category>
            <dc:creator><![CDATA[Alireza Alimohammadi]]></dc:creator>
            <pubDate>Fri, 09 Jan 2026 08:04:32 GMT</pubDate>
            <atom:updated>2026-01-09T08:04:13.561Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Names Matter</h3><p>If you have ever installed Ubuntu, checked system logs, or written a Dockerfile, you’ve probably seen names like:</p><p><strong>Trusty, Focal, Jammy, Noble</strong></p><p>At first glance, they might look playful — or even irrelevant.</p><p>But in Ubuntu, names are part of the story, not just decoration.</p><h3>Ubuntu: “I am because we are”</h3><p>It tells us:</p><ul><li>Individual growth is inseparable from collective growth</li><li>Technology, like humans, thrives in community</li><li>Trust and responsibility are more important than short-term gains</li></ul><p>Ubuntu’s philosophy is not a slogan; it is the foundation of its design and community.</p><h3>From Philosophy to Open Source</h3><p>Open Source works because of the same idea:</p><ul><li>Someone writes code</li><li>Others read it</li><li>Someone improves it</li><li>Everyone benefits</li></ul><p>Ubuntu doesn’t just use open source; it <strong>embodies</strong> it.</p><p>That’s why Ubuntu is:</p><ul><li>Community-driven</li><li>Transparent in decision-making</li><li>Built in the open, not behind closed doors</li></ul><p>Ubuntu is not only free as in price, but it is <strong>shared in responsibility</strong>.</p><h3>Why Animals? Why Adjectives?</h3><p>Each Ubuntu release follows a clever rule:</p><p><strong>Adjective</strong> + <strong>Animal</strong> in alphabetical order</p><p>Examples:</p><ul><li>Trusty Tahr</li><li>Focal Fossa</li><li>Jammy Jellyfish</li><li>Noble Numbat</li></ul><p>These names are not random. They:</p><ol><li>Give each release a <strong>personality</strong>:</li></ol><ul><li>Trusty → reliability</li><li>Focal → focus</li><li>Noble → calm confidence</li></ul><p>2. Make versions <strong>memorable</strong>, unlike just numbers.</p><p>3. Quietly communicate what the <strong>user experience should feel like</strong>.</p><h3>LTS: Philosophy in Practice</h3><p>Here’s where philosophy meets engineering.</p><p><strong>LTS (Long Term Support) releases</strong> are Ubuntu’s way of respecting human time and trust.</p><h4>What LTS Means</h4><ul><li>Supported years with updates and fixes</li><li>Stability prioritized over flashy new features</li><li>Designed for people and organizations to <strong>rely on</strong></li></ul><h4>Why LTS Reflects Ubuntu’s Philosophy</h4><ul><li>Technology exists for humans, not the other way around</li><li>Communities, businesses, and users need stability</li><li>Ubuntu chooses long-term trust over short-term excitement</li></ul><p>In short:</p><blockquote><strong>Ubuntu LTS exists because long-term trust matters more than short-term innovation.</strong></blockquote><p>Names like Trusty, Xenial, Bionic, Focal, and Noble reinforce the feeling of stability, trust, and reliability.</p><p>The human-centered philosophy is reflected in the name and extends to the engineering decisions.</p><h3>Ubuntu in the Professional World</h3><p>Ubuntu is often seen as a “desktop Linux.”</p><p>In reality, it quietly powers much of modern infrastructure:</p><ul><li>Cloud workloads</li><li>Containers</li><li>DevOps pipelines</li><li>Production systems</li></ul><p>Why?</p><blockquote><strong>Systems built with human-centered thinking are more predictable, maintainable, and trusted.</strong></blockquote><h3>Conclusion</h3><p>Ubuntu reminds us that:</p><ul><li>Technology starts with people</li><li>Collaboration beats isolation</li><li>Meaning matters, even in engineering</li></ul><blockquote><strong>Ubuntu is where philosophy becomes an operating system.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42a4d92c7c3b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/ubuntu-when-an-operating-system-starts-with-being-human-42a4d92c7c3b">Ubuntu: When an Operating System Starts with Being Human</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Kafka Events to Real-Time Alerts: A Hands-On Guide with ksqlDB]]></title>
            <link>https://medium.com/cloudnativepub/from-kafka-events-to-real-time-alerts-a-hands-on-guide-with-ksqldb-b25ee372a6c9?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/b25ee372a6c9</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[ksqldb]]></category>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Zeinab Dashti]]></dc:creator>
            <pubDate>Tue, 02 Dec 2025 06:04:22 GMT</pubDate>
            <atom:updated>2025-12-02T06:04:20.975Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SONc_v_55H1dhei-G0lgXA.jpeg" /></figure><p>Real-time data is everywhere: logins, payments, sensors, clicks. The value of this data often fades within minutes — or even seconds — if you don’t act on it.</p><p>You could write custom Kafka consumers and build an event-processing microservice, but that involves coding, deployment, and testing before you even get to the business logic.</p><p>This is where streaming SQL shines. With just SQL queries, you can continuously process events as they arrive, spot suspicious patterns, and trigger actions in real time.</p><p>Among the available tools, including Apache Flink SQL, Spark Structured Streaming, RisingWave, and ksqlDB, each has its strengths. Flink is now considered the most advanced option, and Confluent has shifted much of its focus there. But ksqlDB remains one of the easiest ways to get started, especially if you already use Kafka. It’s lightweight, simple to spin up, and ideal for proof-of-concepts or smaller real-time use cases that don’t require writing Java or Python.</p><p>In this article, I’ll show you how to use ksqlDB to detect brute-force login attempts in real time. We’ll build the environment with Docker, stream login events into Kafka, define a detection rule in SQL, and generate alerts that downstream systems can consume.</p><h3>Detect Brute-force Login Attempts in Real Time</h3><h4>Step 1: Environment Setup</h4><p>We’ll use Docker Compose to run ZooKeeper, Kafka broker, Schema Registry, ksqlDB server, and client.</p><pre>services:<br>  zookeeper:<br>    image: confluentinc/cp-zookeeper:7.4.0<br>    ports:<br>      - &quot;2181:2181&quot;<br>    environment:<br>      ZOOKEEPER_CLIENT_PORT: 2181<br><br>  kafka:<br>    image: confluentinc/cp-kafka:7.4.0<br>    ports:<br>      - &quot;9092:9092&quot;     # internal (Docker network)<br>      - &quot;29092:29092&quot;   # external (host -&gt; container)<br>    environment:<br>      KAFKA_BROKER_ID: 1<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br><br>      # Two listeners: one for containers, one for your laptop<br>      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:29092<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT<br>      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT<br><br>      # Single-broker friendly settings<br>      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1<br>      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1<br>      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1<br>      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0<br>    depends_on:<br>      - zookeeper<br><br>  schema-registry:<br>    image: confluentinc/cp-schema-registry:7.4.0<br>    ports:<br>      - &quot;8081:8081&quot;<br>    environment:<br>      SCHEMA_REGISTRY_HOST_NAME: schema-registry<br>      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081<br>      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092<br>    depends_on:<br>      - kafka<br><br>  ksqldb-server:<br>    image: confluentinc/cp-ksqldb-server:7.4.0<br>    ports:<br>      - &quot;8088:8088&quot;<br>    environment:<br>      KSQL_LISTENERS: http://0.0.0.0:8088<br>      KSQL_BOOTSTRAP_SERVERS: kafka:9092<br>      KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081<br>      KSQL_KSQL_SERVICE_ID: &quot;ksql_local_&quot;<br>      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: &quot;true&quot;<br>      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: &quot;true&quot;<br>    depends_on:<br>      - kafka<br>      - schema-registry<br><br>  ksqldb-cli:<br>    image: confluentinc/cp-ksqldb-cli:7.4.0<br>    container_name: ksqldb-cli<br>    depends_on:<br>      - ksqldb-server<br>    entrypoint: [&quot;/bin/sh&quot;,&quot;-c&quot;,&quot;sleep infinity&quot;]  # keep the container running<br>    tty: true<br>    volumes:<br>      - ./ksql:/scripts   # optional: put .sql files here</pre><h4>Step 2: Create Kafka Topic</h4><p>Once the stack is started and services are all up and running, we create a Kafka topic for the login attempts</p><pre>kafka-topics --bootstrap-server localhost:29092 --create --topic login_attempts --partitions 1 --replication-factor 1Step 3: Example Event Schema</pre><p>The events will have the following schema:</p><pre>{<br>  &quot;user_id&quot;: &quot;u123&quot;,<br>  &quot;ip&quot;: &quot;203.0.113.42&quot;,<br>  &quot;success&quot;: false,<br>  &quot;event_time&quot;: &quot;2025-08-20T10:05:30.000Z&quot;,<br>  &quot;ua&quot;: &quot;Mozilla/5.0&quot;<br>}</pre><p>In this schema, <em>event_time</em> is the source-of-truth timestamp. And the data will be partitioned by <em>user_id</em>, ensuring accurate per-user aggregations.</p><h4>Step 3: Define the ksqlDB Streams</h4><p>Open ksqlDB CLI:</p><pre>docker exec -it ksqldb-cli ksql http://ksqldb-server:8088</pre><p>Create the source stream over your existing topic using AVRO:</p><pre>SET &#39;auto.offset.reset&#39;=&#39;earliest&#39;;<br><br>CREATE STREAM login_attempts_raw (<br>  user_id STRING KEY,<br>  ip STRING,<br>  success BOOLEAN,<br>  event_time TIMESTAMP,<br>  ua STRING<br>) WITH (<br>  KAFKA_TOPIC=&#39;login_attempts&#39;,<br>  VALUE_FORMAT=&#39;AVRO&#39;,<br>  PARTITIONS=1,<br>  TIMESTAMP=&#39;event_time&#39;<br>);</pre><p>By setting TIMESTAMP=event_time, KsqlDB uses that field as the Kafka record timestamp for windowing; otherwise, it uses processing time.</p><h4>Step 4: Detection Rule (Windowed Aggregation)</h4><p>We flag a user if they attempt three failed logins within five minutes. To do that, we need to check “any” 5-minute span (more sensitive than fixed 5-minute blocks) and emit “alerts” only when the count crosses 5.</p><pre>CREATE TABLE failed_login_alerts<br>  WITH (KAFKA_TOPIC=&#39;failed_login_alerts&#39;, VALUE_FORMAT=&#39;AVRO&#39;) AS<br>SELECT<br>  user_id,<br>  COUNT(*)      AS fail_count,<br>  WINDOWSTART   AS window_start,<br>  WINDOWEND     AS window_end<br>FROM login_attempts_raw<br>WINDOW HOPPING (SIZE 5 MINUTES, ADVANCE BY 1 MINUTE, GRACE PERIOD 1 MINUTE)<br>WHERE success = false<br>GROUP BY user_id<br>HAVING COUNT(*) = 3<br>EMIT CHANGES;</pre><p>This query creates a table indexed by <em>user_id and window</em> that updates as events arrive.</p><p>The ADVANCED BY 1 MINUTE creates a new hopping window that starts every minute, and GRACE PERIOD lets slightly late events still count.</p><h4>Step 5: Watch Alerts Live</h4><p>Run the following command and produce a burst of failures for a user.</p><pre>SELECT * FROM failed_login_alerts EMIT CHANGES;</pre><p>As soon as the entry records for a user match the query condition, you will see a record of the user.</p><p>Once you have the stream, consumers can read the <em>failed_login_alerts</em> table as “alert events” and respond to them.</p><h3>Final Thoughts</h3><p>Streaming SQL isn’t here to replace full-blown stream processing frameworks, but it fills an important gap: making real-time event processing accessible without heavy coding.</p><p>KsqlDB may not be the most advanced streaming SQL engine, but it’s still one of the simplest ways to turn Kafka events into real-time insights.</p><p>The brute-force detection example here is basic, yet it demonstrates how quickly you can transform raw events into actionable alerts with just SQL. For anyone starting with real-time processing, ksqlDB remains a practical entry point before moving on to heavier frameworks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b25ee372a6c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/from-kafka-events-to-real-time-alerts-a-hands-on-guide-with-ksqldb-b25ee372a6c9">From Kafka Events to Real-Time Alerts: A Hands-On Guide with ksqlDB</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Firewalls Think Fast: How Bloom Filters Help WAFs Outsmart Hackers]]></title>
            <link>https://medium.com/cloudnativepub/when-firewalls-think-fast-how-bloom-filters-help-wafs-outsmart-hackers-4b2ddd375e5e?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/4b2ddd375e5e</guid>
            <category><![CDATA[bloom-filter]]></category>
            <category><![CDATA[filtering]]></category>
            <category><![CDATA[waf]]></category>
            <category><![CDATA[memory-efficiency]]></category>
            <dc:creator><![CDATA[Alireza Alimohammadi]]></dc:creator>
            <pubDate>Sun, 26 Oct 2025 07:55:49 GMT</pubDate>
            <atom:updated>2025-10-26T07:55:45.416Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction — The Speed Dilemma of Modern Firewalls</h3><p>Every second, a web application firewall (WAF) faces a flood of requests — some innocent, others malicious. The challenge? <strong>Check millions of IP addresses</strong> against a database of known attackers without slowing the network to a crawl.</p><p>Traditional lookups in large databases are accurate but too slow. That’s where a clever probabilistic data structure comes in — the <strong>Bloom Filter</strong>.</p><h3>What Is a Bloom Filter?</h3><p>A <strong>Bloom Filter</strong> is a compact memory structure that tells you:</p><blockquote><em>“This item is </em>definitely not<em> in the list — or </em>might be<em> in it.”</em></blockquote><p>It’s fast, space-efficient, and okay with a small margin of error. Perfect for scenarios where <strong>speed and memory are critical</strong>, like a WAF</p><h3>How It Works — A Simple Example with IPs</h3><p>Imagine a row of light bulbs — each can be <strong>ON (1)</strong> or <strong>OFF (0)</strong>.</p><p>When a new IP comes in, the firewall runs it through <strong>several</strong> <strong>hash functions (3 functions here, generate 3 results)</strong>. Each hash produces a position in the bulb row, and the bulbs at those positions get turned ON.</p><h4>Example: Adding Bad IPs</h4><ol><li><strong>IP:</strong> 192.168.1.10 → hash positions: 3, 7, 9</li></ol><pre>[0, 0, 0, 1, 0, 0, 0, 1, 0, 1]</pre><ol><li><strong>IP:</strong> 10.0.0.45 → hash positions: 2, 4, 7</li></ol><pre>[0, 0, 1, 1, 1, 0, 0, 1, 0, 1]</pre><p>Notice that position 7 overlaps — that’s fine, Bloom Filter allows overlaps.</p><h4>Checking New IPs</h4><ul><li><strong>IP:</strong> 203.0.113.5 → hash positions: 1, 5, 9<br>Check bulbs: 1 and 5 are OFF → <strong>definitely clean</strong></li><li><strong>IP:</strong> 198.51.100.22 → hash positions: 2, 4, 7<br>All bulbs ON → <strong>maybe bad </strong>(needs deeper inspection)</li></ul><p>This “maybe” is called a <strong>false positive</strong>, but Bloom Filter <strong>never misses a known bad IP</strong> — no false negatives</p><h3>WAFs in Action — Bloom Filters as the First Line of Defense</h3><p>In live WAF systems, millions of IPs and patterns are tracked. Instead of querying huge databases every time:</p><ol><li>The request is hashed and checked against Bloom Filters for IP, URL, and known attack signatures.</li><li>If any filter says “definitely not present,” the request passes instantly.</li><li>If all filters say “maybe,” the request is analyzed in depth using a database or AI model.</li></ol><p>This pre-filtering saves <strong>massive CPU and memory</strong> while keeping real-time protection</p><h3>Why Not Just Use a HashMap?</h3><p>You might wonder — “Why not just store all bad IPs in a HashMap or HashSet?”</p><ul><li><strong>Accuracy:</strong> HashMaps are exact, no false positives.</li><li><strong>Memory:</strong> Each IP or key requires dozens of bytes.</li><li><strong>Scalability problem:</strong> Tracking 50 million IPs could require <strong>gigabytes of memory</strong>.</li></ul><p>Bloom Filter, in contrast, only uses a <strong>few bits per IP</strong>, sacrificing a tiny chance of false positives for huge savings in speed and memory.</p><blockquote><em>In other words: HashMap is smart, but Bloom Filter is street-smart — perfect for real-time systems under heavy load.</em></blockquote><h3>AI + Bloom Filter — Smarter Security</h3><p>Modern WAFs often integrate <strong>machine learning</strong>:</p><ul><li>Bloom Filter acts as the <strong>first gate</strong>, quickly filtering out “definitely safe” requests.</li><li>Machine learning models analyze only the “maybe bad” requests for sophisticated patterns.</li></ul><p>This combo gives the <strong>best of both worlds</strong>: speed from Bloom Filter, intelligence from AI</p><h3>Beyond Security — Other Real-World Uses</h3><ul><li>Search engines use them to avoid re-crawling known pages.</li><li>Databases like Cassandra and HBase use them to skip unnecessary disk reads.</li><li>Email spam filters rely on them to detect known spammers instantly.</li></ul><h3>Conclusion — The Smart Simplicity of Bloom Filters</h3><p>Bloom Filter is like a <strong>lightning-fast, memory-efficient gatekeeper</strong>:</p><ul><li>If bulbs are OFF → the request is <strong>definitely clean</strong>.</li><li>If bulbs are ON → <strong>maybe suspicious</strong>, check further.</li></ul><p>In cybersecurity, speed, memory efficiency, and reliability are key.<br>Bloom Filters let WAFs handle <strong>millions of requests in real time</strong>, while AI provides <strong>deep intelligence for the tricky cases</strong>.</p><blockquote><em>Tiny, clever, and powerful — that’s Bloom Filter, the unsung hero of modern security systems</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b2ddd375e5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/when-firewalls-think-fast-how-bloom-filters-help-wafs-outsmart-hackers-4b2ddd375e5e">When Firewalls Think Fast: How Bloom Filters Help WAFs Outsmart Hackers</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dockerize Your Integration Tests with Testcontainers]]></title>
            <link>https://medium.com/cloudnativepub/dockerize-your-integration-tests-with-testcontainers-218440425537?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/218440425537</guid>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[testcontainer]]></category>
            <category><![CDATA[dockerization]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Alireza Alimohammadi]]></dc:creator>
            <pubDate>Tue, 02 Sep 2025 11:41:38 GMT</pubDate>
            <atom:updated>2025-09-02T11:41:06.395Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*jcOX1UWtupDXRMYqUC-aNA.jpeg" /></figure><p>As developers, we all know the importance of writing tests. But when it comes to testing our application’s interaction with external services like databases, message queues, or APIs, things can get complicated. We often resort to complex mock frameworks or fragile local setups that are difficult to manage and replicate.</p><p>But what if I told you there’s a better way? A way to run your tests against a real-world, production-like environment, without the headache of manual setup.</p><p>That’s where <strong>Testcontainers</strong> comes in.</p><h3>What is Testcontainers?</h3><p>In simple terms, Testcontainers is a Java library that provides a lightweight, throwaway instance of a Docker container for your tests. Think of it as <strong>“Docker for your tests.”</strong> It allows you to programmatically create and manage containers as part of your test lifecycle. When your tests finish, the container is automatically stopped and removed.</p><p>The power of Testcontainers lies in its ability to spin up any service that has a Docker image. Whether it’s a MongoDB database, a Redis cache, or a Kafka cluster, Testcontainers handles all the heavy lifting for you.</p><h3>The Advantages of Using Testcontainers</h3><h3>1. True Integration Testing</h3><p>Unlike mocks, which can lead to false positives if the mock behavior doesn’t perfectly match the real service, Testcontainers lets you test against the genuine service. This gives you confidence that your code will work in a live environment.</p><h3>2. Isolation and Consistency</h3><p>Each test run gets its own clean, isolated container. This means tests won’t interfere with each other, eliminating common issues caused by shared state or leftover data from previous runs. You can be sure that a test failing is due to a code issue, not an environmental one.</p><h3>3. No More Manual Setup</h3><p>Forget about the time-consuming process of installing and configuring a local database. With Testcontainers, all you need is a running Docker daemon. <strong>Your tests will pull the required image and start the container automatically</strong>. This makes your project setup much simpler for new team members.</p><h3>4. Language Agnostic</h3><p>While this article focuses on Java and Spring Boot, Testcontainers has libraries for a wide range of languages, including Python, Go, and .NET.</p><h3>How to Set Up a Spring Boot Project with Testcontainers</h3><p>Let’s walk through a practical example of using Testcontainers with Spring Boot to test against a MongoDB database.</p><h3>Prerequisites</h3><p>Make sure you have Docker installed and running on your machine. Testcontainers will use the Docker daemon to manage containers.</p><h3>1. Add Dependencies</h3><p>First, add the necessary dependencies to your pom.xml (or build.gradle):</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;org.testcontainers&lt;/groupId&gt;<br>    &lt;artifactId&gt;mongodb&lt;/artifactId&gt;<br>    &lt;scope&gt;test&lt;/scope&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;org.testcontainers&lt;/groupId&gt;<br>    &lt;artifactId&gt;junit-jupiter&lt;/artifactId&gt;<br>    &lt;scope&gt;test&lt;/scope&gt;<br>&lt;/dependency&gt;</pre><h3>2. Create an Abstract Base Class for Reusability</h3><p>To avoid repeating the same setup code in every test class, it’s a best practice to create an abstract base class. This is where you configure your container once and for all.</p><pre>@Testcontainers<br>public abstract class AbstractIntegrationTest {<br><br>    @Container<br>    protected static MongoDBContainer mongoDBContainer = new MongoDBContainer(&quot;mongo:6.0.4&quot;)<br>            .withExposedPorts(27017);<br><br>    @DynamicPropertySource<br>    static void setProperties(DynamicPropertyRegistry registry) {<br>        registry.add(&quot;spring.data.mongodb.uri&quot;, mongoDBContainer::getReplicaSetUrl);<br>    }<br>}</pre><ul><li>@Testcontainers: This annotation is key. It enables Testcontainers integration for the test class.</li><li>@Container: This tells Testcontainers to manage the lifecycle of our MongoDBContainer instance. The container will start before any tests in a class and shut down after they are completed.</li><li>@DynamicPropertySource: This is a powerful Spring Boot feature. It allows us to dynamically set a property in the Spring context. In this case, we&#39;re setting spring.data.mongodb.uri to the replica set URL of our running container, ensuring our application connects to the right place.</li></ul><h3>3. Write Your Test Class</h3><p>Now, your actual test class just needs to extend this base class. This simple inheritance makes all the Testcontainers magic available to you, without any duplicated code.</p><pre>@SpringBootTest<br>@ActiveProfiles(&quot;test&quot;)<br>class MyServiceTest extends AbstractIntegrationTest {<br>    // Your tests here<br>}</pre><p>By extending AbstractIntegrationTest, MyServiceTest automatically gets its own isolated MongoDB container instance and is ready to run your tests.</p><h3>A Better Way to Test</h3><p>Testcontainers provides a robust and reliable way to handle dependencies in your integration tests. It solves the persistent problems of environment setup and test isolation by leveraging the power of Docker. By adopting this approach, you can write more meaningful tests and have greater confidence in the quality of your code. Give it a try on your next project!</p><h3>A Final Note</h3><p>A common issue for developers is encountering a “permission denied” error when running these tests, as Testcontainers needs to be able to execute Docker commands like “<em>docker ps</em>” without “<em>sudo</em>”. To resolve this, you must add your user to the Docker group by running:</p><p><strong>sudo usermod -aG docker $USER</strong> in your terminal. For example, I run: sudo usermod -aG docker alireza</p><p>After executing this command, <strong>remember to log out and log back in for the changes to take effect</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=218440425537" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/dockerize-your-integration-tests-with-testcontainers-218440425537">Dockerize Your Integration Tests with Testcontainers</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ubuntu Package Management Showdown: Who Wins — apt or apt-get? The Battle of the Package Wizards]]></title>
            <link>https://medium.com/cloudnativepub/ubuntu-package-management-showdown-who-wins-apt-or-apt-get-the-battle-of-the-package-wizards-8bf7a9ecb269?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/8bf7a9ecb269</guid>
            <category><![CDATA[package-management]]></category>
            <category><![CDATA[ubuntu]]></category>
            <dc:creator><![CDATA[Alireza Alimohammadi]]></dc:creator>
            <pubDate>Fri, 25 Jul 2025 07:06:17 GMT</pubDate>
            <atom:updated>2025-07-25T07:06:17.782Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ubuntu Package Management Showdown: Who Wins — apt or apt-get? The Battle of the Package Wizards</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*FShgVwcHFVXXMG2LVkevEg.png" /></figure><h3>Introduction: The Package Management Jungle in Ubuntu</h3><p>Imagine you’re in a giant supermarket, but instead of groceries, you’re shopping for software. Welcome to Ubuntu’s world of package management — the magic behind installing, updating, and removing apps on your system.</p><p>In this digital supermarket, packages are the products, and package managers are the super-smart clerks who know exactly where every item is and how to get it into your cart without chaos. On Ubuntu, the main players are apt and the legendary apt-get. Who’s better? Who’s more powerful? Let’s break it down.</p><h3>Ubuntu Repositories: The Treasure Trove</h3><p>Before we dive into the tools, let’s understand the treasure chest — Ubuntu repositories. Think of these as the warehouses that store all the software packages:</p><ul><li><strong>Official repositories: </strong>The main store, managed by Ubuntu, with tested, stable apps.</li><li><strong>Regional mirrors:</strong> Nearby warehouses (like those ir.archive.ubuntu.com) for faster access.</li><li><strong>PPAs and third-party sources:</strong> The special shops selling niche, cutting-edge, or experimental goodies.</li></ul><p>Your system checks these repositories whenever you install or update apps. You can customize which warehouses you trust, but remember — the closer and more reliable, the faster your downloads and updates!</p><h3>apt and apt-get: The Dynamic Duo</h3><p>Now, onto the star players:</p><h3>apt — The Friendly Front-End</h3><ul><li>Introduced in Ubuntu 16.04 to make package management less intimidating.</li><li>It’s like the friendly clerk who speaks your language, offers colorful output, progress bars, and is generally easier to use.</li><li>Perfect for <strong>daily tasks</strong>, like installing (apt install), updating (apt update), or upgrading (apt upgrade) your system.</li></ul><h3>apt-get — The Old Guard</h3><ul><li>The veteran, older sibling with years of experience.</li><li>It’s <strong>more detailed</strong>, a bit more technical, and <strong>ideal for scripting and automation.</strong></li><li>Supports options like dist-upgrade, download-only, or simulate that apt either simplifies or omits.</li></ul><h3>The Real Question: What Can apt-get Do That apt Can&#39;t?</h3><p>Think of apt-get as the secret agent of package management—carrying some hidden gadgets that apt doesn’t reveal.</p><p><strong>What features does </strong><strong>apt-get have that </strong><strong>apt doesn’t?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8F6CRTObv-kVqppFG_4TCw.png" /></figure><p>In short, apt-get is like the expert technician—more precise and better suited for scripting, automation, or fixing broken systems.</p><h3>Special Mentions: PPAs &amp; Third-Party Sources — The Niche Shops</h3><p>Now, let’s talk about those special shops — PPAs (Personal Package Archives) and other third-party sources:</p><ul><li><strong>PPAs (Personal Package Archives)</strong> are like little <strong>boutique stores or specialty stalls</strong> set up by individual developers or teams.</li><li>They offer the <strong>latest versions</strong> of software or niche apps <strong>not available</strong> in the official repositories.</li><li>You add a PPA to your system, and it becomes part of your trusted sources, so you can install and update those niche goodies just like any other package.</li></ul><p><strong>Caution:</strong> Since these are outside the official store, sometimes the quality or stability can vary. Always trust reputable PPAs, and remove them if things go sideways.</p><p>Example:</p><pre>sudo add-apt-repository ppa:graphics-drivers/ppa<br>sudo apt update<br>sudo apt install nvidia-driver-XXX</pre><p>This way, you get the latest Nvidia drivers, even if they aren’t yet in the official Ubuntu repositories.</p><p><strong>Third-party sources</strong> are similar — sites or individuals hosting deb packages or other formats. Use them at your own risk, hero!</p><h3>Bottom Line</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*39ibIbAoFTvpXDLfO8pItw.png" /></figure><p><strong>And don’t forget:</strong></p><ul><li>PPAs and third-party sources are the <strong>special shops</strong> — useful but risky. Choose wisely!</li></ul><h3>Final Word: Master the Package Universe</h3><p>Learning the difference between these two tools is like understanding the difference between a <strong>sports car</strong> and a <strong>race car</strong> — both drive you places, but each has its strengths.</p><p>So, next time you’re browsing the Ubuntu supermarket, remember: <strong>apt</strong> is your friendly shopping buddy, and <strong>apt-get</strong> Is the secret agent ready for high-stakes missions?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bf7a9ecb269" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/ubuntu-package-management-showdown-who-wins-apt-or-apt-get-the-battle-of-the-package-wizards-8bf7a9ecb269">Ubuntu Package Management Showdown: Who Wins — apt or apt-get? The Battle of the Package Wizards</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolving Kafka Integration Strategy: Choosing the Right Tool as Requirements Grow]]></title>
            <link>https://medium.com/cloudnativepub/evolving-kafka-integration-strategy-choosing-the-right-tool-as-requirements-grow-d9f7aaf56d80?source=rss----4b146f5391bb---4</link>
            <guid isPermaLink="false">https://medium.com/p/d9f7aaf56d80</guid>
            <category><![CDATA[spring-kafka]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[kafka-streaming]]></category>
            <category><![CDATA[confluent-schema-registry]]></category>
            <category><![CDATA[apache-camel]]></category>
            <dc:creator><![CDATA[Zeinab Dashti]]></dc:creator>
            <pubDate>Wed, 23 Jul 2025 04:57:30 GMT</pubDate>
            <atom:updated>2025-07-23T04:57:30.897Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e4Q1U3PyNGyy1_Tm2csdWQ.jpeg" /></figure><p>Building Kafka data pipelines is a journey rather than a one-time decision. At the simplest stage, a lightweight consumer may suffice. As requirements expand, pipelines naturally grow more complex. Various frameworks have emerged to meet these evolving needs, each with its own balance of reliability, scalability, and operational simplicity. Choosing the right one depends on your current challenges and future goals.</p><p>In this article, we’ll explore a maturing integration strategy. Starting with a basic use case, we’ll advance through real-time stream processing, high-throughput replication, and finally, full end-to-end orchestration, demonstrating why Spring Kafka, Kafka Streams, Kafka Connect, and Apache Camel each excel at different stages.</p><p><em>This article is part of an </em><a href="https://medium.com/@zdb.dashti"><em>ongoing series</em></a><em> on the Apache Kafka Streams API, examining various aspects of developing applications using this API and Spring Boot.</em></p><h3>Schema-Driven Serialization</h3><p>Throughout the examples below, we assume messages are encoded in Avro and managed via Confluent Schema Registry. This lets each framework handle (de)serialization and schema compatibility without custom glue: Spring Kafka uses the Avro deserializer, Kafka Streams the SpecificAvroSerde, Kafka Connect the AvroConverter, and Camel’s Avro dataformat (or Confluent components).</p><h3>Evolutionary Scenarios: From Simple Ingestion to Complex Integration</h3><p>Imagine an online retail platform that publishes an Order event to the Kafka topic orders-topic whenever a customer checks out, each event includes order ID, customer ID, amount, and timestamp. As the platform evolves, simple persistence gives way to demands for real-time analytics, high-volume data replication, and end-to-end orchestration across multiple systems. To address these needs, we’ll follow a four-stage journey—starting with basic “consume &amp; insert,” progressing into stateful stream processing, then high-throughput pipelines, and finally full multi-system workflows—illustrating why Spring Kafka, Kafka Streams, Kafka Connect, and Apache Camel each become the natural choice.</p><h4>Stage 1: Basic Order Persistence</h4><p>Initially, the only requirement is:</p><blockquote>Consume each message from orders-topic and insert a corresponding record into the orders table of our relational database, ensuring at-least-once delivery without data loss.</blockquote><p>We model incoming events as:</p><pre>public class Order {<br>    private String orderId;<br>    private String customerId;<br>    private Long orderAmount;<br>    private LocalDateTime orderDateTime;<br>}</pre><p><strong>Why Spring Kafka fits here</strong></p><p>This scenario involves a simple consumption and database insert with no complex processing. Spring Kafka makes this “consume-and-insert” pattern trivial by embedding Kafka consumers into the Spring ecosystem. Once you’ve configured the Avro deserializer (per our Schema-Driven Serialization), you can declare a simple listener method to receive each Order event and persist it via JPA:</p><pre>@Autowired<br>private OrderRepository orderRepository;<br><br>@KafkaListener(topics = &quot;orders-topic&quot;, groupId = &quot;order-consumer-group&quot;)<br>public void listen(Order avroOrder) {<br><br>   // Map the incoming Order event to a JPA entity mapped to orders table<br>   var orderEntity = new OrderEntity(<br>       avroOrder.getOrderId(),<br>       avroOrder.getCustomerId(),<br>       avroOrder.getOrderAmount(),<br>       LocalDateTime.ofInstant(avroOrder.getOrderDateTime(), ZoneId.systemDefault()),<br>   );<br><br>   orderRepository.save(orderEntity);<br>}</pre><p>Spring’s listener container handles polling, deserialization, offset commits, and retries with sensible defaults. Out of the box (using the default AckMode.BATCH), it will commit the offsets for an entire batch only after all records in that batch have been processed successfully. If your orderRepository.save() throws an exception anywhere in the batch, none of that batch’s offsets are committed, and every message in the batch will be redelivered, giving you at-least-once delivery without any manual offset-management code.</p><p>If you need finer control over when each record’s offset is committed — for example, to commit every record immediately after it’s saved or to group acknowledgments in custom ways — you can switch to manual acknowledgment (AckMode.MANUAL or MANUAL_IMMEDIATE) and call ack.acknowledge() in your listener at exactly the point you choose. Manual ack let you optimize batching, implement custom error-handling (dead-lettering, alerting), or align offset commits with downstream side effects.</p><p>For scenarios demanding the strongest consistency — where the database insert and the Kafka offset commit must occur in one atomic operation — you can enable Kafka transactions. By defining a KafkaTransactionManager (and, if desired, chaining it with your JDBC PlatformTransactionManager), Spring will open a Kafka transaction around your listener. Upon successful completion, both the database save and the offset commit are committed together; on failure, they are rolled back together, ensuring exactly-once processing semantics within the listener.</p><h4>Stage 2: Real-Time Session Metrics</h4><p>With basic persistence in place, the product team now needs live insights into customer behavior:</p><blockquote>They want to know how much each customer spends during an active shopping session — say, any sequence of orders within a five-minute window — so they can trigger in-flight upsells or detect fraud when someone’s order volume spikes in a short time.</blockquote><p>This requirement extends beyond the stateless consumption we previously had. If we tried to do this with just Spring Kafka and the database, the approach might be to write each order to the DB (which we have), and then have some scheduled job or query that periodically computes the sum of orders per customer in the last 5 minutes. However, doing that in real-time for every event is non-trivial and inefficient: you’d be constantly polling the database or maintaining a caching layer, managing window boundaries (when does a session start and end?), handling late arrivals, and so on. It would add latency (since queries might run every minute, not instant) and put load on the database with frequent reads.</p><p><strong>Why Kafka Streams becomes a stronger candidate</strong></p><p>Kafka Streams excels when you need stateful, event-time processing directly in your application. Instead of handling each message in isolation, it lets you maintain per-key state and build windowed computations.</p><p>Kafka Streams provides a session window abstraction that groups records by key (e.g., customerId) into sessions that close after a defined inactivity gap (for example, five minutes). If another event for the same key arrives before the gap elapses, the session extends; if it comes afterward, a new session begins. Overlapping sessions are merged automatically, and you can configure a grace period to accommodate late-arriving events.</p><p>Under the covers, Kafka Streams keeps your aggregates in state stores and, for fault tolerance, records every update to Kafka changelog topics. If an instance fails, another can replay those changelogs to rebuild its state and resume processing without data loss.</p><p>Below is a simplified Java DSL example that consumes Order events, groups them into five-minute sessions with a one-minute grace period, aggregates the order amounts per session, and emits SessionSummary records to a new topic:</p><pre>@Bean<br>public KStream&lt;String, Order&gt; kStream(StreamsBuilder builder) {<br><br>   // Configure Serdes (assuming Order and SessionSummary are Avro types with Schema Registry)<br>   Map&lt;String,String&gt; serdeConfig = Collections.singletonMap(<br>       &quot;schema.registry.url&quot;, schemaRegistryUrl<br>   );<br>   SpecificAvroSerde&lt;Order&gt; orderSerde = new SpecificAvroSerde&lt;&gt;();<br>   orderSerde.configure(serdeConfig, false);<br>   SpecificAvroSerde&lt;SessionSummary&gt; summarySerde = new SpecificAvroSerde&lt;&gt;();<br>   summarySerde.configure(serdeConfig, false);<br><br>   // 1. Stream from the orders topic<br>   KStream&lt;String, Order&gt; orders = builder.stream(<br>       &quot;orders-topic&quot;,<br>       Consumed.with(Serdes.String(), orderSerde)<br>          // ensure we use order timestamp for event-time<br>          .withTimestampExtractor(new OrderTimestampExtractor())<br>   );<br><br>   // 2. Group orders by customerId for aggregation<br>   KGroupedStream&lt;String, Order&gt; byCustomer = orders.groupBy(<br>       (String key, Order value) -&gt; value.getCustomerId(),<br>       Grouped.&lt;String, Order&gt;with(Serdes.String(), orderSerde)<br>   );<br><br>   // 3. Define a session window of 5 minutes with a 1 minute grace period for late events<br>   SessionWindowedKStream&lt;String, Order&gt; sessions =<br>       byCustomer.windowedBy(<br>          SessionWindows.ofInactivityGapAndGrace(<br>              Duration.ofMinutes(5),<br>              Duration.ofMinutes(1))<br>   );<br><br>   // 4. Aggregate orders in each session to compute total amount per session<br>   KTable&lt;Windowed&lt;String&gt;, Long&gt; totals = sessions.aggregate(<br>       () -&gt; 0L,<br>       (String custId, Order o, Long runningTotal) -&gt;<br>          runningTotal + o.getOrderAmount(),<br>       (String custId, Long aggOne, Long aggTwo) -&gt;<br>          aggOne + aggTwo,<br>       Materialized.with(Serdes.String(), Serdes.Long())<br>   );<br><br>   // 5. Convert the session aggregate KTable to a stream of SessionSummary records<br>   KStream&lt;String, SessionSummary&gt; summaryStream = totals.toStream()<br>       .map((Windowed&lt;String&gt; windowedKey, Long total) -&gt; {<br>          String customerId = windowedKey.key();<br>          Instant start = Instant.ofEpochMilli(windowedKey.window().start());<br>          Instant end   = Instant.ofEpochMilli(windowedKey.window().end());<br><br>          SessionSummary summary = SessionSummary.newBuilder()<br>              .setCustomerId(customerId)<br>              .setSessionStart(start)<br>              .setSessionEnd(end)<br>              .setTotalAmount(total)<br>              .build();<br><br>          return KeyValue.pair(customerId, summary);<br>   });<br><br>   // 6. Output the session summaries to another Kafka topic (could be consumed by an analytics DB sink)<br>   summaryStream.to(<br>       &quot;order-session-aggregates-topic&quot;, <br>       Produced.with(Serdes.String(), summarySerde)<br>   );<br>   <br>   return orders;<br>}</pre><p>By offloading session aggregation to Kafka Streams, you relieve your database of continuous query load and gain low-latency, fault-tolerant processing. Furthermore, Kafka Streams supports exactly-once semantics (when enabled), so even in the face of crashes or retries, each order contributes precisely once to the final aggregate.</p><h4>Stage 3: High-Throughput Bulk Replication</h4><p>Now that real-time session summaries are being produced (on the Kafka topic order-session-aggregates-topic from the previous step), the analytics team wants to use this data for long-term analysis. They ask:</p><blockquote>Can we get a nightly or continuous feed of these session summary records into our analytics store (for example, a PostgreSQL data warehouse, Snowflake, Redshift, or even Elasticsearch)? We want to run heavy queries and build dashboards, but we prefer not to query Kafka directly or disrupt the real-time pipeline.</blockquote><p>One way to do this would be to write another custom consumer application: basically, read from order-session-aggregates-topic and bulk insert into the analytics DB. However, this is exactly the kind of “data pipeline plumbing” that Kafka Connect is designed to handle without custom code.</p><p><strong>Why Kafka Connect is ideal here</strong></p><p>Kafka Connect offers a ready-to-use solution for transferring data from Kafka topics to external systems, including popular databases and warehouses, via connectors. Instead of coding, you can configure a sink connector for your target system. For instance, Confluent offers a JDBC Sink Connector that can write to any database with a JDBC driver. There are also connector plugins for specific data warehouses (e.g., a Snowflake Sink Connector and an Elasticsearch Sink). Connect will handle all the heavy lifting: it will automatically consume messages from the topic, use an internal pool of workers to batch them if needed, apply any schema or format conversions, and write them to the target. It will also keep track of the offsets and ensure exactly-once delivery if possible (for JDBC sinks, exactly-once is usually achieved by executing in a transaction or upsert mode).</p><p>Setting up Kafka Connect can be as simple as writing a JSON config like the following (for a JDBC sink writing to PostgreSQL):</p><pre>{<br>  &quot;name&quot;: &quot;order-session-aggregates-jdbc-sink&quot;,<br>  &quot;config&quot;: {<br>    &quot;connector.class&quot;:&quot;io.confluent.connect.jdbc.JdbcSinkConnector&quot;,<br>    &quot;tasks.max&quot;:&quot;1&quot;,<br>    &quot;topics&quot;:&quot;order-session-aggregates-topic&quot;,<br>    &quot;connection.url&quot;:&quot;jdbc:postgresql://postgres:5432/ordersdb&quot;,<br>    &quot;connection.user&quot;:&quot;postgres&quot;,<br>    &quot;connection.password&quot;:&quot;postgres&quot;,<br>    &quot;insert.mode&quot;:&quot;insert&quot;,<br>    &quot;table.name.format&quot;:&quot;order_session_aggregates&quot;,<br>    &quot;auto.create&quot;:&quot;true&quot;,<br>    &quot;auto.evolve&quot;:&quot;true&quot;,<br>    &quot;key.converter&quot;:&quot;org.apache.kafka.connect.storage.StringConverter&quot;,<br>    &quot;value.converter&quot;:&quot;io.confluent.connect.avro.AvroConverter&quot;,<br>    &quot;value.converter.schema.registry.url&quot;:&quot;http://schema-registry:8081&quot;<br>  }<br>}</pre><p>Once this connector is deployed, Kafka Connect will spawn a task that starts reading from order-session-aggregates-topic. It will use the Avro schema of SessionSummary to create or evolve the target table (with columns for each field of the summary), and then every new message on the topic will result in a row inserted into the table. The connector will commit offsets back to Kafka Connect’s internal offsets topic to record how far it has read.</p><p>By adding Kafka Connect, we free our application from “last-mile” integration code. We don’t need to deploy a new service for this ETL — Connect runs as a service and can handle the throughput (it’s designed to handle high-volume data pipelines). We also gain operational benefits: the connector can be monitored via Kafka Connect REST API or monitoring tools, and we get built-in retry and dead-letter queue capabilities. Connect is also schema-aware, meaning it will catch schema mismatches or evolution issues. Scaling out is as simple as increasing &quot;tasks.max&quot; or adding more Connect worker instances—the framework automatically balances tasks across the cluster.</p><h4>Stage 4: Multi-System Routing and Orchestration</h4><p>A few weeks later, the team’s needs expand even further:</p><blockquote>Whenever a new order event comes in, several different actions need to happen <em>in addition</em> to inserting into the database:</blockquote><blockquote>1. Persist the incoming Order to the orders database.<br>2. Enrich it by calling a Customer Profile REST API.<br>3. Send the enriched order to the Shipping Service REST endpoint.<br>4. Email order confirmation to the customer via SMTP.<br>5. If any step fails, apply retries or route the message to a dead-letter queue for later inspection.</blockquote><p>You could attempt to handle this with Spring Kafka by writing a listener that calls all these services sequentially in code, with try-catch blocks around each, and maybe using Spring’s RestTemplate/WebClient and JavaMail API. But very soon, you’d be building a lot of plumbing: handling different failure cases for each call, ensuring one failure doesn’t block the others incorrectly, maybe needing to spawn parallel calls, etc. Maintaining this logic in imperative Java code can become a nightmare, and it’s exactly where an integration framework shines.</p><p><strong>Why Apache Camel is the best fit here</strong></p><p>Apache Camel enables us to declare the entire flow in one coherent route, leveraging built-in components and patterns for connectivity, transformation, and fault tolerance. Below is a concise Java DSL route that accomplishes these steps, assuming Avro-encoded orders via Schema Registry and standard Spring Boot properties for Kafka and the database:</p><pre>public class OrderIntegrationRoute extends RouteBuilder {<br><br>    @Override<br>    public void configure() throws Exception {<br><br>      // Global error handler: retry up to 3 times with 5s delay, then send to dead-letter topic<br>       onException(Exception.class)<br>          .maximumRedeliveries(3).redeliveryDelay(5000)<br>          .handled(true)<br>          .to(&quot;kafka:order-errors-topic&quot;);<br><br>       // 1) Consume Avro‐encoded Order from Kafka<br>       from(&quot;kafka:{{app.topic.name}}&quot;<br>          + &quot;?brokers={{spring.kafka.bootstrap-servers}}&quot;<br>          + &quot;&amp;groupId={{spring.kafka.consumer.group-id}}&quot;<br>          + &quot;&amp;valueDeserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer&quot;<br>          + &quot;&amp;schemaRegistryUrl={{spring.kafka.properties.schema.registry.url}}&quot;<br>          + &quot;&amp;autoOffsetReset=earliest&quot;)<br>          .routeId(&quot;order-end-to-end&quot;)<br><br>          // 2) Map Avro Order → JPA entity and persist<br>          .process(new Processor() {<br>              public void process(Exchange ex) {<br>              Order avro = ex.getIn().getBody(Order.class);<br>              OrderEntity entity = new OrderEntity(<br>                  avro.getOrderId().toString(),<br>                  avro.getCustomerId().toString(),<br>                  avro.getOrderAmount(),<br>                  LocalDateTime.ofInstant(avro.getOrderDateTime(), ZoneId.systemDefault())<br>              );<br>              ex.getIn().setBody(entity);<br>              }<br>          })<br>          .to(&quot;jpa://space.zeinab.demo.model.OrderEntity?usePersist=true&amp;flushOnSend=true&quot;)<br><br>          // 3) Enrich: call CRM and merge via AggregationStrategy<br>          .enrich()<br>              .simple(&quot;http://crm-service/api/customers/${body.customerId}&quot;)<br>              .aggregationStrategy(new AggregationStrategy() {<br>                 public Exchange aggregate(Exchange original, Exchange resource) {<br>                     OrderEntity order = original.getIn().getBody(OrderEntity.class);<br>                    CustomerProfile profile = resource.getIn().getBody(CustomerProfile.class);<br>                    order.setCustomerName(profile.getName());<br>                    order.setCustomerTier(profile.getTier());<br>                    order.setCustomerEmail(profile.getEmail());<br>                    original.getIn().setBody(order);<br>                    return original;<br>                 }<br>          })<br><br>          // 4) Ship: POST enriched order as JSON<br>          .marshal().json()<br>          .to(&quot;http://shipping-service/api/shipments?httpMethod=POST&quot;)<br><br>          // 5) Email confirmation<br>          .setHeader(&quot;subject&quot;, constant(&quot;Order Confirmation&quot;))<br>          .setHeader(&quot;To&quot;, simple(&quot;${body.customerEmail}&quot;))<br>          .setHeader(&quot;From&quot;, constant(&quot;noreply@yourcompany.com&quot;))<br>          .setBody(simple(<br>              &quot;Hi ${body.customerName},\n\n&quot; +<br>                  &quot;Your order ${body.orderId} for amount ${body.orderAmount} has been processed.\n&quot; +<br>                  &quot;Thank you for shopping with us!&quot;<br>          ))<br>          .to(&quot;smtp://{{mail.smtp.server}}?username={{mail.username}}&amp;password={{mail.password}}&quot;)<br><br>          .log(&quot;Order ${body.orderId} processed end-to-end&quot;);<br>    }<br>}</pre><p>How it works:</p><ul><li>Kafka consumption: Uses Camel’s Kafka component with Avro deserialization from Schema Registry.</li><li>Persistence: A lightweight processor maps the Avro Order to a JPA OrderEntity, then the JPA component persists it.</li><li>Enrichment: The .enrich() EIP issues an HTTP GET to the CRM service; the custom AggregationStrategy merges the returned CustomerProfile into the existing OrderEntity.</li><li>Shipping: The enriched entity is marshaled to JSON and POSTed to the shipping API.</li><li>Email: Camel’s Mail component sends a confirmation, using headers set from the enriched entity.</li><li>Error handling: A global onException retries any exception three times with a 5-second delay, then routes failed exchanges to the Kafka topic order-errors-topic for manual review.</li></ul><p>This single Camel route declaratively orchestrates the entire multi-system workflow — freeing you from boilerplate and complex error-handling code, while ensuring each step is executed reliably.</p><h3>Final Thoughts</h3><p>There’s no one-size-fits-all when it comes to Kafka integration. For simple message persistence, Spring Kafka offers tight Spring integration and familiar transaction management. When you need real-time, stateful computations — like windowed aggregations — Kafka Streams provides a lightweight, fault-tolerant library with exactly-once guarantees. To move large volumes of data between Kafka and external stores without hand-coding, Kafka Connect excels with its configuration-driven, pluggable connectors. And when an event must trigger multi-step workflows — such as calling REST APIs, databases, email, or other systems — Apache Camel’s rich DSL and component ecosystem enables you to orchestrate complex routes declaratively.</p><p>In practice, you might combine these: use Spring Kafka for basic ingestion, Kafka Streams for live analytics, Kafka Connect for data replication, and Camel for orchestration. By matching each tool to its sweet spot, you build pipelines that are reliable, scalable, and maintainable — without having to reinvent the wheel.</p><p>All the source code is available in the <a href="https://github.com/zeinab-dashti/kafka-event-processing-solutions-demo">GitHub repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9f7aaf56d80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cloudnativepub/evolving-kafka-integration-strategy-choosing-the-right-tool-as-requirements-grow-d9f7aaf56d80">Evolving Kafka Integration Strategy: Choosing the Right Tool as Requirements Grow</a> was originally published in <a href="https://medium.com/cloudnativepub">CloudCraft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>