<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vishal Sharma on Medium]]></title>
        <description><![CDATA[Stories by Vishal Sharma on Medium]]></description>
        <link>https://medium.com/@sharma.viz?source=rss-2fb209599ed6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*_5YtpdRRhdNyKJuj</url>
            <title>Stories by Vishal Sharma on Medium</title>
            <link>https://medium.com/@sharma.viz?source=rss-2fb209599ed6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:57:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sharma.viz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Colocating Input Partitions with Kafka Streams When Consuming Multiple Topics: Sub-Topology Matters!]]></title>
            <link>https://medium.com/expedia-group-tech/colocating-input-partitions-with-kafka-streams-when-consuming-multiple-topics-sub-topology-matters-f92da955c905?source=rss-2fb209599ed6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f92da955c905</guid>
            <category><![CDATA[kafka-partition]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <dc:creator><![CDATA[Vishal Sharma]]></dc:creator>
            <pubDate>Tue, 11 Nov 2025 12:01:57 GMT</pubDate>
            <atom:updated>2025-11-11T12:01:57.978Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/subpage/engineering">Expedia Group Technology — Engineering</a></h4><h4>Understanding how sub-topology design influences partition co-location</h4><figure><img alt="A man takes a picture with mountains in the background" src="https://cdn-images-1.medium.com/max/1024/0*20bkJ7XymFm1wmnd" /></figure><h4>Abstract</h4><p>At <a href="https://www.expediagroup.com/">Expedia Group</a>™, we handle large-scale, near real-time data processing. We use Kafka as messaging broker and streaming technologies around Kafka to consume and publish messages. Kafka Streams is one of those framework/library.</p><p>This article outlines the behavior of Kafka Streams when consuming from multiple topics (specifically two) with identical partition counts and similar key strategies. It details our expectations regarding partition assignments, the unexpected distribution behavior we encountered in production, and the architectural changes we implemented to enforce partition colocation — critical for achieving caching efficiency in our use case.</p><h4>Use case</h4><p>We needed to consume events from two Kafka topics, both configured with the <strong>same number of partitions</strong> and emitting <strong>similarly keyed records</strong>. The processing pipeline involved <strong>transforming the keys using an external API</strong>, which introduced latency and external dependency cost.</p><p>To optimize, we introduced a <strong>local in-memory cache (Guava)</strong> to avoid redundant API calls — specifically, ensuring that if Key1 appeared in both topics, it would be transformed only once. Importantly, this was <strong>not a join use case</strong>; we were not combining records across topics, just <strong>sharing a cache across both input streams</strong>. Given the distributed nature of our application—with multiple Kafka Streams instances—we assumed partition alignment would naturally support this caching layer.</p><h4>Expectation</h4><p>We expected Kafka Streams to assign partitions deterministically such that <strong>the same partition index across both topics</strong>(e.g., Partition 0 from Topic 1 and Partition 0 from Topic 2) would be routed to the <strong>same Kafka Streams instance</strong>.</p><p>This partition-level colocation was key to ensuring cache locality: identical keys would consistently arrive at the same processing node, allowing effective cache reuse and minimizing external API calls.</p><figure><img alt="Kafka partition assignment diagram showing how partitions from two topics are consistently assigned to the same Kstream instances, with arrows indicating the partition-to-instance mapping." src="https://cdn-images-1.medium.com/max/1024/1*uk-CF6bXuM_Ju2zzhlm_jg.png" /></figure><h4>Production behavior</h4><p>Contrary to our expectations, we observed in production that <strong>identical keys were being processed by different Kafka Streams instances</strong>. A closer look revealed that <strong>matching partition numbers across the two topics were not colocated</strong> — e.g., Partition 0 from Topic 1 was handled by one instance, while Partition 0 from Topic 2 was assigned to another.</p><p>This broke our caching assumption: because the keys were being processed on separate nodes, each instance maintained its own cache and invoked the external API redundantly.</p><figure><img alt="Kafka actual partition assignment diagram showing misaligned partition distribution where corresponding partitions from two topics are assigned to different Kstream instances, with red arrows highlighting the mismatched assignments." src="https://cdn-images-1.medium.com/max/1024/1*1rV24I5ty5QAy6zatSsLMg.png" /></figure><h4>Root cause analysis</h4><p>This behavior led us to a deeper investigation into <strong>Kafka Streams’ topology construction</strong>. What we found was pivotal: the application had <strong>implicitly created two independent sub-topologies</strong> — each associated with one topic, with its own source node, processing logic, and sink.</p><p>Since there was <strong>no shared operator or state between the two processing paths</strong>, Kafka Streams treated them as <strong>entirely separate execution graphs</strong>. And here’s the critical insight:</p><blockquote><strong><em>Kafka Streams only guarantees partition colocation across topics if those topics are consumed within the same sub-topology and have the same number of partitions.</em></strong></blockquote><p>In our case, the two topics were in isolated sub-topologies, so no such colocation guarantee was enforced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QF38fIF7fPv2nhGzBfpRUQ.png" /></figure><h4>Architectural fix</h4><p>To solve this, we <strong>replaced the local Guava cache with a Kafka Streams state store</strong>, which is a distributed, fault-tolerant storage abstraction. Crucially, we <strong>attached this shared state store to both processing branches</strong>, effectively introducing a common dependency.</p><p>This change forced Kafka Streams to <strong>merge the two sub-topologies into a single unified topology</strong>, as the processors were now linked via shared state.</p><p>Once this was done, Kafka Streams <strong>assigned same-index partitions from both topics to the same task</strong>. As a result, identical keys from both topics were consistently routed to the same instance, and our shared cache (now backed by a state store) worked as intended — ensuring a single API call per unique key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*feW1z9IEMh9qMmT_umT5qg.png" /></figure><h4>Final result</h4><p>After the topology unification, our Kafka Streams application exhibited the desired behavior:</p><ul><li><strong>Partition colocation was achieved</strong> across the two topics.</li><li><strong>Identical keys were always routed to the same instance</strong>, enabling effective cache reuse.</li><li><strong>External API calls were reduced significantly</strong>, improving overall system performance and resilience.</li></ul><h4>Conclusion</h4><p>This experience reinforced an important architectural lesson:<br><strong>In Kafka Streams, topology design directly influences partition assignment behavior.</strong></p><p>Partition colocation is not solely based on partition counts; it also depends on whether the input topics are part of the <strong>same sub-topology</strong>. When building distributed, cache-sensitive stream processing applications, architects and developers must be deliberate in how processing graphs are constructed — especially when cross-topic coordination is needed without an explicit join.</p><p>Using a shared state store as a unifying element is a powerful pattern — not only for sharing data, but also for influencing execution behavior in a distributed Kafka Streams application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f92da955c905" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/colocating-input-partitions-with-kafka-streams-when-consuming-multiple-topics-sub-topology-matters-f92da955c905">Colocating Input Partitions with Kafka Streams When Consuming Multiple Topics: Sub-Topology Matters!</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>