<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tom Kaszuba on Medium]]></title>
        <description><![CDATA[Stories by Tom Kaszuba on Medium]]></description>
        <link>https://medium.com/@tkaszuba?source=rss-37dcdc28ae43------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fmW1faOSFeipQdb46jHcew.png</url>
            <title>Stories by Tom Kaszuba on Medium</title>
            <link>https://medium.com/@tkaszuba?source=rss-37dcdc28ae43------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:03:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tkaszuba/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to purge Kafka Streams DSL state stores]]></title>
            <link>https://tkaszuba.medium.com/how-to-purge-kafka-streams-dsl-state-stores-cc8d8f77cb1b?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc8d8f77cb1b</guid>
            <category><![CDATA[state-management]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Thu, 02 May 2024 10:21:23 GMT</pubDate>
            <atom:updated>2024-05-02T10:21:23.987Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Sometimes I encounter questions on how to remove records from the Kafka Streams DSL created state stores, so that an aggregation can restart aggregating from scratch for example.</p><p>There are two primary ways, with both being fairly simple to do. One is to use Tombstones and the other is the processor API.</p><h3>Problem</h3><p>Say you have the following simple stream that does a simple count across keys.</p><pre>    builder.stream(Incoming, Consumed.with(stringSerde, integerSerde))<br>        .groupByKey()<br>        .aggregate(<br>            () -&gt; 0,<br>            (k, v, agg) -&gt; agg + v,<br>            Materialized.&lt;String, Integer, KeyValueStore&lt;Bytes, byte[]&gt;&gt;as(AggStoreName)<br>                .withCachingDisabled())<br>        .toStream()<br>        .to(Outgoing);<br>  }</pre><p>Since the backup topic is compacted, and hence has infinite retention, this counter will keep counting forever. How can you reset this counter to start from the initializer again from outside of the stream with an event pushed through the incoming topic?</p><h4>Tombstones</h4><p>The easiest way to delete or purge a key in a state store is to use a null value, aka Tombstone. If you aren’t familiar with the semantics of tombstones in Kafka Streams I encourage you to read my previous article on <a href="https://tkaszuba.medium.com/hard-vs-soft-deletes-in-kafka-9bf523528465">hard vs soft deletes</a>.</p><p>But sending a tombstone directly into the incoming topic will produce the following message:</p><pre>[WARN] Skipping record due to null key or value. topic=[input-topic] partition=[0] offset=[4]</pre><p>This is by design, as explained on the <a href="https://users.kafka.apache.narkive.com/vqANVT1E/converting-a-stream-to-a-table-groupby-reduce-vs-stream-to-builder-table">kafka mailing list</a>.</p><p>We therefore need to pass some sort of event into the aggregation that we can act upon and issue the tombstone from withing the aggregation. For the sake of the example I will use a “DeleteEvent” (Integer.MIN_VALUE) but normally you would do this using a field in Avro or Protobuf.</p><p>The stream now becomes:</p><pre>public final class SimpleAggregateDelete {<br><br>  public static final String Incoming = &quot;input-topic&quot;;<br>  public static final String Outgoing = &quot;output-topic&quot;;<br>  public static final String AggStoreName = &quot;agg-store&quot;;<br>  public static final int DeleteEvent = Integer.MIN_VALUE;<br>  private static final Serde&lt;String&gt; stringSerde = Serdes.String();<br>  private static final Serde&lt;Integer&gt; integerSerde = Serdes.Integer();<br><br>  public static void createStream(final StreamsBuilder builder) {<br>    builder.stream(Incoming, Consumed.with(stringSerde, integerSerde))<br>        .groupByKey()<br>        .aggregate(<br>            () -&gt; 0,<br>            (k, v, agg) -&gt; (v == DeleteEvent) ? null : agg + v,<br>            Materialized.&lt;String, Integer, KeyValueStore&lt;Bytes, byte[]&gt;&gt;as(AggStoreName)<br>                .withCachingDisabled())<br>        .toStream()<br>        .filter((k, v) -&gt; v != null)<br>        .to(Outgoing);<br>  }<br>}</pre><p>If we encounter a “DeleteEvent” we will issue a tombstone. Since the tombstone will be passed downstream we also need to filter it out after the aggregation call, in this case after the .toStream() call.</p><p>Testing the stream, shows that the aggregation is properly reset after the “DeleteEvent” is issued.</p><pre>  @Test<br>  public void testDelete() {<br>    String key = &quot;Key1&quot;;<br><br>    inputTopic.pipeInput(key, 1);<br>    inputTopic.pipeInput(key, 1);<br>    inputTopic.pipeInput(key, 1);<br>    inputTopic.pipeInput(key, 1);<br><br>    inputTopic.pipeInput(key, null);<br>    inputTopic.pipeInput(key, SimpleAggregateDelete.DeleteEvent);<br><br>    inputTopic.pipeInput(key, 1);<br>    inputTopic.pipeInput(key, 1);<br>    inputTopic.pipeInput(key, 1);<br>    inputTopic.pipeInput(key, 1);<br><br>    assertEquals(1, outputTopic.readRecord().value());<br>    assertEquals(2, outputTopic.readRecord().value());<br>    assertEquals(3, outputTopic.readRecord().value());<br>    assertEquals(4, outputTopic.readRecord().value());<br><br>    assertEquals(1, outputTopic.readRecord().value());<br>    assertEquals(2, outputTopic.readRecord().value());<br>    assertEquals(3, outputTopic.readRecord().value());<br>    assertEquals(4, outputTopic.readRecord().value());<br>  }</pre><h4>Processor API</h4><p>The second way of deleting data from a DSL created state store is to use the processor API. This way is more flexible but perhaps a bit more messy since the processor has to be placed downstream from the DSL, the DSL is what creates the state store first, we can’t delete something from a missing state store. Unfortunately the DSL doesn’t allow you to create a state store and inject it into the DSL which would circumvent this problem.</p><p>The stream now becomes:</p><pre>  public static void createStream(final StreamsBuilder builder) {<br>    builder.stream(Incoming, Consumed.with(stringSerde, integerSerde))<br>        .groupByKey()<br>        .aggregate(<br>            () -&gt; 0,<br>            (k, v, agg) -&gt; (v == DeleteEvent) ? DeleteEvent : agg + v,<br>            Materialized.&lt;String, Integer, KeyValueStore&lt;Bytes, byte[]&gt;&gt;as(AggStoreName)<br>                .withCachingDisabled())<br>        .toStream()<br>        .processValues(StorePurgeProcessor::new, AggStoreName)<br>        .to(Outgoing);<br>  }</pre><p>As you can see the “DeleteEvent” now has to be passed through the aggregation down into the processor to trigger the delete. If the processor could be placed before the aggregation this would not be required.</p><p>The processor is straightforward and looks like the following:</p><pre>  private static class StorePurgeProcessor<br>      extends ContextualFixedKeyProcessor&lt;String, Integer, Integer&gt; {<br><br>    private TimestampedKeyValueStore&lt;String, Integer&gt; dslStore;<br><br>    @Override<br>    public void init(FixedKeyProcessorContext&lt;String, Integer&gt; context) {<br>      super.init(context);<br>      this.dslStore = context.getStateStore(AggStoreName);<br>    }<br><br>    @Override<br>    public void process(FixedKeyRecord&lt;String, Integer&gt; record) {<br>      if (record.value() == DeleteEvent) dslStore.delete(record.key());<br>      else context().forward(record);<br>    }<br>  }</pre><p>If we receive the “DeleteEvent” we call store.delete which deletes the value. (it issues a tombstone to the changelog topic in the background) If not, it functions like a normal pass through processor.</p><p>Running the same tests produces the same correct result.</p><p>You can find the code for both examples in my <a href="https://github.com/tkaszuba/kafka/tree/main/examples/src/main/java/com/tkaszuba/kafka/streams/examples/aggregate">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc8d8f77cb1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Historized SCD lookups with Versioned Stores and IQv2 in Kafka Streams]]></title>
            <link>https://tkaszuba.medium.com/historized-scd-lookups-with-versioned-stores-in-kafka-streams-0c40f2ac4b67?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c40f2ac4b67</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[master-data-management]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Wed, 21 Feb 2024 09:09:10 GMT</pubDate>
            <atom:updated>2024-03-06T10:00:53.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*JdRLeTE2jpaTCtpy7E0M2w.png" /></figure><h3>Introduction</h3><p>In my previous article I posed a problem on how to <a href="https://tkaszuba.medium.com/historized-scd-lookups-with-range-queries-in-kafka-streams-91375e08180d">enable lookups on SCD data that is historized</a> by using Range Queries. <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores">KIP-889</a>, introduced with Kafka Streams version 3.5, seems to also tackle the historization problem with RocksDB segments. Is it then possible to replicate the same functionality using Versioned Stores instead of using Range Queries? Lets find out.</p><h3>Setup</h3><p>The first difference between a Versioned Stores and normal Key/Value Stores is the lack of iterators. There is currently no support for range queries or even the normal <em>all</em> iterator. It’s also not possible to find out how many elements currently are in the store. I’m sure this functionality will show up in the future but as of version 3.7 this isn’t the case.</p><p>The second surprise is that no in-memory store option is provided. All Versioned Stores are therefore RocksDB stores, which makes sense since it relies heavily on the segmenting feature of RocksDB.</p><p>Either way this isn’t a hindrance for our use case and we can proceed with the investigation.</p><p>Since we won’t be supporting any range queries we don’t need to worry about the sort order, hence a big chunk of the boiler plate code from the previous example is now gone. We also don’t need to mess around with implementing the <em>Comparable </em>interface making work with any sort of keys, such as Avro or Protobuf a breeze. This is a huge plus for the Versioned Store solution.</p><p>To initialize the store we will still use a globalKTable as previously. The setup of the lookup store is done using a custom processor:</p><pre>builder.addGlobalStore(<br>    new VersionedKeyValueStoreBuilder&lt;&gt;(<br>        Stores.persistentVersionedKeyValueStore(<br>            VersionedLookupStoreName, Duration.ofDays(5 * 365)),<br>        stringSerde,<br>        stringSerde,<br>        Time.SYSTEM),<br>    LookupTopic,<br>    Consumed.with(stringSerde, keyValueSerde),<br>    VersionedStoreSetupProcessor::new);</pre><p>I’m using a KeyValue class to represent our SCD data, even though it is not a very good semantical representation, but for this example it is good enough. The Versioned Store needs to be passed the retention, I’ve chosen 5 years. This is a big difference to a normal Key/Value store which has infinite retention by default. The segment size can also be defined, which can affect query performance.</p><p>There is not much magic in the processor, we take the event timestamp and insert it into the Versioned Store along with the value.</p><pre>private static class VersionedStoreSetupProcessor<br>    extends ContextualProcessor&lt;String, KeyValue&lt;String, Long&gt;, Void, Void&gt; {<br><br>  private VersionedKeyValueStore&lt;String, String&gt; lookupStore;<br><br>  @Override<br>  public void init(ProcessorContext&lt;Void, Void&gt; context) {<br>    super.init(context);<br>    this.lookupStore = context.getStateStore(VersionedLookupStoreName);<br>  }<br><br>  @Override<br>  public void process(Record&lt;String, KeyValue&lt;String, Long&gt;&gt; record) {<br>    lookupStore.put(record.key(), record.value().key, record.value().value);<br>  }<br>}</pre><h3>Querying</h3><p>Querying the SCD data is dead simple by calling the get method on the Versioned Store with a provided timestamp. If no timestamp is passed then the latest value is returned.</p><pre>VersionedRecord&lt;V&gt; get(K key, long asOfTimestamp); </pre><p>The processor then becomes:</p><pre>private static class VersionedStoreLookupProcessor<br>    extends ContextualProcessor&lt;String, KeyValue&lt;String, Long&gt;, String, Optional&lt;String&gt;&gt; {<br><br>  private VersionedKeyValueStore&lt;String, String&gt; lookupStore;<br><br>  @Override<br>  public void init(ProcessorContext&lt;String, Optional&lt;String&gt;&gt; context) {<br>    super.init(context);<br>    this.lookupStore = context.getStateStore(VersionedLookupStoreName);<br>  }<br><br>  @Override<br>  public void process(Record&lt;String, KeyValue&lt;String, Long&gt;&gt; record) {<br>    Optional&lt;String&gt; lookup =<br>        Optional.ofNullable(lookupStore.get(record.value().key, record.value().value))<br>            .map(VersionedRecord::value);<br>    context().forward(new Record&lt;&gt;(record.key(), lookup, context().currentStreamTimeMs()));<br>  }<br>}</pre><h3>Testing</h3><p>The topology test driver fully supports Versioned Stores so testing follows the same pattern as in the previous example. Running the previous tests now gives:</p><pre>HistorizedLookupsWithVersionedStoresTest &gt; testLookupOrdering() PASSED<br>HistorizedLookupsWithVersionedStoresTest &gt; testLatestMatch() PASSED<br>HistorizedLookupsWithVersionedStoresTest &gt; testBetweenMatch() PASSED<br>HistorizedLookupsWithVersionedStoresTest &gt; testEarliestMatch() FAILED<br>    org.opentest4j.AssertionFailedError at HistorizedLookupsWithVersionedStoresTest.java:145<br>HistorizedLookupsWithVersionedStoresTest &gt; testExactMatch() PASSED</pre><p>As you can see the “earliest match” test fails. This is expected since the Versioned Store has no sense of order associated to it and hence we can’t <em>round up</em> as in the previous example. As stated previously, the <em>round up</em> feature could be viewed as controversial since how can an event be matched if the SCD never existed. In the case of Versioned Stores this question becomes even more murky since the SCD might have existed but has expired in the mean time. The old case of <a href="https://tkaszuba.medium.com/hard-vs-soft-deletes-in-kafka-9bf523528465">soft vs hard deletes</a> again.</p><p>One important factor to take into consideration when using Versioned Store queries is the performance. The get method is optimized for returning the latest value, this will always be O(1) performance. For the earlier records, the Versioned Store has to be scanned through all segments and hence will be a lot slower. If there are a lot of segments then this will impact performance.</p><h3>Interactive Queries version 2</h3><p>Kafka version 3.7 introduces <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-968%3A+Support+single-key_multi-timestamp+interactive+queries+%28IQv2%29+for+versioned+state+stores">KIP-968</a> which allows versioned stores to be queried using the second iteration of interactive queries dubbed <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-796%3A+Interactive+Query+v2">IQv2</a>.</p><p>Since interactive queries return an iterator it is possible to replicate the “round up” feature we had with the range queries using interactive queries.</p><p>Our before value function now becomes:</p><pre>    private Optional&lt;VersionedRecord&lt;String&gt;&gt; beforeValue(String key) {<br>      QueryResult&lt;VersionedRecordIterator&lt;String&gt;&gt; result =<br>          lookupStore.query(<br>              MultiVersionedKeyQuery.&lt;String, String&gt;withKey(key)<br>                  .fromTime(Instant.ofEpochMilli(0L))<br>                  .withAscendingTimestamps(),<br>              PositionBound.unbounded(),<br>              new QueryConfig(false));<br>      try (VersionedRecordIterator&lt;String&gt; iterator = result.getResult()) {<br>        return iterator.hasNext() ? Optional.ofNullable(iterator.next()) : Optional.empty();<br>      }<br>    }</pre><p>To construct the query request we need to pass the key, the from/to timestamp limit and the sort order of the timestamps. Since we only care about the from we set that to the minimum value and leave the max as default. We also want to order the timestamps by ascending to take the first timestamp as we did with the range queries.</p><p>We extend the process method to include this query:</p><pre>    @Override<br>    public void process(Record&lt;String, KeyValue&lt;String, Long&gt;&gt; record) {<br>      Optional&lt;String&gt; lookup =<br>          Optional.ofNullable(lookupStore.get(record.value().key, record.value().value))<br>              .or(() -&gt; beforeValue(record.value().key))<br>              .map(VersionedRecord::value);<br>      context().forward(new Record&lt;&gt;(record.key(), lookup, context().currentStreamTimeMs()));<br>    }</pre><p>Testing the topology now gives an all green result:</p><pre>HistorizedLookupsWithVersionedStoresTest &gt; testLookupOrdering() PASSED<br>HistorizedLookupsWithVersionedStoresTest &gt; testLatestMatch() PASSED<br>HistorizedLookupsWithVersionedStoresTest &gt; testBetweenMatch() PASSED<br>HistorizedLookupsWithVersionedStoresTest &gt; testEarliestMatch() PASSED<br>HistorizedLookupsWithVersionedStoresTest &gt; testExactMatch() PASSED</pre><h3>Conclusion</h3><p>Using Versioned Stores it is possible to replicate all of the functionality that we had defined in the previous example using Range Queries. The major advantage of using Versioned Stores over Range Queries is the simplicity. The amount of boiler plate is minimal plus Avro and Protobuf can be used as is for keys. The disadvantages are that querying Versioned Stores is slower, which might be an issue with a large set of SCD values or segments. To top off, certain state stores features are not yet implemented such as an in-memory db store or some iterators.</p><p>All in all, Versioned Stores are a great addition to the Kafka Streams framework and I’m eager to see how they evolve.</p><p>Fully working code can be found in my <a href="https://github.com/tkaszuba/kafka/blob/main/examples/src/main/java/com/tkaszuba/kafka/streams/examples/lookup/HistorizedLookupsWithVersionedStores.java">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c40f2ac4b67" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Historized SCD lookups with Range Queries in Kafka Streams]]></title>
            <link>https://tkaszuba.medium.com/historized-scd-lookups-with-range-queries-in-kafka-streams-91375e08180d?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/91375e08180d</guid>
            <category><![CDATA[master-data-management]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Sun, 18 Feb 2024 18:44:21 GMT</pubDate>
            <atom:updated>2024-02-21T09:19:42.050Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>About 5 years ago, while working for a client, I encountered an interesting problem related to lookups on slowly changing dimensions. As the project grew, it became apparent that historical events need to be linked to historical SCD data derived from Master Data. At the time all lookups were done using joins in Kafka Streams which require an exact match to fire, this would be “impossible” to do with historized data that was defined by a date range. I argued that this should still be possible in Kafka Streams with some data messaging but alas the label of “impossible” stuck and a solution using an external database to do these queries was chosen.</p><p>While there is nothing wrong with the external database approach, there are clear benefits to not leaving the Kafka Streams framework, mainly:</p><ul><li>transaction support</li><li>faster query times</li><li>lazy evaluated iterator instead of db cursor</li><li>OOM safety</li><li>less operational risk</li></ul><p>Hence here I am, 5 years later while between contracts, showing how this can be done and highlighting an underutilized and misunderstood feature of Kafka Streams: <strong>Range Queries</strong>.</p><h3>The Problem</h3><p>Slowly Changing Dimensions change over time and are therefore valid within a certain time range. When processing historical events we need to link with the SCD that is valid during the time period of the historical event. We therefore can model the SCD as the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GqeZWIxQxT2tRo0nryzhVg.png" /><figcaption>SCD changing over time</figcaption></figure><p>In the graphic above we have 4 date events for Key1 modeled in blue and 4 date events for Key2 modeled in red. They can overlap but for demonstration purposes it is easier to show them separately.</p><p>We then have the following use cases:</p><h4>Exact Match</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ms-duB0QOVD0Z0g6PqbscA.png" /><figcaption>Exact match between Historic event and SCD date</figcaption></figure><p>This is the simplest scenario where the Historic event date matches the SCD date exactly and a normal join would suffice. Since dates and especially date/times rarely match such a scenario is very rare.</p><h4>Between Match</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KaBgcflya6o0TQAZTEtnBg.png" /><figcaption>Range match between Historic event and SCD date</figcaption></figure><p>The most common scenario is where the Historic event date lands between two SCD events. In such a case we want to “round down” and take the earlier date. The blue DE is then linked to the blue D2 and the red DE is linked to the red D3.</p><h4>Earliest Match</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NtN_th7J-XcwyNfXJjfMUA.png" /><figcaption>Earliest match between Historic event and SCD date</figcaption></figure><p>In this case a historic event has a date that is earlier than the first SCD event. It is up to discussion on what to do in such a case but usually we want to “round up” to the earliest available SCD event. The blue DE is then linked to the blue D1 and the red DE is linked to the red D1, ignoring the blue D4.</p><h4>Latest Match</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_tk9NGpNNH5RJunzfwQ8ww.png" /><figcaption>Latest match between Historic event and SCD date</figcaption></figure><p>The last scenario is where the historic event has a date that is still valid with the current SCD event. In such a case we want to “round down” and match to the latest. The blue DE is then linked to the blue D4, ignoring the red D1, and the red DE is linked to the red D4.</p><h3>Sorting</h3><p>The entire trick to range queries lies in the sort order of the keys in the state stores. All state stores in Kafka Streams behave like a <strong>NavigableMap</strong>. The in-memory store, for example, uses a <strong>TreeMap. </strong>Looking at the java doc we see the following:</p><p><em>A Red-Black tree based NavigableMap implementation. The map is sorted according to the </em><strong><em>natural ordering of its keys</em></strong><em>, or by a </em><strong><em>Comparator</em></strong><em> provided at map creation time, depending on which constructor is used.</em></p><p>If simple primitive keys, such as strings or numbers are used, then we have the keys already sorted in the state stores and don’t need to do more work. For complex objects, such as compound keys, we need to provide a <strong>Comparator</strong>.</p><p>I’m a big fan of not using Avro/Protobuf/Json for internal keys due to schema versioning issues hence I will extend the already existing KeyValue in Streams and make it comparable.</p><pre>public class ComparableKeyValue&lt;K extends Comparable, V extends Comparable&gt; extends KeyValue&lt;K, V&gt;<br>    implements Comparable&lt;KeyValue&lt;K, V&gt;&gt; {<br><br>  public ComparableKeyValue(final K key, final V value) {<br>    super(key, value);<br>  }<br><br>  public ComparableKeyValue(final KeyValue&lt;K, V&gt; keyValue) {<br>    this(keyValue.key, keyValue.value);<br>  }<br><br>  @Override<br>  public int compareTo(KeyValue&lt;K, V&gt; other) {<br>    int keyCompare = key.compareTo(other.key);<br>    return (keyCompare == 0) ? value.compareTo(other.value) : keyCompare;<br>  }<br>}</pre><p>We also need a Serde to tell the state stores how to store this complex object which I have provided in my <a href="https://github.com/tkaszuba/kafka/blob/main/serdes/src/main/java/com/tkaszuba/kafka/serdes/ComparableKeyValueSerde.java">github</a>.</p><p>We can now test to ensure our keys are sorted in the correct order in the state store with a unit test that utilizes the topology test driver.</p><h4>Unit Test</h4><p>Following the scenario outlined in the problem section we will need 8 SCD events for the blue key1 and the red key2. The event dates we will store as longs, replicating Avro logical date types.</p><pre>  private final ComparableKeyValue&lt;String, Long&gt; key1V1 = create(key1, LocalDate.of(2021, 1, 1));<br>  private final ComparableKeyValue&lt;String, Long&gt; key1V2 = create(key1, LocalDate.of(2021, 2, 1));<br>  private final ComparableKeyValue&lt;String, Long&gt; key1V3 = create(key1, LocalDate.of(2021, 3, 1));<br>  private final ComparableKeyValue&lt;String, Long&gt; key1V4 = create(key1, LocalDate.of(2021, 4, 1));<br>  <br>  private final ComparableKeyValue&lt;String, Long&gt; key2V1 = create(key2, LocalDate.of(2022, 1, 1));<br>  private final ComparableKeyValue&lt;String, Long&gt; key2V2 = create(key2, LocalDate.of(2022, 2, 1));<br>  private final ComparableKeyValue&lt;String, Long&gt; key2V3 = create(key2, LocalDate.of(2022, 3, 1));<br>  private final ComparableKeyValue&lt;String, Long&gt; key2V4 = create(key2, LocalDate.of(2022, 4, 1));</pre><p>We will then send these events out of order to highlight that insert order is not used.</p><pre>  private void pipeToLookupOutOfOrder() {<br>    lookupTopic.pipeInput(key1V3, &quot;key1V3&quot;);<br>    lookupTopic.pipeInput(key1V1, &quot;key1V1&quot;);<br>    lookupTopic.pipeInput(key2V2, &quot;key2V2&quot;);<br>    lookupTopic.pipeInput(key2V1, &quot;key2V1&quot;);<br>    lookupTopic.pipeInput(key1V2, &quot;key1V2&quot;);<br>    lookupTopic.pipeInput(key1V4, &quot;key1V4&quot;);<br>    lookupTopic.pipeInput(key2V4, &quot;key2V4&quot;);<br>    lookupTopic.pipeInput(key2V3, &quot;key2V3&quot;);<br>  }</pre><p>Using a normal iterator we can see that in fact all the events are now ordered using the Comparator of the ComparableKeyValue.</p><pre>  @Test<br>  public void testLookupOrdering() {<br>    pipeToLookupOutOfOrder();<br><br>    KeyValueStore&lt;ComparableKeyValue&lt;String, Long&gt;, String&gt; store =<br>        testDriver.getKeyValueStore(HistorizedLookupsWithRangeQueries.VersionedLookupStoreName);<br><br>    assertEquals(8, store.approximateNumEntries());<br><br>    var list = new ArrayList&lt;ComparableKeyValue&lt;String, Long&gt;&gt;();<br>    try (KeyValueIterator&lt;ComparableKeyValue&lt;String, Long&gt;, String&gt; iterator = store.all()) {<br>      while (iterator.hasNext()) {<br>        list.add(iterator.next().key);<br>      }<br>    }<br><br>    assertEquals(key1V1, list.get(0));<br>    assertEquals(key1V2, list.get(1));<br>    assertEquals(key1V3, list.get(2));<br>    assertEquals(key1V4, list.get(3));<br>    assertEquals(key2V1, list.get(4));<br>    assertEquals(key2V2, list.get(5));<br>    assertEquals(key2V3, list.get(6));<br>    assertEquals(key2V4, list.get(7));<br>  }</pre><h3>Range Queries</h3><p>All key value state stores in Kafka Streams support range queries through the processor API.</p><pre>        KeyValueIterator&lt;K, V&gt; range(K from, K to);<br>default KeyValueIterator&lt;K, V&gt; reverseRange(K from, K to)</pre><p>Since the KeyValueIterator is lazy evaluated it allows us to deal with each record incrementally, similar to how a db cursor acts. In our case we only want to return one value, hence only one iterator evaluation will be required making the read very fast at O(1).</p><p>We require two range queries where only one will ever be evaluated.</p><p>The beforeValue range query sets the min date starting from 0 and uses a <strong>reverseRange</strong> to “round down” to a value before the event time. While the afterValue uses the Max Instant second to do a <strong>range</strong> query to “round up” to the next available value.</p><pre>    @Override<br>    public void process(Record&lt;String, ComparableKeyValue&lt;String, Long&gt;&gt; record) {<br>      Optional&lt;String&gt; lookup = beforeValue(record.value()).or(() -&gt; afterValue(record.value()));<br>      context().forward(new Record&lt;&gt;(record.key(), lookup, context().currentStreamTimeMs()));<br>    }<br><br>    private Optional&lt;String&gt; beforeValue(ComparableKeyValue&lt;String, Long&gt; value) {<br>      ComparableKeyValue&lt;String, Long&gt; min = new ComparableKeyValue&lt;&gt;(value.key, 0L);<br>      try (KeyValueIterator&lt;ComparableKeyValue&lt;String, Long&gt;, ValueAndTimestamp&lt;String&gt;&gt; iterator =<br>          lookupStore.reverseRange(min, value)) {<br>        return iterator.hasNext()<br>            ? Optional.ofNullable(iterator.next().value.value())<br>            : Optional.empty();<br>      }<br>    }<br><br>    private Optional&lt;String&gt; afterValue(ComparableKeyValue&lt;String, Long&gt; value) {<br>      ComparableKeyValue&lt;String, Long&gt; max =<br>          new ComparableKeyValue&lt;&gt;(value.key, Instant.MAX.getEpochSecond());<br>      try (KeyValueIterator&lt;ComparableKeyValue&lt;String, Long&gt;, ValueAndTimestamp&lt;String&gt;&gt; iterator =<br>          lookupStore.range(value, max)) {<br>        return iterator.hasNext()<br>            ? Optional.ofNullable(iterator.next().value.value())<br>            : Optional.empty();<br>      }<br>    }</pre><h4>Testing</h4><p>We will now a issue a query command to model our Historic event and see if it is linked to the correct historized SCD event as outlined in the “Problem” section.</p><p><strong>Exact Match</strong></p><pre>  @Test<br>  public void testExactMatch() {<br>    pipeToLookupOutOfOrder();<br><br>    queryCommandTopic.pipeInput(commandId, key1V2);<br>    var res = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res.key());<br>    assertEquals(&quot;key1V2&quot;, res.value());<br><br>    queryCommandTopic.pipeInput(commandId, key2V3);<br>    var res2 = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res2.key());<br>    assertEquals(&quot;key2V3&quot;, res2.value());<br>  }</pre><p><strong>Between Match</strong></p><pre>  @Test<br>  public void testBetweenMatch() {<br>    pipeToLookupOutOfOrder();<br><br>    queryCommandTopic.pipeInput(commandId, create(key1, LocalDate.of(2021, 2, 27)));<br>    var res = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res.key());<br>    assertEquals(&quot;key1V2&quot;, res.value());<br><br>    queryCommandTopic.pipeInput(commandId, create(key2, LocalDate.of(2022, 3, 27)));<br>    var res2 = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res2.key());<br>    assertEquals(&quot;key2V3&quot;, res2.value());<br>  }</pre><p><strong>Earliest Match</strong></p><pre>  @Test<br>  public void testEarliestMatch() {<br>    pipeToLookupOutOfOrder();<br><br>    queryCommandTopic.pipeInput(commandId, create(key1, LocalDate.of(2020, 2, 27)));<br>    var res = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res.key());<br>    assertEquals(&quot;key1V1&quot;, res.value());<br><br>    queryCommandTopic.pipeInput(commandId, create(key2, LocalDate.of(2021, 3, 27)));<br>    var res2 = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res2.key());<br>    assertEquals(&quot;key2V1&quot;, res2.value());<br>  }</pre><p><strong>Latest Match</strong></p><pre>  @Test<br>  public void testLatestMatch() {<br>    pipeToLookupOutOfOrder();<br><br>    queryCommandTopic.pipeInput(commandId, create(key1, LocalDate.of(2022, 2, 27)));<br>    var res = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res.key());<br>    assertEquals(&quot;key1V4&quot;, res.value());<br><br>    queryCommandTopic.pipeInput(commandId, create(key2, LocalDate.of(2023, 3, 27)));<br>    var res2 = queryResponseTopic.readRecord();<br>    assertEquals(commandId, res2.key());<br>    assertEquals(&quot;key2V4&quot;, res2.value());<br>  }</pre><p><strong>Results</strong></p><p>All tests pass with the range queries working as expected.</p><pre>HistorizedLookupsWithRangeQueriesTest &gt; testLookupOrdering() PASSED<br>HistorizedLookupsWithRangeQueriesTest &gt; testLatestMatch() PASSED<br>HistorizedLookupsWithRangeQueriesTest &gt; testBetweenMatch() PASSED<br>HistorizedLookupsWithRangeQueriesTest &gt; testEarliestMatch() PASSED<br>HistorizedLookupsWithRangeQueriesTest &gt; testExactMatch() PASSED<br>HistorizedLookupsWithRangeQueriesTest &gt; testLatestRangeMatch() PASSED</pre><h3>Conclusion</h3><p>As you can see it is not “impossible” to do range lookups on historized data but it does require a bit more boiler plate to massage the keys.</p><p>p.s: I’ve skipped over some of the more boring parts but the fully working example can be found in my <a href="https://github.com/tkaszuba/kafka/blob/main/examples/src/main/java/com/tkaszuba/kafka/streams/examples/lookup/HistorizedLookupsWithRangeQueries.java">github.</a></p><h4><strong>Up Next</strong></h4><p>KIP-889 has <a href="https://www.confluent.io/blog/introducing-versioned-state-store-in-kafka-streams/">introduced Versioned Stores</a> which seem to be a perfect fit for resolving the problem of historized SCD lookups. Lets see if it is possible to implement the same functionality using versioned stores. <a href="https://tkaszuba.medium.com/historized-scd-lookups-with-versioned-stores-in-kafka-streams-0c40f2ac4b67">https://tkaszuba.medium.com/historized-scd-lookups-with-versioned-stores-in-kafka-streams-0c40f2ac4b67</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91375e08180d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Synchronize two lookup topics using foreign key joins in Kafka Streams]]></title>
            <link>https://tkaszuba.medium.com/synchronize-two-lookup-topics-using-foreign-key-joins-in-kafka-streams-c37f23f4ef7e?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/c37f23f4ef7e</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[master-data-management]]></category>
            <category><![CDATA[state-management]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 10:12:55 GMT</pubDate>
            <atom:updated>2024-02-14T21:54:29.948Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1: The theory</h4><h3>Introduction</h3><p>Global kTables in Kafka streams are a great way to enrich your streaming applications with data from other topics. In order to use global kTables effectively, having a properly prepared data feed is important. A snapshot of the latest data using <a href="https://tkaszuba.medium.com/hard-vs-soft-deletes-in-kafka-9bf523528465">hard deletes</a> guarantees that we only keep the latest events. This allows our global kTables to be built quickly and to use in memory stores which are less finicky and faster than RocksDB.</p><p>Lookup topic feeds for global kTables are often derived from other lookup topics, perhaps a master data feed. In such a scenario, we will need to create specialized derived lookup topics specifically keyed/partitioned for particular lookups. Using foreign key joins, it is possible to propagate state changes in the master topic to the derived topics ensuring that they stay in sync with the master.</p><h3>State changes</h3><p>For the sake of the article I will take a very simple example of a master data feed that contains a pair of key/value strings. From the master data feed we would like to derive a mirror of this topic, hence a pair of value/key strings. For simplicity we will assume that we have a 1 to 1 mapping between key and value. If we have 1 to many relationship things get more tricky.</p><p>We will then have the following use cases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*wsU7B6vMSHc2Tb6p0UyFQg.png" /></figure><p>The conclusion here is that both an update and a delete in the master topic need to trigger a delete in the derived topic. We also need to detect duplicates in order to not delete an update that was just applied. Finding the events that need to be deleted will be the role of the foreign key join between the key in the master topic and the value of the derived topic.</p><h3>Topology</h3><h4><strong>Topics</strong></h4><p>We will need two topics, the master and the derived topic. They need to exist up front and be created outside of the topology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*QtaIJaUEhB6vQ7zDza4yGA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*I3ym9Ds3zzt-7zf6XhFjJg.png" /></figure><h4><strong>Tables</strong></h4><p>There are three tables required to handle the delete use cases in the derived topic. The current state of the derived topic, the deletes state of the derived topic and the final update state of the derived topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*IJa0Zb63Ob-tLN1AL7suWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*Qvbv2G__l1aYD8Blmo3VEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*pP-S0ysdQYxUEjrWQnPSeQ.png" /></figure><p>The foreign key join will be done between the current state kTable and the deletes state kTable. It has the following signature.</p><pre>&lt;VR, KO, VO&gt; KTable&lt;K, VR&gt; join(final KTable&lt;KO, VO&gt; other,<br>                                final Function&lt;V, KO&gt; foreignKeyExtractor,<br>                                final ValueJoiner&lt;V, VO, VR&gt; joiner,<br>                                final Materialized&lt;K, VR, KeyValueStore&lt;Bytes, byte[]&gt;&gt; materialized);</pre><p>Since our duplicate detection mechanism will be done in the value joiner, it has no access to the keys, therefore will need to pass also the key in the value. For the purpose of the example I have used a tuple but it could also be an Avro/Protobuf/Json/KeyValue class, whatever, as long as we have access to the key in the value.</p><h4><strong>Final topology</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4-jAWCyJpdIb9FIk1bwBTQ.png" /><figcaption>Stream topology</figcaption></figure><p>There are two main parts to the topology. The straight forward inserts/updates and the special handling of the deletes. For the deletes handling the derived topic feeds the current state, gets updated through the foreign key join, updates the derived topic and then updates the current state. This circular logic will of course cause an infinite loop which has to be accounted for with proper cleanup of the state stores.</p><h3>Conclusion</h3><p>That’s about it for the theory, the solution is a lot more nuanced since several cases need to be accounted for:</p><ul><li>Handling of tombstones</li><li>Event Suppression</li><li>Cleanup of state stores</li><li>Caching and Logging</li><li>Duplicate detection</li></ul><p>I will discuss this in part two.</p><p>For the curious the working code can already be found in my <a href="https://github.com/tkaszuba/kafka/blob/main/examples/src/main/java/com/tkaszuba/kafka/streams/examples/lookup/LookupSyncWithForeignKeys.java">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c37f23f4ef7e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hard vs Soft deletes in Kafka]]></title>
            <link>https://tkaszuba.medium.com/hard-vs-soft-deletes-in-kafka-9bf523528465?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bf523528465</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Tue, 23 Jan 2024 20:35:04 GMT</pubDate>
            <atom:updated>2024-03-10T07:15:34.447Z</atom:updated>
            <content:encoded><![CDATA[<p>There are several ways on how to model a delete event in Kafka with each having their own advantages and disadvantages. While using explicit delete events seem like the no brainer correct approach every time, it is not the preferred way to do this in Kafka and could lead to serious problems down the line. Employing a Kafka expert is a good way to identify potential operational issues before committing to an approach that has previously worked flawlessly in other distributed systems.</p><p>Tombstones in Kafka are events with a null value. If tombstone events are encountered in a compacted topic they tell the Kafka broker that the existing event should be permanently deleted. The tombstone itself is then permanently removed after the <a href="https://docs.confluent.io/platform/current/installation/configuration/topic-configs.html#delete-retention-ms">delete.retention.ms</a> has passed. Since the tombstone is permanently deleted it is therefore impossible to tell if the record was deleted or if it ever existed. Explicit delete events on the other hand do not use the built in Kafka deletion mechanism but just mark the event as deleted using an event such as the following and expect the consumer to deal with this event explicitly.</p><pre>{<br>  ...<br>  &quot;isDeleted&quot;: true<br>}</pre><p>Delete events are therefore a “soft” delete, while using tombstones is a “hard” delete. There is of course nothing wrong with the “soft” approach and in fact many times it is something that is flat out required in a full audit scenario or dealing with delete events explicitly but it does introduce some operational problems.</p><p>The main one being, old irrelevant data is never cleaned up, it just hangs around for years. All those old deleted products, users, transactions, schemas are there for the lifetime of the topic since a compacted topic by default has infinite retention, unless a “compact,delete” policy is specifically set. During a replay scenario, the consumers will need to deal with perhaps 5 year old events that people have long forgotten about. Even in a simple schema migration scenario, these old events will need to be accounted for. For topics with domain objects that have very short lifespans it causes the amount of data to explode. I’ve seen topics go from 10 mil to 25k with proper deletion strategies. All that irrelevant data also needs to be replicated to other broker nodes, which impacts the broker performance. Leaving the broker aspects aside having less relevant data over a ton of irrelevant data also has its advantages in Kafka streams clients. State stores (kTables) from topics with less data are rebuilt much faster and hence make services stand up quicker.* Having smaller relevant datasets for lookups is also more advantageous since a more performant in memory store could be used over RocksDB. Tombstones/null values are <a href="https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics">handled more efficiently</a> during a left/outer join since you do not take the serde hit. Therefore doing a join that gets filtered out anyways will incur performance penalties.</p><p>In conclusion, soft deletes with explicit delete events are great but they should not be the default if the scenario doesn’t specifically require them. Using tombstones helps topics stay lean and should be first evaluated before using explicit delete events.</p><p>*The Confluent Schema Registry is a good example of this as it uses soft deletes as a default. If the same schema registry is used to bring up and down multiple environments, for ex. testing, all those old schemas are hanging around in the backup topic even though they are not visible through the REST endpoints. If the amount of schemas gets big enough the startup times of the schema registry will suffer. Waiting 15 minutes for the schema registry to restart is not a nice calling card for a highly available environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bf523528465" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The curious case of missing headers in Kafka Streams]]></title>
            <link>https://tkaszuba.medium.com/the-curious-case-of-missing-headers-in-kafka-streams-c0099d077938?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0099d077938</guid>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Sun, 25 Dec 2022 16:54:46 GMT</pubDate>
            <atom:updated>2022-12-25T16:54:46.737Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dzqBe1wQEyMBAQH_B6YgHA.png" /></figure><p><a href="https://cwiki.apache.org/confluence/display/kafka/kip-82+-+add+record+headers">KIP-82</a> Introduced headers to Kafka and they’ve been a hit ever since. Headers allow you to pass metadata information such as logging context, error information or data lineage of a message without having to dirty your payload. At least in theory. Once you try to use Kafka headers in Kafka Streams you quickly find out that there is no support for them when doing any sort of stateful transformations even though the extension points in the API are there.</p><h3>Why do we need headers in Kafka Streams?</h3><p>The examples pointed out earlier of course hold also with Kafka Streams but there is one extra use case that most people do not think about where headers would greatly aid in performance: <strong>batching</strong>.</p><p>For me batching is when you suppress a real time stream until some sort of condition is met, it could be for example a buffer before an insert into an external system or perhaps a transpose of a certain number of messages into one message. I’ve used suppression to make DSL joins work the same as an SQL join for example. In advanced Kafka streams applications some sort of batching/suppression is routinely used to control data that is pushed out to downstream services. So how can headers help in this regard?</p><p>These are the good old signatures of the Kafka Serializer and Deserializer:</p><pre>public interface Serializer&lt;T&gt; extends Closeable {<br><br>    byte[] serialize(String topic, T data);<br><br>    default byte[] serialize(String topic, Headers headers, T data) {<br>        return serialize(topic, data);<br>    }<br>}<br><br>public interface Deserializer&lt;T&gt; extends Closeable {<br><br>    T deserialize(String topic, byte[] data);<br><br>    default T deserialize(String topic, Headers headers, byte[] data) {<br>        return deserialize(topic, data);<br>    }<br>}</pre><p>As you will notice it contains two separate serialize/deserialize methods, one with headers and one without headers. Observing the default with headers implementations we see that it drops the headers which points out that you will need to override these methods if you want to pass headers further along. What is a more interesting fact though is that the serialization and the deserialization of headers are treated “separately” from the payload. This fact is super important since it lets us treat headers as first class citizens. We can now decide what to do with the payload by just looking at the metadata without having to deserialize the entire payload! We could then include indexing information related to the payload without having to store it in a separate state store.</p><h3>Batching use case</h3><p>I’ll take the case of transposing a certain number of messages into one message as this was a case I have had with a former client. We have 600 individual Avro events that we would like to send downstream as one Avro event with an array of 600 items.</p><p>With the DSL generally we would need to do the following to get this done:</p><ol><li>Use the <a href="https://kafka.apache.org/33/javadoc/org/apache/kafka/streams/kstream/KGroupedTable.html#aggregate-org.apache.kafka.streams.kstream.Initializer-org.apache.kafka.streams.kstream.Aggregator-org.apache.kafka.streams.kstream.Aggregator-">aggregate method</a> to transform the 600 Avro events into 1 Avro event.</li><li>Call the <a href="https://kafka.apache.org/33/javadoc/org/apache/kafka/streams/kstream/KTable.html#toStream--">toStream method</a> to output the changelog events. If you have transactions on this call will result in 600 update events since caching is turned off.</li><li>Use the <a href="https://kafka.apache.org/33/javadoc/org/apache/kafka/streams/kstream/KStream.html#filter-org.apache.kafka.streams.kstream.Predicate-">filter method</a> as our suppression mechanism to drop the changelog events that don’t satisfy a business condition.</li></ol><p>This will get the job done but it will be <strong>VERY</strong> slow.</p><p>The reason why this is slow is because that aggregation method has to call the Avro deserializer 180.300 + 600 times.</p><p>But we have only 600 items how can that result in 181k calls? Any time you have an embedded Avro, which an array item is, the Avro deserializer does a recursive call and treats the embedded Avro like a separate deserialize call. Therefore as the Avro object grows we get the following sequence plus 600 for the parent Avro.</p><p>1 + 2 + 3 + … + n:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/1*WtxHILuqWi_ToN05_fLOVw.png" /></figure><p>This is a lot of overhead for ultimately ONE transposed event.</p><p>So is there a better way to do this? <strong>YES</strong> and headers can really help us out here.</p><h3>Processor API to the rescue</h3><p>Since the DSL is not up to the task we need to go to the bare bones and create ourselves a transformer that will do what the DSL does for us. The general solution is this. We serialize each Avro event into a state store as a List and use the headers as an index. When a new event pops in we add the serialized event to the serialized collection and update the index. When we are ready to forward the event downstream we will only deserialize then which will result in precisely 601 deserialization calls.</p><h4>List Serde</h4><p>Instead of using the embedded Array of an Avro object we need to create our own Serde that allows us to serialize a List:</p><pre>public class ListSerde&lt;T&gt; implements Serde&lt;List&lt;T&gt;&gt;</pre><p>I won’t go into the details of how to create such a Serde but suffice it to say that the serialization saves an object in Kafka as follows:</p><p>| number of items in the collection | size of first serialized event | first serialized event | size of second serialized event | second serialized event| … | size of n serialized event | n serialized event |</p><p>Therefore if we want to add a new object to the serialized collection all we need to do is to update the number of items in the collection at the head of the collection (this is a fixed number of bytes) and attach the serialized event to the tail of the serialized collection. In this way we don’t need to deserialize anything and work directly on the level of bytes.</p><h4>The Index</h4><p>We don’t specifically need to use Headers to create the index, we could just as well use a second state store for keeping the metadata, BUT since the Kafka API has given us all the extension points why not take advantage of this mechanism.</p><p>In order to serialize and deserialize the headers we need the following two new Serdes.</p><pre>public class HeadersSerde implements Serde&lt;List&lt;Tupple&lt;String, Byte[]&gt;&gt;&gt;<br>public class HeaderSerde implements Serde&lt;Tupple&lt;String, Byte[]&gt;</pre><p>It’s a shame that Kafka doesn’t provide a Serde for the Headers, which I hope they rectify in a future release, but for the time being you can grow your own by leveraging the List Serde created previously and store the headers as a tupple of a string and a byte[].</p><pre>public interface Headers extends Iterable&lt;Header&gt;<br>public interface Header {   <br>    String key();<br>    byte[] value();   <br>}</pre><p>We now need one final Serde to link the Headers with the Value.</p><pre>public class HeadersWithValueSerde&lt;T&gt; implements Serde&lt;T&gt; {<br> private final HeadersSerde headers;<br> protecte final Serde&lt;T&gt; values;<br> public HeadersWithValueSerde(Serde&lt;T&gt; serde) {<br>  values = values;<br> }<br>}</pre><p>In this Serde we will internally use the HeadersSerde to first Serialize the headers, put them at the beginning and then put the serialized value at the tail. For our use case we don’t want our values to be serialized so we will use the Byte Array Serde which does nothing:</p><pre>public class IndexHeadersSerde extends HeadersWithValueSerde&lt;Byte[]&gt;<br> public IndexHeadersSerde() {<br>  super(Serdes.ByteArraySerde)<br> }<br>}</pre><p>For the sake of brevity I’m skipping the steps on serializing the events and adding them to the List that I described earlier which are the input to the IndexHeadersSerde.</p><p>Using the IndexHeadersSerde will now allow us to get the Headers apart from the values. As an index we need to put in any information that will allow us to work with the serialized value collection and trigger the stop of the suppression (deserialization) event:</p><ul><li>num of items in the value collection</li><li>a business rule if applicable</li></ul><p>In my particular use case it was the sum of probabilities, once they summed up to 1 we can consider the aggregation finished and can forward the event.</p><h3>Conclusion</h3><p>Of course this index approach will not work if the particular suppression trigger is not easily modeled. In such a case it is beneficial to break the event model into smaller parts so that they can easily be parallelized and then brought together after.</p><p>There’s a lot of use cases for headers, using them as an index in the Processor API is just one of them. Hopefully in the future the Kafka Streams API will be extended to fill in the missing gaps so that they can be leveraged to their fullest.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0099d077938" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to sink a Kafka roll out]]></title>
            <link>https://tkaszuba.medium.com/how-to-sink-a-kafka-roll-out-6dc127c555c4?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dc127c555c4</guid>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Mon, 07 Nov 2022 21:57:31 GMT</pubDate>
            <atom:updated>2023-02-08T06:08:42.631Z</atom:updated>
            <content:encoded><![CDATA[<p>Has management started to flirt with the idea of doing event sourcing and rolling out a Kafka cluster? Have you read up on the technology and you want nothing to do with it? What is the best way to nip this rollout in the bud? Is arguing whether <a href="https://kai-waehner.medium.com/is-apache-kafka-a-database-ddc310898f5c">Kafka is a database</a> or <a href="https://medium.com/towards-data-science/you-can-replace-kafka-with-a-database-39e13b610b63">not the answer</a>? Maybe arguing that long term storage of data in Kafka is a ticket to hell (it’s <a href="https://www.confluent.io/blog/okay-store-data-apache-kafka/">not</a>)? Perhaps arguing that there are <a href="https://www.jesse-anderson.com/why-i-recommend-my-clients-not-use-ksql-and-kafka-streams/">no snapshots for replays in Kafka Streams the answer</a>? (there are) Maybe arguing that you can do e<a href="https://medium.com/dna-technology/why-we-dropped-event-sourcing-with-kafka-streams-when-given-a-second-chance-b904a80bc4be">vent streaming with postgres is the answer</a>? Sure, all of these <strong>could</strong> be good enough reasons but they can be argued either way if you encounter a heavyweight opponent. But why risk it? There is an easier way to sink a Kafka roll out. It’s called <strong>GOVERNANCE. </strong>Or should I say, a complete lack of Governance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dzqBe1wQEyMBAQH_B6YgHA.png" /></figure><p>So what exactly do I mean by Governance? I think the best way is to look at use cases that will pop up when you have a reasonable size project rolled out into an enterprise production environment.</p><h4>What is the best way to roll out changes to Kafka in production?</h4><p>How do you reset a consumer group or an application id? How do you create a new topic? Perhaps you would like to change the retention on an existing topic, what is the best way of achieving this?</p><p>If you have spent some time with Kafka your first answer will be the command line tools that come with Kafka. Well yes but no. Since we are working in an enterprise environment, security, (hopefully) has put up firewalls that prevent you from running these command line tools from your local machine. So what do you do now?</p><ul><li>You can describe what to do in the release notes, but for many commands this will require a lot of documentation that can be easily misinterpreted by the implementers.</li><li>You can try to script it using bash commands and include that with the release.</li><li>How about an own Kafka installer? Why not! You’ve got deep pockets right?</li></ul><h4>What is the best way to bring up a new environment?</h4><p>Business has a new client and they need a new testing environment fast. What is the best way how to bring up this new environment? Do you:</p><ul><li>Go through every release note and recreate the environment from scratch? Most likely not since this will take weeks.</li><li>Do you maintain central scripts that can create an environment up to a certain version? Could work but every release will require an update to the scripts, it’ll require some work.</li><li>Copy an existing environment? Might work if you are environment agnostic but most likely there is that one mischievous environment variable out there that foils your plans. Or even better, what if you prefix the topic names with the environment? Come on now, it’s ok to admit it, I know you do, <a href="https://github.com/gwenshap/kafka-examples/blob/master/MirrorMakerHandler/src/main/java/com/shapira/examples/TopicSwitchingHandler.java">we all do</a>.</li></ul><h4>How do you copy data across environments?</h4><p>This one is easy right. Mirror Maker. Except when your data is somehow tied to your environment. For example, you have different schema registries and hence different schema ids in each environment. Confluent of course provides tools that can deal with this but if you don’t have a confluent liscence, well, I guess you need to grow your own. Since I’m not a 100% sure about the need for that liscence, for the sake of argument, lets say there is something specific added to your message header that identifies the environment for auditing. How do you handle this now? Does Kafka provide you a means to copy across environment specific data? Sure, you can grow your own with Kafka Connect or Apache nifi or whatever your heart desires.</p><h4>How do you do impact analysis?</h4><p>How do you track the changes in your environment? If you make a change to a topic who needs to be informed? During a release which consumers will be affected? Does Kafka out of the box provide any sort of data lineage facilities? Kind of, it provides a topology graph on the level of Streams but not on a plain vanilla consumer/producer. Of course to make this information searchable you will need to dump it to some sort of a graph DB and maintain it or rely on cloud providers, like Azure, who offer to infer this for you by looking at the published metrics. How deep are those pockets again?</p><h4>How do you manage topics?</h4><p>The roll out is a great success. It is in fact so successful that everyone wants to use Kafka. Suddenly you have over 10k topics. How do you keep track who is the owner of a topic? Is the data in the topic a golden source that can’t be lost? Perhaps the topic is a legacy topic and needs to be kept around only for audit purposes? Does that topic contain GDPR data? Can you enforce a naming convention? Can you delete that dev topic from that one developer who does all their development directly in the dev environment? (Shouts out to Bobby!) What if you need to maintain multiple versions of the same topic? Can you link ACLs against a business unit like you can in active directory? It’s a good thing that Kafka provides you the means for maintaining topics out of the box, oh wait, it doesn’t, you need to grow your own.</p><h4>How can you check the state of the environment?</h4><p>Did you know that <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-516:+Topic+Identifiers">creating and deleting topics is an asynchronous operation?</a> Those release bash scripts work great, except a developer decided to clean a topic by first deleting it and then recreating it. (I understand this is the worst way of doing this, please don’t spam me in the comments ;)) The script ran without errors, except the topic was never recreated because the delete happened after the create script ran. Or better yet, how about checking a schema for compatibility before you roll out a new version of a service instead of just crashing in production? Maybe there is a way to do some sort of a sanity check on the environment to see that it is in an expected state? Of course! But you need to grow your own.</p><h4>How do you migrate schemas?</h4><p>Who needs to migrate schemas when every attribute is optional, right Google? But perhaps you’re an old school DB guy that likes tight schemas that don’t require all your downstream consumers to implement the same data quality checks over and over again. <a href="https://tkaszuba.medium.com/avro-schema-evolution-strategies-on-kafka-3c072a9a5347">How then do you migrate the schema and the data in production?</a> DBs have flyway scripts, since Kafka is a “DB” it must also have some sort of flyway scripts as well right? Sure, if you grow your own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*mRDgD-oMoSnHLD3WD2Zo_A.jpeg" /><figcaption>Hey it’s Saul Goodman! Protobuf is here!</figcaption></figure><p>You see a common trend here right? Of course, most of these issues also exist in the traditional DB world, which is of course correct. The big difference between Kafka and the traditional DB world is that nobody talks about Governance in Kafka. When you first encounter a Governance Kafka issue you ask yourself, hmm, this seems like a pretty common issue, somebody must have already solved it or at least mentioned it right? Well, no. It seems that either most rollouts hack through the short comings or they’ve already grown their own tools. Which is exactly the case with my current client and their own flyway like mechanism for maintaining Kafka environments. Hopefully in the future I can write about as it’s pretty simple but very powerful.</p><p>So what can you do about Governance in Kafka if you don’t want to invest the time and money in growing your own tools? Well, as it turns out <a href="https://www.confluent.io/blog/governing-data-with-confluent-stream-governance/">Confluent has recently released Governance tools</a> that deal with some of the issues I’ve mentioned above but as far as I know they are only available in the Confluent cloud or with a special Confluent Center build. Did nobody tell you that you would need to migrate into the Confluent cloud after 6 months in production? Silly you. Maybe there’s still time to sink that roll out!</p><p><strong>ps</strong>: I love Kafka and everything about Kafka. If you do take this beautiful journey do it with a clear head so that there are no hard feelings down the line. I’d like to not have to argue with you about whose definition of what a database is, is right.<br><strong>pss</strong>: Ooh, a <a href="https://www.youtube.com/watch?v=vX7J8jARUVM">new contender has entered the ring</a>. I know nothing about the license costs but a nice UI to manage ACLs and metadata of topics looks very nice</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dc127c555c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom Suppression in Kafka Streams]]></title>
            <link>https://tkaszuba.medium.com/custom-suppression-in-kafka-streams-f1d04c7c0098?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1d04c7c0098</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Sun, 10 Oct 2021 21:41:22 GMT</pubDate>
            <atom:updated>2021-10-11T05:59:09.205Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are heavy into stateful transformations in Kafka Streams you’ve most likely hit a case where you wanted to slow or suppress the emitted events and forward only the last pertinent aggregated event, for example, to reason about the end of a batched load.</p><p>In Kafka 2.1, the suppression function was introduced to the <a href="https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/kstream/KTable.html#suppress-org.apache.kafka.streams.kstream.Suppressed-">KTable</a> DSL. While this function works fine for streams that have a lot of updates, <a href="https://rgannu.medium.com/kafka-streams-suppress-surprise-975d4f56870c">it does not work well for topics that receive updates in batches or are fairly stale</a>. While it might seem like an oversight, this behavior is by design and matches how the session window works in the DSL; a session window will only close if a new event arrives to open a new window and closes the old one. A <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time">KIP-424</a> has been put forward to extend the suppression function to also allow suppression based on the wall clock time, but it seems nobody has picked it up as of yet.</p><p>Thankfully, implementing your own suppression is fairly straightforward with the transformer API. It also gives you much more flexibility in customizing its behavior.</p><h3><strong>Prerequisites</strong></h3><p>In order to suppress events we need a way to park the events and only forward them when a certain condition is met. In order to do that we need two main things:</p><ol><li>A State Store</li><li>A Windowing Mechanism</li></ol><p>For this suppression transformer I decided to use a <a href="https://kafka.apache.org/26/javadoc/org/apache/kafka/streams/kstream/SessionWindows.html">session window</a>. A session window represent a period of activity separated by a defined gap of inactivity. Since the transformer is based on the wall clock time the session window will close after a certain period of inactivity once the wall clock time has passed. For example, if our session window is sized to be 10s and a new event arrives 10s <em>after</em> the last event it will fall into a new session window.</p><p>There are several ways to keep track of the metadata required to know when a session window was opened, my favorite is to use <a href="https://kafka.apache.org/26/javadoc/org/apache/kafka/streams/state/ValueAndTimestamp.html">ValueAndTimestamp</a> class. I usually choose this class because it doesn’t wrap the key in a <a href="https://kafka.apache.org/26/javadoc/org/apache/kafka/streams/kstream/Windowed.html">Windowed[K</a>] abstraction, which makes finding keys directly in the state store difficult, furthermore, a specialty <a href="https://kafka.apache.org/26/javadoc/org/apache/kafka/streams/state/TimestampedKeyValueStore.html">TimestampedKeyValueStore</a> already exists that does all the work for you. Creating such a store with the stores builder is a simple process using Stores.timestampedKeyValueStoreBuilder().build()</p><h3>Partitioning</h3><p>Since a suppression function should not change the key the transformer will implement the ValueTransformerWithKey interface. This interface provides us the following transformation method which only requires the value to be returned instead of a KeyValue pair.</p><pre>override def transform(readOnlyKey: K, value: V): V = ???</pre><h3>Punctuation</h3><p>We are only interested in the wall clock time and hence we want only the ProcessorContext to be forwarding our messages. The trick to make this happen is to make the transform method return a null value. The full transform method looks like the following:</p><pre>override def transform(readOnlyKey: K, value: V): V = {<br>  val timestamp = {<br>    if (useMessageTime)<br>      context.get.timestamp()<br>    else<br>      Instant.now.toEpochMilli<br>  }</pre><pre>  store.get.put(readOnlyKey, ValueAndTimestamp.make[V] <br>   (value, timestamp + windowSize))<br> <br>  null.asInstanceOf[V]<br>}</pre><p>The magic happens in the store.put method which puts a key into the state store. If the same key already exists it overwrites it with a new session window, therefore extending it.</p><p>A timestamp override is also provided that allows one to choose the wall clock time over the event timestamp. Choosing to overwrite the event timestamp with the wall clock time effectively changes the session window into a normal <a href="https://kafka.apache.org/26/javadoc/org/apache/kafka/streams/kstream/TimeWindows.html">tumbling time window.</a></p><p>We now need to set the Punctuator to traverse our entire store and forward all the windows that have closed based on regular scheduled intervals.</p><pre>cxt.schedule(<br>  Duration.ofMillis(schedule), <br>  PunctuationType.WALL_CLOCK_TIME, msgTime =&gt; {<br>   autoClose(store.get.all()) { iterator =&gt; <br>     while (iterator.hasNext) {<br>       val window = iterator.next()<br>       if (msgTime &gt; window.value.timestamp()) {<br>         cxt.forward(window.key, window.value.value())<br>         store.get.delete(window.key)<br>       }<br>     }<br>   }<br> })</pre><p><strong>Note:</strong> It is important to ensure that the state store iterator is closed else you will get <a href="https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html">memory leaks</a>.</p><h3>Event Time vs Processing Time</h3><p>Since we are working with the wall clock we need to work with the processing time and not the event time. There are two main reasons for this:</p><ol><li>Since the event timestamp in a Kafka topic is written by the client (by default) we could receive out of order events. Ie: The timestamps could be in random order.</li><li>Our session window will close immediately if the events are old, for example during a replay or restore from backup topic.</li></ol><p>To ensure that we are only working with the processing time we need to tell the streams app that we want to overload the provided event time with the wall clock time when the event was ingested. This is done by setting the <a href="https://jaceklaskowski.gitbooks.io/mastering-kafka-streams/content/kafka-streams-TimestampExtractor.html">TimestampExtractor</a> to use the <a href="https://kafka.apache.org/26/javadoc/org/apache/kafka/streams/processor/WallclockTimestampExtractor.html">WallclockTimestampExtractor</a>.</p><h3>The Code</h3><p>The entire transformer looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80120a8d9c24ebdcd47ae44c5166550d/href">https://medium.com/media/80120a8d9c24ebdcd47ae44c5166550d/href</a></iframe><p>Testing it is straightforward with the Kafka Transformer testing kit:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bd595c54bbd8897a0fed60d905d89ab/href">https://medium.com/media/4bd595c54bbd8897a0fed60d905d89ab/href</a></iframe><h3>Conclusion</h3><p>The transformer API is a powerful way to implement custom functionality on top of the Kafka Streams DSL. A custom suppression transformer is one way to extend the framework to handle cases that Kafka Streams does not handle intrinsically.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1d04c7c0098" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speed up your Kafka Deserialization with a Caching Serde]]></title>
            <link>https://tkaszuba.medium.com/speed-up-your-kafka-deserialization-with-a-caching-serde-87fa6206fc38?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/87fa6206fc38</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[serdes]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[avro]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Thu, 22 Apr 2021 13:28:23 GMT</pubDate>
            <atom:updated>2021-04-22T13:28:23.623Z</atom:updated>
            <content:encoded><![CDATA[<p>The Kafka streams api uses local state stores backed up by a change log topic to maintain state. Since Kafka works on the level of bytes the state stores also require that this level of granularity be used. The translation between the domain language and the Kafka world is the responsibility of the <a href="https://aseigneurin.github.io/2018/08/06/kafka-tutorial-7-kafka-streams-serdes-and-avro.html">Serde</a>. This works alright but does impose a certain performance penalty since reading and writing needs to “always” go through the Serde, especially when using heavy Serdes like <a href="https://code.yawk.at/org.apache.avro/avro/1.9.1/org/apache/avro/generic/GenericDatumReader.java">Avro</a>. I put the always in quotes since it is possible to bypass the Serde with custom state store implementations but it does require some effort to work around the default implementation.</p><p>Performance problems with Deserialization are nicely explained by Lei Chen from Bloomberg during the <a href="https://www.confluent.io/kafka-summit-lon19/why-my-streaming-job-is-slow/">Kafka Summit</a>. There are several ways of tackling this problem. The approach taken by Bloomberg is to create a custom state store. Which is a very good solution and one that works in all situations, I’m actually confused why the framework doesn’t have a State Store like this by default. How to do this and the problems associated with this implementation can be found <a href="https://dev.to/hpgrahsl/towards-a-mongodb-backed-apache-kafka-streams-state-store-20ik">here</a>. But there is also a much simpler way that might be enough for your use case, that of a caching Serde.</p><p>If your data doesn’t change often on either read or write to the state store then caching on the level of the Serde might be enough. It is simple to implement and doesn’t require any interference with the streams api.</p><p>I’ll focus only on the deserialization but the same concepts apply also to serialization. The caching deserializer works by maintaining an in memory cache. On reading it first searches the cache to see if this value has already been deserialized, if it has then it returns this value, if it hasn’t then it calls the inner deserializer and stores the deserialized value in the cache. If the maximum size of the cache is reached it starts purging the oldest entries. Any sort of cache can be used for this purpose, the one presented below uses a LinkedHashMap because it functions like a stack so the oldest entries can be removed when a certain size of the cache is reached.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7c8ed4217bd75fa3d16457ecd9a6804/href">https://medium.com/media/e7c8ed4217bd75fa3d16457ecd9a6804/href</a></iframe><p>The CachingDeserializer uses a Byte object wrapper for the key instead of the byte array as it’s easier to work with. When the cache size is reached an iterator is used to remove the first item in the list and hence no expensive traversals need to take place.</p><p>Using the Deserializer is as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45a6addd34b54dde31015a1b7c0327bd/href">https://medium.com/media/45a6addd34b54dde31015a1b7c0327bd/href</a></iframe><p><strong>Note: </strong>When storing complex mutable objects (such as an Avro) in the cache you should make a clone of the returned object else you will modify the original object in the cache. When reading from the state store directly you do not need to worry about this since the object is cloned during deserialization. You might include this already in the Serde depending on how the Serde will be used.</p><p>So the natural question arises is why would you just not skip this all together and keep the cache directly in memory without using any state stores in the background? I guess it all depends on your use case, state stores provide fault tolerance and deal with multiple partitions very well but do introduce a lot of complexity that is sometimes difficult to deal with. Hopefully in new releases the API will grow and it will become easier to work with them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87fa6206fc38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avro Schema Evolution Strategies on Kafka]]></title>
            <link>https://tkaszuba.medium.com/avro-schema-evolution-strategies-on-kafka-3c072a9a5347?source=rss-37dcdc28ae43------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c072a9a5347</guid>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[avro]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[schema]]></category>
            <category><![CDATA[schema-evolution]]></category>
            <dc:creator><![CDATA[Tom Kaszuba]]></dc:creator>
            <pubDate>Fri, 25 Dec 2020 22:31:15 GMT</pubDate>
            <atom:updated>2020-12-25T22:44:03.240Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have spent any significant time with <a href="http://avro.apache.org/">Avro</a> (or <a href="https://developers.google.com/protocol-buffers/">Protbuf</a>) and are using the <a href="https://docs.confluent.io/platform/current/schema-registry/index.html">Confluent Schema Registry</a> you probably have encountered a breaking schema change characterized by the following mysterious exception.</p><pre>{<br>  “error_code”: 409, <br>  “message”: “Schema being registered is incompatible with an earlier schema”<br>}</pre><p>What happens is the schema registry validates the schema sent by the producer against the “version” that is stored in the schema registry for whatever schema evolution “strategy” that you have set, be it either Forward, Backward or None. The reason that I have version and strategy in quotes is because there is no actual real versioning done by the schema registry. All that the schema registry does is to do a validation check against a previously registered schema to see if you haven’t made a breaking schema change. This is all the schema registry does. If you think this seems like a heavy handed approach to schema validation then stayed tuned, it gets better.</p><p>So you have a breaking schema change? What should you do? Oddly enough there is very little information on the net (that I found) on how to handle this properly. Even Confluent seems oddly silent when I asked them the correct way of tackling this. Most of the answers I have found are along the lines of <a href="https://medium.com/data-rocks/schema-evolution-is-not-that-complex-b7cf7eb567ac">avoid making breaking schema changes</a>. Now, if you told a database designer to design his schema in such a way as to never need to change it, they’d show you the door. So why is schema evolution not talked about in streaming systems? Probably, because just like in the DB world, it’s hard, maybe even harder? So here are my strategies on how to handle this problem in increasing complexity; with advantages and disadvantages to each approach. Some of them have been tested but some are theories and might not be viable or even correct so don’t hold it against me. :)</p><h3>Migration</h3><p>Before getting to the strategies a note on how to migrate between schemas. In order to do this you will need to maintain multiple Avro schema versions either in the same schema using unions or in the same or different libraries using namespaces. Version info will either need to be appended to the name or to the package info or embedded as separate versioned objects. Some way for both schemas to live in the same context.</p><p>Ex:</p><ul><li>my.company.schema1.1</li><li>my.company.v1_1.schema</li></ul><p>If you plan on doing any sort of migration this needs to be thought about from the beginning before rolling the system out to third parties.</p><h3>Strategies</h3><h4>1. No Versioning, No Migration, No Problem</h4><p>If data loss is of no concern and versioning information does not need to be maintained then this is by far the simplest approach. Every time a breaking change is encountered a script is run that deletes the old schema in the schema registry. If you have new data coming in with the old schema then it will fail to deserialize and will be moved to the DLQ.</p><p><strong>If you are also storing data as Avro in local state stores</strong>, they will also need need to be cleaned up as well since a restart of the service will replay the data which will cause an incompatibility exception to be thrown. This for me is a serious flaw in the design of the Confluent Serdes. There is no point in having an internally maintained domain model be the subject of a schema validation check used to validate interface endpoints. I wish Confluent would allow you to override this behavior.</p><p><strong>Advantages</strong></p><ul><li>Easy to automate</li><li>Only the latest version of the schema is maintained</li><li>Simplicity</li></ul><p><strong>Disadvantages</strong></p><ul><li><strong>Potential Data loss</strong></li><li>Upstream and down stream interfaces need to change at the same time as the breaking schema change occurs</li></ul><h4>2. Multiple Versions in One Schema, No Migration</h4><p>By using unions in Avro it is possible to extend the Avro Schema by attaching new field versions. Think of it like an “or” statement. In this way you can work around the compatibility checks of the schema registry, the downstream systems can then decide on which action to take given which version of the data is present. This concept is very similar to the approach taken by Protobuf where you are not allowed to delete fields and just add new ones. It is described here: <a href="https://softwaremill.com/schema-registry-and-topic-with-multiple-message-types/">https://softwaremill.com/schema-registry-and-topic-with-multiple-message-types/</a></p><p><strong>Advantages</strong></p><ul><li>No data loss</li><li>Gives times for upstream and downstream systems to adapt to the change</li></ul><p><strong>Disadvantages</strong></p><ul><li>Breaks the one schema, one version contract in one topic</li><li>The Schema can grow to be very large</li></ul><h4>3. Topic Naming Versioning, No Migration</h4><p>Versioning is maintained by adding the version information to the topic name and this topic is wired to the new version of the service.</p><p>Ex: my_topic_0_1_20201002</p><p>This has the advantage of maintaining the old data in the old topic giving time for this data to expire or be archived, it also gives other systems time to adapt to the changes. If you don’t have a lot of topics then this might be a viable strategy.</p><p><strong>Advantages</strong></p><ul><li>No data loss</li><li>Gives times for upstream and downstream systems to adapt to the change</li><li>No schema mixing in the same topic</li></ul><p><strong>Disadvantages</strong></p><ul><li>Topic governance most likely has to be done centrally through some sort of registry</li><li>Possible explosion of topics</li><li>Could be difficult to support</li></ul><h4>4. Topic Naming Versioning, Topology Migration</h4><p>This builds on the previous strategy but this time with the addition of migration of data from the previous schema/topic. Migration is done by adding the old schema/topic to the topology as the input along with the new schema/topic. The old data is then migrated to the new schema in code and then merged to the normal input stream. Such a strategy would be beneficial if an upstream system can’t be easily migrated so multiple ingest channels need to be maintained.</p><pre>public KStream&lt;SomeKey, SomeValue&gt; process(KStream&lt;SomeKey_v1, SomeValue_v1&gt; oldStream, KStream&lt;SomeKey_v2, SomeValue_v2&gt; newStream)<br>{</pre><pre>  var migratedOldStream = oldStream.mapValues(…)<br>  return newStream.merge(migratedOldStream)</pre><pre>}</pre><p><strong>Advantages</strong></p><ul><li>As above</li><li>Helps to migrate upstream data for migrated downstream services</li></ul><p><strong>Disadvantages</strong></p><ul><li>As above</li><li>Doesn’t help in migrating data in local state stores</li><li>More complicated service configuration</li><li>For topologies that have multiple inputs it will lead to an explosion of input topics</li><li>Migration code hard coded in the topology, cleanup of old migrations will require a new release of the micro service</li><li>Ordering not maintained</li></ul><h4>5. Topic Naming Versioning, Dedicated Migration Service</h4><p>Similar to the previous strategy but this time there is a dedicated service deployed that migrates data between the topics/schemas. Such an approach helps in not polluting business logic with migration logic. Once a migration is finished only this micro service needs to be updated and the business micro service does not need to be redeployed.</p><p><strong>Advantages</strong></p><ul><li>No data loss</li><li>Gives times for upstream and downstream systems to adapt to the change</li><li>No schema mixing in the same topic</li><li>No pollution of business logic with migration logic, therefore two different teams can work independently</li><li>Business service untouched and deals only with the latest schema</li><li>Migration service can easily be brought down or changed</li></ul><p><strong>Disadvantages</strong></p><ul><li>Topic governance most likely has to be done centrally through some sort of registry</li><li>Explosion of topics</li><li>Doesn’t help in migrating data in state stores</li><li>Migration service configuration complicated</li><li>Ordering not maintained</li></ul><h4>6. Schema Naming Versioning, Topology migration</h4><p>Similar to approach 2, this approach mixes different schemas in the same topic. The Kafka application has to then deal with different schemas itself and not assume that a certain type of schema will always arrive.</p><p>Mixing schemas in the same topic is very well described in this blog entry: <a href="https://www.confluent.de/blog/put-several-event-types-kafka-topic/">https://www.confluent.de/blog/put-several-event-types-kafka-topic/</a></p><p>The downsides of schema mixing in the same topic is described here: <a href="https://groups.google.com/forum/#!topic/confluent-platform/XQTjNJd-TrU">https://groups.google.com/forum/#!topic/confluent-platform/XQTjNJd-TrU</a></p><p>The process method must either use a generic record or a specific record base abstraction to allow various Avro message types. In the process method it either calls instanceOf or looks for a specific field in the schema to determine what message arrived. Migration is done after identifying the specific schemas.</p><pre>public KStream&lt;SomeKey, SomeValue&gt; process(KStream&lt;SpecificRecordBase, SpecificRecordBase&gt; stream)<br>{<br>  var schemaSplit = stream.branch((k,v) -&gt; v instanceOf Schemav2, (k,v) -&gt; v.getSchema().getField(“someField”) != null)<br>  <br>  var migratedOldSchema = schemaSplit[1].mapValues(…)<br>  var schemaSplit[0].merge(migratedOldSchema)</pre><pre>}</pre><p><strong>Advantages</strong></p><ul><li>No data loss</li><li>Gives times for upstream and downstream systems to adapt to the change</li><li>No need for topic governance</li><li>No explosion of topics</li><li>Service configuration doesn’t change</li><li>Ordering maintained</li></ul><p><strong>Disadvantages</strong></p><ul><li>Schema mixing in the same topic</li><li>Doesn’t help in migrating data in state stores</li><li>Migration code hard coded in the topology, cleanup of old migrations will require a new release of the service</li><li>Migration code can’t easily be outsourced to a separate service since there is only one input topic</li></ul><p><strong>7. Schema Naming Versioning, Serde migration</strong></p><p>Similar to the previous approach except migration is done on the level of message serialization and deserialization (Serde). This approach separates business logic from migration logic. The Serde contains information about the old schemas and performs the migration to the latest schema on deserialization or serialization. It has the advantage of allowing migration of backup topic data in state stores between schemas. Since the Serde also has access to the the schema registry it is possible to not hard code schema version information and work on the level of the Generic Record and do migrations based on which fields exist in the schema. This makes the migration implementation more flexible but more difficult to test and error prone. The Confluent Serdes are not extensible due to private members, therefore the Serdes need to be created from scratch.</p><p><strong>Advantages</strong></p><ul><li>No data loss</li><li>Gives times for upstream and downstream systems to adapt to the change</li><li>No need for topic governance</li><li>No explosion of topics</li><li>Ordering maintained</li><li>State Stores can be migrated</li><li>Migration code isolated from business logic</li><li>Migration logic can be turned on and off on the level of configuration</li><li>More flexible generic migration</li></ul><p><strong>Disadvantages</strong></p><ul><li>Schema mixing in the same topic</li><li>Micro service configuration a bit more complicated due to using multiple Serdes instead of one SpecificAvroSerde</li><li>More difficult code than in the topology migration</li><li>Migration code included in service so removal requires a new version to be rolled out</li><li>Migration code can’t easily be outsourced to a separate service since there is only one input topic</li></ul><h3>Schema Registry</h3><p>So with all the hassle of versioning and migration that needs to be developed what exactly is the point of the schema registry? It doesn’t really do much, it increases the architecture complexity and it sometimes fails with connection timeouts when creating multiple topics due to it’s one writer architecture.</p><p>I think the only place that the schema registry makes sense is when controlling 3rd party connections or in simple Kafka architectures. When you start rewriting Serdes or start doing migrations in your services you quickly realize that you need to detect breaking schema changes anyway, so why have a centralized schema registry?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c072a9a5347" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>