<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saïd Bouras on Medium]]></title>
        <description><![CDATA[Stories by Saïd Bouras on Medium]]></description>
        <link>https://medium.com/@saidbouras?source=rss-b934e82c6ae6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QyENnklf0q9XoDW9dZtQug.jpeg</url>
            <title>Stories by Saïd Bouras on Medium</title>
            <link>https://medium.com/@saidbouras?source=rss-b934e82c6ae6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:42:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saidbouras/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Our Spark job missed its 4 s SLO. Real-Time Mode got it under 300 ms]]></title>
            <link>https://medium.com/@saidbouras/our-spark-job-missed-its-4-s-slo-real-time-mode-got-it-under-300-ms-518e4f5bf33f?source=rss-b934e82c6ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/518e4f5bf33f</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[Saïd Bouras]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 20:34:10 GMT</pubDate>
            <atom:updated>2026-11-16T20:34:10.751Z</atom:updated>
            <content:encoded><![CDATA[<h4>Moving a Kafka-to-Kafka Structured Streaming job to Databricks Real-Time Mode</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e6KkCnaklPTM-tiN0i8HOg.png" /><figcaption>Same job, micro-batch vs RTM. End to end = input topic to output topic.</figcaption></figure><p>Our stakeholders gave us one number: 4 seconds. A train occupancy event has 4 seconds to get from our input Kafka topic to the output topic that operational tools read.</p><p>Our Spark job usually needed about 2 seconds per micro-batch. On a bad day it needed 4 seconds or more, and every one of those batches broke the SLO.</p><p>We moved the job to Real-Time Mode (RTM) on Databricks. Processing latency is now around 100 ms (87 ms on average). RTM also gives us something micro-batch never did: a real end-to-end latency, input topic to output topic. It averages 205 ms, with p99 around 300 ms, and the dashboard shows zero SLO breaches.</p><blockquote><strong><em>TL;DR</em></strong><em> We moved a Kafka-to-Kafka Spark job from micro-batch to Databricks Real-Time Mode. Micro-batch averaged 1.93 s per batch, with spikes to 4 s and 57 SLO breaches. RTM is at ~87 ms average processing (~100 ms p99), ~205 ms end to end on average, and zero breaches. Most of the work was removing code that assumed it saw a whole batch: </em><em>foreachBatch, a sorted dedup, an explode with </em><em>groupBy, a self-join. RTM needs task slots for every stage at once, so we sized the cluster by counting tasks (3 + 4 = 7, on 8 slots). We also added a listener that sends RTM&#39;s latency percentiles to Datadog, and a test on 1,000 real messages. A </em><em>try_to_date refactoring shipped in the same release and sent trains with </em><em>coaches: [] to production.</em></blockquote><p>This is for data engineers running Kafka-to-Kafka Spark Structured Streaming jobs with a hard latency target or deciding whether Real-Time Mode is worth a proof of concept.</p><h3>The job and the chain around it</h3><p>I’m a data engineer working on passenger-flow data for a public transport operator. Trains on the network have load sensors. As a train runs between stations, each coach reports the weight it measures.</p><p>Our streaming job turns those weights into an estimated number of passengers per coach and per train. Each coach type has two capacities: a comfort capacity, which is the comfortable load, and a maximum capacity, which is the crowded limit. The job computes an occupancy rate against both.</p><p>Operational tools and passenger information need these numbers while the train is still on the line. A count that arrives late describes a train that has already moved on. That is where the 4-second target comes from.</p><p>The chain runs from a train-side producer to on-prem Kafka, then MirrorMaker to an AWS MSK input topic, then the Spark job on Databricks, then an MSK output topic read by operational consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*On3a26zD4z29-nNaEiaQiQ.png" /><figcaption>The chain, and the stages inside the job.</figcaption></figure><p>Each Kafka message carries one or more vehicle reports. A vehicle report has a train, a station, a stop status (atdock, leaving, transit) and an occupancies array with one entry per coach.</p><p>The job does four things:</p><ol><li>Deduplicate. Weights are unstable while passengers board, so we keep only the first transit report after the train leaves a station. Later transit reports for the same station are dropped.</li><li>Enrich. Two static reference tables: a balance weight per line (the average weight of a passenger) and a topology table with the comfort capacity and maximum capacity of each coach type.</li><li>Compute. Passengers per coach is the measured weight divided by the balance weight. Then train totals, occupancy rates, and a validity check: if any coach exceeds its maximum capacity, the whole train report is rejected.</li><li>Publish one Avro record per train to the output topic, keyed by train id.</li></ol><p>The 4-second requirement in our migration plan is defined from “input Kafka availability to publication on the downstream Kafka topic”. That covers only the Spark part of the chain, and everything Spark does counts against us: queueing, reading, state access, serialization and the Kafka write.</p><h3>Why micro-batch kept missing the target</h3><p>The old job was a classic micro-batch pipeline:</p><pre>val validatedStream = reader<br>  .option(&quot;subscribe&quot;, readTopic)<br>  .option(&quot;maxOffsetsPerTrigger&quot;, &quot;10000&quot;)<br>  .option(&quot;minPartitions&quot;, numCores)<br>  .load()<br>  .repartition(numCores)<br><br>validatedStream.writeStream.foreachBatch { (batchDf: DataFrame, _: Long) =&gt;<br>  val formatted = ComputeAndFormatRT.computeAndFormatRT(batchDf, configDf, topologyDf)<br>  // ... select key, Avro value and headers, write to Kafka ...<br>}<br>  .trigger(Trigger.ProcessingTime(0))<br>  .start()</pre><p>ProcessingTime(0) starts the next batch as soon as the previous one finishes. A record waits for the current batch to end, then for its own batch.</p><p>Each batch pays for planning, offsets, a state store commit and a checkpoint, and runs its stages one after the other.</p><p>Inside foreachBatch we exploded every vehicle report into one row per coach, then rebuilt trains with a groupBy. The code comment said it plainly: &quot;requires a batch DataFrame, must be called from foreachBatch&quot;. The maximum-capacity check was a self-join: find the trains with an over-capacity coach, then join them back.</p><p>When things went well, a batch took about 2 seconds (1.93 s on average on our dashboard). Regularly it went to 4 seconds and beyond. We had already merged two shuffles into one and tuned task counts. A good day still meant 2 seconds, with little margin for the bad ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HbbumTqgZAvPn_t-YLFyrA.png" /><figcaption>Before: 1.93 s average batch, and 57 times over the 4 s SLO.</figcaption></figure><h3>What Real-Time Mode is, and when it fits</h3><p>RTM is a trigger type for Structured Streaming. It’s available on Databricks Runtime 18, and that’s why we moved the job to DBR 18 for this migration. Apache Spark 4.3 consolidates it in open source, with stateful support (aggregations, dropDuplicates, transformWithState). We run on Databricks, where it needs two things: the cluster flag spark.databricks.streaming.realTimeMode.enabled=true and Trigger.RealTime(...) on the query. Databricks also requires classic compute with autoscaling, Photon and spot instances turned off.</p><p>The mental model that helped us: in micro-batch, Spark plans a batch, runs it to completion, commits, and repeats. In RTM, the batch is long-running. Tasks stay up and records flow through the stages as they arrive.</p><p>The trigger interval only controls how long each of those batches lasts before Spark checkpoints. Our Trigger.RealTime(&quot;3 minutes&quot;) means a checkpoint every 3 minutes. Records still go out as soon as they are processed</p><p>Two things follow from this, and they drove the migration:</p><ul><li>Every stage runs at the same time, so you need task slots for all of their tasks at once.</li><li>Stateful code sees records one at a time. With a synchronous StatefulProcessor, handleInputRows is called once per input row instead of once per key per batch, so logic that relied on seeing a whole batch has to change.</li></ul><p>RTM fits when you have a latency target in the sub-second to low-seconds range, a Kafka-in, Kafka-out shape, and per-record logic.</p><p>It fits less well when your logic needs to see a batch of records together, when you depend on foreachBatch or a Delta sink (neither is supported), or when cost matters more than latency, because you pay for dedicated, always-on task slots. Aggregations and inner stream-stream joins are supported, but they add state and waiting that eat into a tight latency budget, so we designed them out.</p><p>RTM also gives no latency promise during failure or recovery: a failed batch replays from the last checkpoint, and a longer batch means a longer replay. We haven’t measured ours yet.</p><h3>What had to change</h3><p>Most of the work was rewriting the pipeline so that a vehicle report becomes the unit of processing from start to finish. The core commit was +517/-178 lines across 9 files.</p><h4>Trigger and sink</h4><p>The foreachBatch block had to go: RTM doesn&#39;t support it. We now write with the native Kafka sink:</p><pre>val query = kafkaPayload.writeStream<br>  .format(&quot;kafka&quot;)<br>  .options(writeOpts)<br>  .option(&quot;topic&quot;, writeTopic)<br>  .option(&quot;checkpointLocation&quot;, CheckpointsPath)<br>  .outputMode(&quot;update&quot;)<br>  .trigger(Trigger.RealTime(&quot;3 minutes&quot;))<br>  .queryName(query_name)<br>  .start()</pre><p>We also removed maxOffsetsPerTrigger, minPartitions and the repartition. They are micro-batch knobs, and the repartition added a shuffle that the stateful operator already does by key.</p><p>The query also got a new checkpoint location: the stateful operator changed, and we wanted to keep the old checkpoint intact for rollback.</p><p>The producer options stayed: lz4, linger.ms=5, acks=1. acks=1 is a latency choice. We accept that a broker failure can lose an acknowledged write in exchange for not waiting on replicas.</p><h4>Deduplication: flatMapGroupsWithState to transformWithState</h4><p>The old dedup used flatMapGroupsWithState. For each train/station key it received all the rows of the batch, grouped them by timestamp, sorted them, and walked through them in time order:</p><pre>grouped.toSeq.sortBy(_._1).foreach { case (ts, rs) =&gt;<br>  if (ts &gt; maxTs) {<br>    val status = rs.head.getString(stopStatusIdx)<br>    if (status == TransitStatus &amp;&amp; lastStatus != TransitStatus) output ++= rs<br>    // ...<br>  }<br>}</pre><p>flatMapGroupsWithState isn&#39;t supported in RTM, so this had to be rewritten anyway. The new version is a StatefulProcessor passed to transformWithState (processing-time mode, update output), with a ValueState and a 30-minute TTL refreshed on every input. The core of it:</p><pre>override def handleInputRows(key: String, inputRows: Iterator[Row],<br>                             timerValues: TimerValues): Iterator[Row] =<br>  if (key == PassthroughSentinel) inputRows<br>  else inputRows.filter { row =&gt;<br>    val previous = if (state.exists()) state.get() else OccupancyState(0L, &quot;&quot;)<br>    val ts       = if (row.isNullAt(eventTsIdx)) 0L else row.getLong(eventTsIdx)<br>    val status   = row.getString(stopStatusIdx)<br>    val newer    = ts &gt; previous.maxEventTs<br>    val keep     = newer &amp;&amp; status == &quot;transit&quot; &amp;&amp; previous.lastStatus != &quot;transit&quot;<br>    // ... update state when the report is newer ...<br>    keep<br>  }</pre><p>The sort is gone, and that’s a behaviour change. The new processor decides on arrival order. A report with a timestamp at or below the last one seen for that key is rejected. With batches, an out-of-order pair in the same batch would have been put back in order first.</p><p>We wrote tests that pin down exactly where the two versions differ. They’re named KNOWN GAP so nobody mistakes them for bugs:</p><ul><li>Two reports in the same second, carrying coaches 1–4 and 5–8: the old path merged them into one 8-coach train, RTM publishes coaches 1–4 only.</li><li>A newer atdock report arriving before an older transit: the old path sorted and kept the transit, RTM drops it.</li></ul><p>On our real sample (more on that below), none of these cases showed up, and both versions kept exactly the same station visits. If your producer splits one logical event across several messages, this change matters a lot more for you than it did for us.</p><h4>Building trains without a streaming aggregation</h4><p>The old code exploded coaches into rows and grouped them back. On a streaming DataFrame outside foreachBatch, a groupBy by train becomes a stateful aggregation. Spark would then have to keep state and decide when a train is &quot;complete&quot;, and that wait would land directly on the latency budget.</p><p>Our plan rules it out explicitly: no streaming train aggregation, no stream self-join, no timer.</p><p>We stopped exploding. standardizeVehiclesRT keeps the occupancies array intact, and ComputeVehicleMetricsRT does all the per-coach and per-train work with array higher-order functions on a single row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ahFT1X0Wq-CgCap5_Oxm7A.png" /><figcaption>Before: explode, groupBy, self-join. After: one row per report, array functions.</figcaption></figure><pre>.withColumn(&quot;_coaches&quot;, transform(col(&quot;occupancies&quot;), coach =&gt; {<br>  val ref    = try_element_at(filter(col(&quot;topology&quot;),<br>                 t =&gt; t(&quot;coach_type&quot;) === coach(&quot;coachNumber&quot;)), lit(1))<br>  val weight = coach(&quot;affluence&quot;).try_cast(&quot;int&quot;)<br>  struct(<br>    bround(weight / col(&quot;balance_weight&quot;), 2).as(&quot;passenger_count&quot;),<br>    ref(&quot;coach_max_cap&quot;).as(&quot;coach_max_cap&quot;),<br>    // ... coach order, comfort capacities, invalid flag ...<br>  )<br>}))<br>.withColumn(&quot;_capacity_invalid&quot;, coalesce(<br>  exists(col(&quot;_coaches&quot;), c =&gt; c(&quot;passenger_count&quot;) &gt; c(&quot;coach_max_cap&quot;)), lit(false)))<br>.filter(!col(&quot;invalid_flag&quot;) &amp;&amp; !col(&quot;_capacity_invalid&quot;))<br>.withColumn(&quot;_valid_coaches&quot;, filter(col(&quot;_coaches&quot;), c =&gt; !c(&quot;invalid&quot;)))<br>.withColumn(&quot;_passenger_count&quot;, bround(aggregate(<br>  col(&quot;_valid_coaches&quot;), lit(0.0),<br>  (total, c) =&gt; total + coalesce(c(&quot;passenger_count&quot;), lit(0.0))), 2))</pre><p>The maximum-capacity self-join became one exists. Before RTM, on the micro-batch branch, we had already tried replacing that self-join with a window function. That helps inside foreachBatch, but a non-time window over a streaming DataFrame isn&#39;t an option, so the array version is the one that shipped.</p><p>To make this work, the topology table is pre-grouped into one array per line at startup (topologyByLine), so each vehicle row carries its own lookup table after the join.</p><h4>Static references, checked at startup</h4><p>Both reference tables are loaded once, cached, and broadcast-joined to the stream:</p><pre>val topologyRef = spark.table(&quot;...capacity_topology_ref&quot;)<br>ComputeVehicleMetricsRT.validateReferences(configDf, topologyRef)<br>val topologyDf = ComputeVehicleMetricsRT.topologyByLine(topologyRef).cache()<br>configDf.count()<br>topologyDf.count()</pre><p>validateReferences fails the job at startup if there is more than one balance weight per line, or more than one topology row per line and coach type. A duplicate there would silently multiply rows in the join, and we&#39;d rather crash before reading the first record.</p><p>The trade-off is explicit in the plan: &quot;Reference tables are snapshots loaded at startup. Reference changes require restart.&quot; These tables change rarely, so we accepted it for now.</p><h4>Cluster and configuration</h4><p>RTM changes how you size a cluster. With micro-batch, a small cluster just makes batches slower. With RTM, every stage holds its task slots at the same time, so you count them up front.</p><p>Our input topic has three partitions and the stateful stage has four. That&#39;s 3 + 4 = 7 concurrent tasks. Two r7gd.xlarge workers give 8 slots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7kqmABKqKTE3jzONJi-DQ.png" /><figcaption>3 source + 4 stateful = 7 tasks, on 8 slots (2 x r7gd.xlarge).</figcaption></figure><p>The plan also has a cheaper single-worker variant with 1 shuffle partition (3 + 1 = 4 slots), not yet validated under peak load.</p><p>The relevant part of our job cluster config (Databricks Asset Bundle variables):</p><pre>num_workers: 2                      # fixed, no autoscaling<br>runtime_engine: &quot;STANDARD&quot;          # Photon off<br>spark_conf:<br>  spark.databricks.streaming.realTimeMode.enabled: &quot;true&quot;<br>  spark.sql.adaptive.enabled: &quot;false&quot;<br>  spark.sql.shuffle.partitions: ${var.num_partitions}         # 4<br>  spark.sql.streaming.stateStore.partitions: ${var.num_partitions}<br>  spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage: &quot;true&quot;<br>  spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB: &quot;1024&quot;<br>  # ... state store provider, checkpoint format, listeners ...<br>spark_env_vars:<br>  MALLOC_ARENA_MAX: &quot;2&quot;</pre><p>The full cluster config is in this gist:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/edffcab1d5185fcef04d5ae8f3418232/href">https://medium.com/media/edffcab1d5185fcef04d5ae8f3418232/href</a></iframe><p>AQE is off because adaptive repartitioning would change the number of tasks we sized for. Shuffle and state store partitions are tied to one variable so they can’t drift apart.</p><p>Bounded RocksDB memory and MALLOC_ARENA_MAX=2 come from a separate memory-growth investigation on this job; RTM doesn&#39;t fix that by itself.</p><h3>Observability: a listener for RTM latencies</h3><p>Micro-batch metrics such as batch duration lose their meaning when a batch lasts 3 minutes. RTM reports its own latency percentiles in the query progress, and we wanted them in Datadog next to everything else.</p><p>Our platform team maintains a shared monitoring library with a ComposableStreamingListener attached to every streaming query (throughput, Kafka lag, query health). For RTM we added RTMStreamingListener. It&#39;s registered through spark.sql.streaming.streamingQueryListeners, as shown above, so the job code doesn&#39;t change.</p><p>It reads three metric families from each progress event:</p><ul><li>processingLatencyMs: from the moment RTM reads a record to the moment it starts writing it to the sink.</li><li>sourceQueuingLatencyMs: from the record landing on Kafka (log-append time) to RTM first reading it.</li><li>e2eLatencyMs: from the Kafka append to the start of the sink write.</li></ul><p>None of the three include the sink write itself. That’s one more reason to keep an independent check on the output topic (see below).</p><p>The core of the listener, trimmed:</p><pre>final case class Percentiles(P0: Double, P50: Double, P90: Double, P95: Double, P99: Double)<br>final case class RtmMetrics(<br>  processingLatencyMs: Percentiles,<br>  sourceQueuingLatencyMs: Percentiles,<br>  e2eLatencyMs: Percentiles<br>)<br>final case class ProgressSlice(latencies: Option[RtmMetrics])<br><br>override def handle(progress: StreamingQueryProgress): Unit = {<br>  val slice = readFromArray[ProgressSlice](progress.json.getBytes(StandardCharsets.UTF_8))<br>  slice.latencies match {<br>    case Some(rtm) =&gt;<br>      val tags = extractTags(progress)   // query_name, query_id<br>      emit(&quot;processingLatencyMs&quot;, rtm.processingLatencyMs, tags)<br>      emit(&quot;sourceQueuingLatencyMs&quot;, rtm.sourceQueuingLatencyMs, tags)<br>      emit(&quot;e2eLatencyMs&quot;, rtm.e2eLatencyMs, tags)<br>    case None =&gt;<br>      statsdClient.increment(&quot;rtm.metrics_missing&quot;)<br>  }<br>}<br><br>private def emit(metric: String, p: Percentiles, tags: Seq[String]): Unit = {<br>  statsdClient.gauge(metricName(s&quot;rtm.default.$metric.p50&quot;), p.P50, tags: _*)<br>  // ... p90, p95, p99 ...<br>}</pre><p>The full source is in this gist:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ea938dd79b7b9ce729c951f795b2a40/href">https://medium.com/media/7ea938dd79b7b9ce729c951f795b2a40/href</a></iframe><p>Some choices in the listener:</p><ul><li>It runs on the driver, once per progress event, so nothing touches the per-record path. Spark computes the percentiles; we only forward them.</li><li>The JSON is decoded with a compile-time jsoniter-scala codec that only declares the fields we use.</li><li>A missing RTM block is a signal. If a query silently falls out of real-time mode, the block disappears. The listener then increments rtm.metrics_missing so an alert can fire, and sends no latency values at all for that event. Zeros would look like a very healthy job.</li><li>It goes through DogStatsD over UDP to the local Datadog agent, using one non-blocking client per JVM with aggregation, flushed every 3 seconds.</li></ul><p>In Datadog the metrics land as &lt;prefix&gt;.spark.streaming.rtm.default.&lt;family&gt;.&lt;p50|p90|p95|p99&gt;, tagged with query_name, query_id and env. That&#39;s 12 gauges plus the counter. P0 is parsed but not sent.</p><p>If you copy this: the codec reads whatever JSON key matches the case class field name. Check that name against the progress JSON your runtime actually emits. A mismatch doesn’t fail; you’ll just see rtm.metrics_missing climbing and no gauges.</p><p>The dashboard built on these gauges is in the Results section below.</p><p>We also kept a separate SLA job that reads the output topic, compares each record’s Kafka timestamp with the ingestion_timestamp header we copy from the input record, and counts breaches above 4,000 ms. That check sits outside Spark, so it still works if the listener is wrong.</p><h3>Testing with real data</h3><p>Synthetic unit tests covered the obvious cases: rounding, sorting, negative and missing coach fields, empty occupancies, missing references, maximum-capacity equality and rejection, Avro encoding, and the single-row dedup callback.</p><p>They weren’t enough. What gave us confidence was OccupancySampleTest: 1,000 messages captured from our pre-production topic, in offset order, stored as a gzipped fixture. Those 1,000 messages contain 24,207 vehicle reports, each with 8 coaches.</p><p>The test runs the same fixture through both the RTM path and a frozen copy of the old micro-batch filter (LegacyFilterOccupancyMB, test-only), and asserts:</p><ul><li>both keep exactly the same 827 station visits;</li><li>the published payloads are equal, and every train carries 8 coaches.</li></ul><p>A limit of these tests: locally they run with Trigger.AvailableNow because the real-time trigger needs Databricks. They prove the logic is equivalent and say nothing about RTM behaviour. Compatibility and latency can only be established on the Databricks cluster.</p><h3>Results</h3><ul><li><strong>Spark-side latency</strong> (batch duration vs processingLatencyMs): 1.93 s average batch with spikes to 4 s and beyond, now 86.7 ms average and ~100 ms p99.</li><li><strong>End-to-end latency</strong> (input topic to output topic): no built-in metric before, now 205 ms average and p99 ~300 ms.</li><li><strong>Times over the 4 s SLO:</strong> 57 before, 0 now.</li></ul><p>The first row isn’t a perfect like-for-like. Batch duration was the only latency number micro-batch gave us, and it measures a batch, not a record. RTM reports latency per record, and adds the end-to-end number we never had before.</p><p>The p99 end-to-end line isn’t flat either. On the day below it spikes a few times, up to 2.4 s. That’s still well under 4 s, and the breach counter stays at 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fk35okZ01_tvHumYFyh9ZA.png" /><figcaption>After: per-record processing and end-to-end latency (p50/p95/p99) over a day, zero breaches.</figcaption></figure><p>The cost side: a fixed two-worker cluster on on-demand instances, running all the time, with task slots sized for every stage at once.</p><h3>The one-line bug that sent empty trains to production</h3><p>A few days after the production release, consumers told us that trains were arriving with coaches: []. Trains were published on time, with a passenger count of 0. Our tests had not caught it.</p><p>The cause had nothing to do with RTM. A refactoring commit, a week before the migration, moved our code to “safe” functions (try_cast, try_to_date). It changed the balance weight filter in OccupancyJob.scala from:</p><pre>.filter(col(&quot;valid_to&quot;) === &quot;-&quot; || col(&quot;valid_to&quot;).isNull)</pre><p>to:</p><pre>.filter(col(&quot;valid_to&quot;) === try_to_date(lit(&quot;-&quot;)))</pre><p>try_to_date(&quot;-&quot;) returns NULL, and x = NULL is never true. The filter returned zero rows, so the left join gave every vehicle a NULL balance weight, every passenger count became NULL, and the final step dropped coaches with no count. Nothing failed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HtjHE6WMvXHcwRnTyUdf-g.png" /><figcaption>valid_to filter, then try_to_date(‘-’) = NULL, then 0 rows, then NULL balance weight, then NULL counts, then coaches: [].</figcaption></figure><p>The fix was one line:</p><pre>.filter(col(&quot;valid_to&quot;).isNull || col(&quot;valid_to&quot;).cast(&quot;string&quot;) === &quot;-&quot;)</pre><p>The refactoring shipped in the same release as RTM, so RTM was the first suspect. Running the real fixture through both paths settled it: dedup and train assembly matched exactly, so the problem was upstream, in the reference load.</p><p>Our synthetic tests built their balance weight table in the test itself and never exercised the filter. The fixture now carries a regression test that reproduces the old predicate and asserts it produces empty coaches.</p><h4>What we’d tell ourselves earlier</h4><ul><li>Start debugging from what the consumer sees. “Empty coaches” pointed at the reference join much faster than latency graphs did.</li><li>Keep a real-data fixture and a frozen copy of the old logic. It proves equivalence cheaply and rules out suspects.</li><li>Write down the behaviour you lose. Arrival-order dedup and unmerged split reports are real differences, and the KNOWN GAP tests make them visible to the next person.</li><li>Delivery is still at least once. Retries and checkpoint replay can duplicate output records, so check that your consumers tolerate duplicates.</li></ul><h3>What’s next</h3><ul><li>Refreshing the references without a restart. Today a new balance weight means a restart. That’s fine for tables that rarely change, but we’d like it to be automatic.</li><li>Alerting on rtm.metrics_missing and on the SLA job&#39;s breach count, so a silent fallback or a regression pages someone.</li><li>Measuring the single-worker variant (4 slots) at peak load, before deciding whether the second worker is worth its cost.</li><li>Recovery latency. We measured steady state. Restarts and checkpoint boundaries need their own numbers.</li></ul><h3>If you’re considering RTM</h3><p>If your Spark job goes from Kafka to Kafka, has a hard latency target and works per record, RTM is worth a proof of concept. Most of our effort went into code that quietly assumed it saw a whole batch, so look there first.</p><p>If you’ve run RTM in production, I’d like to hear what you measured.</p><p>I’m a freelance data engineer working on Spark, Kafka, Databricks and cloud infrastructure. <br>If you’re facing a similar migration and want to talk it through, message me on <a href="https://www.linkedin.com/in/saidbouras">LinkedIn</a>. <br>Questions on any step are also welcome in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=518e4f5bf33f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reliable and faster integration tests with Apache Cassandra]]></title>
            <link>https://medium.com/@saidbouras/running-integration-tests-with-apache-cassandra-42305dc260a6?source=rss-b934e82c6ae6------2</link>
            <guid isPermaLink="false">https://medium.com/p/42305dc260a6</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Saïd Bouras]]></dc:creator>
            <pubDate>Fri, 27 Oct 2017 15:20:44 GMT</pubDate>
            <atom:updated>2019-04-19T09:42:04.763Z</atom:updated>
            <content:encoded><![CDATA[<p>by <a href="https://github.com/saidbouras">Saïd Bouras</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tsPalek9LFNx0m-LdwtIBQ.png" /></figure><p>Apache Cassandra is a distributed database for managing large amounts of structured data across many servers, while providing highly available service and no single point of failure. It offers capabilities such as: continuous availability, linear scale performance, operational simplicity, easy data distribution across multiple data centers and cloud availability zones.</p><h4><strong>Embedded databases ?</strong></h4><p>In team development, when working with databases, we often (or absolutely for TDD lovers) need an instance of that database for running integrations tests, many solutions exists categorized as <em>“Embedded Database”</em>. <br>When it concerns Cassandra, in this category, two solutions are often used in the community and both give us a way to start an “<em>EmbeddedCassandra” </em>:</p><ul><li><a href="https://github.com/jsevellec/cassandra-unit">CassandraUnit</a><em> : </em>like <em>DBUnit</em> but for Cassandra, give us many way to start an embedded cassandra with annotations and easy configuration, it’s integrate perfectly and easily with spring projects and give us the possibility to handle many data format files like CQL3, JSON and YAML.</li><li><a href="https://github.com/doanduyhai/Achilles">Achilles framework</a> : is an open-source object mapper for Cassandra that provide among many features (like codec system to define your own types, materialized view support and <strong>GROUP BY</strong> keyword that appeared in version 3.10) a testing library that create a JUnit Rule to start an embedded Cassandra server and a nicely way to insert your tests data with a <em>ScriptExecutor</em> class that handles templating (visit the <a href="https://github.com/doanduyhai/Achilles/wiki/Unit-testing">Achilles github wiki </a>for more details).</li></ul><p>The problems with Embedded database is that it’s not exactly the same as the real one (it’s generally an in-memory software downloaded from the sources), and with cassandra when building your tests in CI pipelines (with maven for example), it may happen that one test is failing because in the previous one, the embedded server may not have stopped yet, so when the current one try to start a new instance, it may result to ports and/or data ressources conflicts (due to <em>Server</em> starting in the same JVM process), so it becomes very difficult to start and handle two or more instances of the same embedded database when executing integration tests.</p><h4><strong>Docker for integration tests</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*7a8Qffxkg7WuePBZUebYSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NjXzaw15-2QPs5dHRW9fmg.png" /></figure><p>An other elegant way is using a container to encapsulate the database and start it in every test class. <br>It provide isolation due to docker container nature so that we can start as many cassandra container as we like, and on top of that, it’s the real database not an embedded one so tests are run in an environment very close to production’s one.<br>To running such a container we can use the<strong> </strong><a href="https://github.com/testcontainers/testcontainers-java">test-containers</a> framework that provides an easy way to start a JUnit rule with a docker image as an argument and start a database (or even web browser for UI testing with selenium) inside a container.</p><p>Here below a test example :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2feeac2e9b729a12485877ba17782be3/href">https://medium.com/media/2feeac2e9b729a12485877ba17782be3/href</a></iframe><p>With a <em>GenericContainer</em> class, we can launch a docker container with the name and the associate desired tag.</p><h4>Cassandra is slow to start</h4><p>All the solutions mentioned above have been tested and finally <em>test-containers</em> was adopted for running integration tests with Cassandra, but we noticed that whether it was with Embedded solutions or containerization, Cassandra is very slow to start (and to stop as well).<br>After looking though it, we found that Cassandra was taking around 30 sec to start and 20 seconds to stop. So for every test class using Cassandra, we was waiting 30 seconds for the database to start and the first test to begin.<br>I did some research to find why cassandra was taking so much time to start, and I found two reasons :</p><ol><li>Gossip : when a cassandra node is starting, a gossip thread try to discover other cassandra nodes in order to build/join and form a cluster.</li><li>Vnodes (or Virtual nodes) : Prior to Cassandra 1.2, we had to specify and calculate a token to every cassandra node wich determine the node’s position in the ring and the “range” or the “portion” of data that node will handle according to its hash value. <br>Now cassandra use Vnodes which allow each node to own a larger number of small partition ranges, tokens are no longer required because vnodes pre-determine with a consistent hashing the node to distribute the data. (More explanation : <a href="https://docs.datastax.com/en/cassandra/3.0/cassandra/architecture/archDataDistributeDistribute.html">here</a>)</li></ol><p>So we noticed that the two features above are not necessary in testing environment because we generally use a single node cassandra cluster when testing. So we try to find out if it was possible to disable gossip and vnodes and it seems it was, by changing properties in the cassandra configuration file : <em>cassandra.yaml</em>.<br>So I build a docker image on top of the cassandra official image. In brief I just create a script which modify the configuration and all properties relative to gossip and vnodes features.<br>When using my own cassandra image instead of the official one, with vnodes and gossip disabled, it takes now around 10 seconds to start, so we gain 20 seconds per tests class and my project build takes now around 5 minutes less than it was before.</p><p>You can check my <a href="https://github.com/saidbouras/cassandra-docker-unit">https://github.com/saidbouras/cassandra-docker-unit</a> for the fast cassandra startup container, the tags I use are mapped to the official cassandra image.</p><p>Comments and PRs are welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42305dc260a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>