<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Leroux Romain on Medium]]></title>
        <description><![CDATA[Stories by Leroux Romain on Medium]]></description>
        <link>https://medium.com/@lerouxrgd?source=rss-94e1427402eb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*_f7hGYFoGIL3_9QiSuGU9A.jpeg</url>
            <title>Stories by Leroux Romain on Medium</title>
            <link>https://medium.com/@lerouxrgd?source=rss-94e1427402eb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:17:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lerouxrgd/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Achieving exactly-once processing with Flink and InfluxDB]]></title>
            <link>https://medium.com/lumen-engineering-blog/achieving-exactly-once-processing-with-flink-and-influxdb-3b876915f8cd?source=rss-94e1427402eb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b876915f8cd</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[flink]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Leroux Romain]]></dc:creator>
            <pubDate>Wed, 05 Jun 2019 17:03:59 GMT</pubDate>
            <atom:updated>2019-06-05T17:03:59.349Z</atom:updated>
            <content:encoded><![CDATA[<p>We have talked a lot about data on the Streamroot tech blog, and for good reason. Powering millions of video sessions online every day, we have a lot of material to work with. Delivering video content — especially with hybrid CDN and peer-to-peer systems — is a highly complex undertaking that requires us to continuously collect and process feedback events to gauge its performance and make sure everything is working properly.</p><p>As we discussed in previous posts, Streamroot uses its data pipeline to <a href="https://medium.com/streamroot-developers-blog/how-to-be-data-driven-when-you-arent-netflix-or-even-if-you-are-part-2-ab-testing-e57ecc46102d">AB test new features</a>, to understand <a href="https://medium.com/streamroot-developers-blog/streamroot-data-science-series-1-per-browser-qos-analysis-f66b7e9578b1">the effects of external factors on our delivery solution</a>, and to improve the overall quality of our products. It is also used to expose certain traffic data to our customers via our graphic user interface, which is what I’d like to talk about today.</p><p>Put simply, we send feedback metrics as json payloads to a Kafka broker from where various consumers process those metrics for different purposes. One of our critical consumers is responsible for feeding our customer-facing time series database InfluxDB. This database is accessed by our customer dashboard, which displays a variety of peer-to-peer delivery and quality of service data for the broadcaster’s streams, platforms, ISPs, etc. In this post I’d like to focus on this critical consumer whose job is to pre-aggregate time series data (hence its name, the aggregator) so that InfluxDB isn’t overloaded and can respond smoothly.</p><p>This aggregator is actually a Flink job that:</p><ul><li>Reads data from Kafka</li><li>Groups data by some key of interest (for instance a stream id)</li><li>Aggregates metrics for each key (for instance counting unique users on a given stream)</li><li>Does all that on time windows of 5 minutes</li><li>Sends the results to InfluxDB</li></ul><p>Flink has really good built-in windowing and aggregation support so most steps weren’t too difficult to implement. However, the last step — that is, sending results to InfluxDB — was tricky to get right as many things can go wrong. For instance, InfluxDB can be down, a Flink machine can fail, etc. And since the resulting data will be exposed to our customers we needed to have strong guarantees such as exactly-once processing and no data loss.</p><p>To ensure that stateful computations are resilient, Flink provides a checkpoint based-mechanism. At regular intervals, it snapshots the states of the stateful components of a given job and saves it to an external backend (for instance files on a distributed storage such as HDFS). As an end user, what we have to do is essentially implement interfaces for our custom components and Flink will do the rest!</p><p>In this post we’ll implement a fully resilient InfluxDB sink (Flink’s abstraction for outputting data); we’ll start with a bare-bones implementation and improve it gradually. All of the code is available on <a href="https://github.com/streamroot/demo-influxer/">Github</a> and the tests can be run locally through a docker-compose setup that is also provided (it will spin up Flink and InfluxDB).</p><h3>Basic implementation — InfluxSinkV1</h3><p>First of all, let’s write a simple implementation of a sink writing data points to InfluxDB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d40c03cebf0a1fbb82f43e356fb63806/href">https://medium.com/media/d40c03cebf0a1fbb82f43e356fb63806/href</a></iframe><p>A sink class must implement SinkFunction&lt;IN&gt;; however, here we choose to extend RichSinkFunction&lt;IN&gt;. The reason for this is that the rich version provides a public void open(...) method that can be used to initialize any kind of complex/non-serializable state. In this case, the transient InfluxDB client is not Serializable which is why we have to create it through this open(...) method.</p><p>The main sink method invoke(...) is quite simple: it takes a Point and uses the influx client to send it.</p><p>Let’s check that everything is working correctly with a quick unit test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb1f75e3f62f857b9ce9c8668e003ab4/href">https://medium.com/media/cb1f75e3f62f857b9ce9c8668e003ab4/href</a></iframe><p>Assuming that you have docker-compose running, as mentioned in the introduction, you can run this test with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fabbc348082912bb3c2254d637a707ca/href">https://medium.com/media/fabbc348082912bb3c2254d637a707ca/href</a></iframe><p>Everything looks good! However, there’s an issue. Writing data points one by one, every time that invoke(...) is called is quite inefficient for InfluxDB. So let&#39;s see what we can do about that.</p><h3>Adding batching support — InfluxSinkV2</h3><p>It turns out that the influx client has built-in support for batching! To use it all we need to do is to call enableBatch(...) as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04f1401ff4a714973d4761fb94e031c4/href">https://medium.com/media/04f1401ff4a714973d4761fb94e031c4/href</a></iframe><p>As for the unit test, we need to change it a bit to take into account the batching duration:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2231c11a47ecf09da5cb9f30f942503d/href">https://medium.com/media/2231c11a47ecf09da5cb9f30f942503d/href</a></iframe><p>You can test it with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/84dbc49a7316442a7fabe60feba323c6/href">https://medium.com/media/84dbc49a7316442a7fabe60feba323c6/href</a></iframe><p>But is it really that simple to handle batching properly? What happens if a failure occurs between two batches? It’s very likely that data points being buffered will get lost as Flink has no idea that we are buffering them.</p><p>So let’s see how we can handle a stateful sink properly with Flink.</p><h3>Fully resilient implementation — InfluxSinkV3</h3><p>To achieve resiliency we need to have the following mechanisms in place:</p><ol><li>Explicitly handle buffered points (leverage Flink’s checkpoints)</li><li>Control batches of data points sent to InfluxDB</li><li>Retry in case of failure (stop processing until recovery succeeds)</li></ol><h3>Explicitly handle buffered points</h3><p>Flink’s documentation mentions managed operator states for handling this situation. This mechanism requires us to implement CheckpointedFunction to leverage Flink&#39;s checkpoints and thus be able to snapshot/recover the data points that are being buffered and batch sent.</p><p>Here are the two new methods that we will have to implement:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4db138f69f7aac1e85faa99f021738fc/href">https://medium.com/media/4db138f69f7aac1e85faa99f021738fc/href</a></iframe><p>So we have to buffer points within our sink implementation. Let’s do that with a simple List&lt;Point&gt;:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3cabb8c4637421f53a151ad1cd39c76/href">https://medium.com/media/c3cabb8c4637421f53a151ad1cd39c76/href</a></iframe><p>In this new version our sink’s main method invoke(...) is now simply buffering points up to batchSize:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d5c836333958826a6b7efc722a0abd7/href">https://medium.com/media/2d5c836333958826a6b7efc722a0abd7/href</a></iframe><p>Let’s implement snapshotState(...) and initializeState(...) to respectively backup and restore bufferedPoints through checkpointedState:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8242aa5389d0001002550bc9bed9cdcd/href">https://medium.com/media/8242aa5389d0001002550bc9bed9cdcd/href</a></iframe><h3>Control batches of data points sent to InfluxDB</h3><p>Since we stopped relying on the influx client’s built-in batching feature, we will have to roll our own. Luckily we can leverage OkHttpClient, on top of which influx client is built, to achieve batching.</p><p>To this end, we’ll start by defining a custom InfluxBatchService as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/856be5100a4fce1cd293e6668957e6ee/href">https://medium.com/media/856be5100a4fce1cd293e6668957e6ee/href</a></iframe><p>Now let’s come back to the mysterious batchWrite(bufferedPoints) that we skipped earlier in the new invoke(...) method and let&#39;s see how it uses our new InfluxBatchService:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85de490d15f22a5a5e2ebf345e881755/href">https://medium.com/media/85de490d15f22a5a5e2ebf345e881755/href</a></iframe><p>It simply iterates through data points and call lineProtocol to format them to the InfluxDB protocol before batch sending them with a POST request.</p><p>There has been a lot of changes since InfluxSinkV1 already, but are we done? Not yet! With our new batching approach we&#39;re flushing data points only when our buffer is full. What if we buffer points that are the result of 10-minute windowed aggregations and that our buffer isn&#39;t full? Well, they won&#39;t be flushed until the next 10-minute window ... Not very nice.</p><p>So let’s add a final piece to our custom batching implementation: a time limit. The idea is that we want to flush our buffered point as soon as one of the two following conditions is met: either the batchSize is reached, or batchFreqMs has elapsed.</p><p>To achieve that we can make use of a SingleThreadScheduledExecutor that will regularly flush the buffered points. To make sure that everything is thread-safe we can simply add synchronized to all methods interacting with bufferedPoints.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e79417dd2dcb265dbca94db1316401b5/href">https://medium.com/media/e79417dd2dcb265dbca94db1316401b5/href</a></iframe><p>Let’s not forget to clean up the scheduled executor when the sink is being stopped:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b748da5829b60785e54e59207d2db81d/href">https://medium.com/media/b748da5829b60785e54e59207d2db81d/href</a></iframe><p>With this new mechanism, invoke(...) will also have to make use of flushPoints():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4e815b0878055fa5604a943bb212895/href">https://medium.com/media/d4e815b0878055fa5604a943bb212895/href</a></iframe><p>And now flushPoints() is responsible for triggering batchWrite(bufferedPoints) and clearing the buffer:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a783be308477de15919443696055bfb/href">https://medium.com/media/7a783be308477de15919443696055bfb/href</a></iframe><p>We finally have a complete batching mechanism in place! But as you can see from the last line in flushPoints(), we&#39;re not doing much when an exception occurs... This brings us to our last feature: proper error handling and retries.</p><h3>Retry in case of failure</h3><p>The last piece of this InfluxSinkV3 is the retry mechanism that will make use of the <a href="https://github.com/jhalterman/failsafe">Failsafe</a> library. What we want is to stop processing when an InfluxDB related error occurs, and have a retry policy to help recover the situation.</p><p>Let’s define a retry policy that will just keep on retrying to send the current batch of data at regular intervals:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/003795f53d75754b3a78664b926736fc/href">https://medium.com/media/003795f53d75754b3a78664b926736fc/href</a></iframe><p>Now we wrap our call to batchWrite(bufferedPoints) with a Failsafe.with(retryPolicy) call. It will handle IOException s and errors returned by InfluxDB by retrying infinitely.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da4079749ac8ce93188df443615c6641/href">https://medium.com/media/da4079749ac8ce93188df443615c6641/href</a></iframe><p>Last but not least, let’s modify batchWrite(...) so that it can be used by Failsafe functions. Let&#39;s also keep track of the number of retries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/006ce1d52b6e5fa8e944e253b2176f8a/href">https://medium.com/media/006ce1d52b6e5fa8e944e253b2176f8a/href</a></iframe><p>And at last we’re all done! For completeness sake <a href="https://github.com/streamroot/demo-influxer/blob/master/src/main/java/io/streamroot/InfluxSinkV3.java">here</a> is the final version of InfluxSinkV3.</p><h3>Testing resiliency</h3><p>Let’s summarize all the features we implemented:</p><ul><li>Back up and restore state with bufferedPoints / checkpointedState</li><li>Custom InfluxBatchService with configurable flushing of batchSize / batchFreqMs</li><li>Error handling and retries with Failsafe</li></ul><p>Our features look good; now we’d like to verify them with some unit tests. Let’s at least check that our retry mechanism is working well.</p><p>For that we’ll use a library called <a href="https://github.com/NetCrusherOrg/netcrusher-java">NetCrusher</a> that can simulate a network failure. Let&#39;s modify our test setup so as to proxy connections to our local InfluxDB through a TcpCrusher so that we can stop and restart TCP traffic at will.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ff92daee77b05cd49322746ec8f44d5/href">https://medium.com/media/9ff92daee77b05cd49322746ec8f44d5/href</a></iframe><p>Our new resiliency test looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d69ff97c785b59ea4e7fb99d13df0090/href">https://medium.com/media/d69ff97c785b59ea4e7fb99d13df0090/href</a></iframe><p>Again assuming that you have docker-compose running, you can run this test with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/09d66ef5f9d7aa995947b040725a6884/href">https://medium.com/media/09d66ef5f9d7aa995947b040725a6884/href</a></iframe><p>The test should pass and we can see in the output that it has retried many times before successfully sending the current batch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccfd7080f85c77413673a54286054438/href">https://medium.com/media/ccfd7080f85c77413673a54286054438/href</a></iframe><h3>Improving usability with a builder — InfluxSinkV4</h3><p>We ended up with a lot of parameters for our InfluxSink which make it a bit cumbersome to use. A simple solution for that is to use a builder to set up all parameters in a clean and fluent fashion.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e35b4522b9c1038a0d6458d8d3a8d253/href">https://medium.com/media/e35b4522b9c1038a0d6458d8d3a8d253/href</a></iframe><p>Let’s see it in action with a complete Flink job that will:</p><ul><li>Read and parse integer data points from a network socket</li><li>Sum data over 10-second windows</li><li>Send the results to a local InfluxDB (running in docker)</li></ul><p>Complete instructions to run this job can be found on <a href="https://github.com/streamroot/demo-influxer/blob/master/README.md">Github</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6f5d00eed678ee4d33f13554048a5e0/href">https://medium.com/media/f6f5d00eed678ee4d33f13554048a5e0/href</a></iframe><h3>Conclusion</h3><p>It’s been quite a journey from the naive InfluxSinkV1 to the fully resilient InfluxSinkV4 but it was worth it! We’ve addressed efficiency issues and stateful computation pitfalls when working with Flink and InfluxDB:</p><ul><li>Batch write data points to InfluxDB</li><li>Regularly back up buffered points with Flink’s checkpoints</li><li>Handle errors with a fine-tuned retry policy</li></ul><p>These mechanisms and the resulting exactly-once processing guarantees have helped us build reliable and scalable data pipelines that give our customers valuable insights about their video traffic. I hope the local demo application highlighted it and that you can use this as a basis for your own Flink applications. Stay tuned for more insights on how we fine tune our data pipeline, and if you are interested in working on these types of challenges, drop us a line at jobs@streamroot.io.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b876915f8cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lumen-engineering-blog/achieving-exactly-once-processing-with-flink-and-influxdb-3b876915f8cd">Achieving exactly-once processing with Flink and InfluxDB</a> was originally published in <a href="https://medium.com/lumen-engineering-blog">Lumen Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>