From Flink to Lambda
IoT devices stream data but the value of it is only determined by the processing pipeline after it.
There was a project a few years back that consumed IoT devices data using AWS Glue to perform scheduled ETL jobs on the data stored in S3. The situation was not great, there was barely any data to process, the update rate was once per hour and yet the bill for the Glue Spark ETL represented 90% of the cloud infrastructure operational costs!
Mastering Flink
Luckily, one year before, AWS had introduced support for Apache Flink in Kinesis Analytics which made our decision clear at that point, ditch a complex multistep pipeline in favour of a real time processing solution that reduced the costs to a tenth (at the given data volume).
Setting up the Flink application was a very interesting task. Our data was not windowed by time which meant we had to develop a custom windowing solutions, which was a nice way to dive into the documentation. To make it more interesting the data stream was not continuous and the windowing had to take that into consideration which also meant developing the custom watermarking, out of order management and timeout logic for our window system. It was a very enjoyable adventure to work in such a solution and it ended working so well that we ditched it one year later.
Abandoning Flink
This might sound crazy, right? The thing is, AWS had updated Lambda with tumbling windows, and we were so caught in the development inertia that we did not hesitate to keep iterating into the new solution, fully serverless. We knew and understood the flow precisely and only because of everything we had learned with Flink were we able to immediately migrate to lambda. I must say that the requirements had slightly changed; the first releases had shown us that Flink was extremely good doing its real time job but the batch re-processing was painful (for our application). Our expectation was a one time setup that would serve to process the data as it was being collected but the final users needed the option to issue corrections in certain processing parameters and then re-process the whole batch. The logic to connect Flink’s batch processing to our API and wait for the results was good but not responsive enough and it was at that particular point in the iteration cycle (when we understood what users wanted now that real time processing was actually real time) that AWS brought the lambda tumbling window.
The final architecture
The tumbling window allowed us to port the two pipelines running in our Flink processor (a windowed in time stream and a second one windowed in a custom feature) without breaking a sweat. The improvement were immediate:
- Simplified structure: No more Kinesis analytics, pipeline to compile java or bucket to host it
- 100x cost reduction: Compared with the initial Glue implementation
- Shared Type Definitions: Writing our Lambda functions in TypeScript allowed us to share type libraries directly between our backend processing and front-end Web application.
Lessons learned
This obviously doesn’t have to be the right solution for everyone but it was for that particular project. Flink has proven useful many more times and there is no replacement in what it can do and how it does it.
Get Jorge Hermoso’s stories in your inbox
Join Medium for free to get updates from this writer.
Iterative development means not to be attached to what you have created and instead use it as a stepping stone for the next improvement. I think every developer will agree that looking at old code is rarely satisfying and, don’t get me wrong, I’m not advocating for some sort of perfectionism that lasts forever never delivering the product. On the contrary, the product needs to get out of the door as soon as possible and the development team needs to have the trust from the product owners to take (controlled) risks. Then and only then will the team be at the right place at the right time, ready to take advantage of the new technologies continuously being released.
