<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eli Orzitzer on Medium]]></title>
        <description><![CDATA[Stories by Eli Orzitzer on Medium]]></description>
        <link>https://medium.com/@e_orz?source=rss-19a46dfe85a6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*GM2-Nz0FEJ4Qm6zJ.</url>
            <title>Stories by Eli Orzitzer on Medium</title>
            <link>https://medium.com/@e_orz?source=rss-19a46dfe85a6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:11:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@e_orz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Learned to Stop Worrying and Love the Bash]]></title>
            <link>https://medium.com/@e_orz/how-i-learned-to-stop-worrying-and-love-the-bash-8ff2ab6c59b2?source=rss-19a46dfe85a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ff2ab6c59b2</guid>
            <category><![CDATA[gzip]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[bash]]></category>
            <dc:creator><![CDATA[Eli Orzitzer]]></dc:creator>
            <pubDate>Sun, 10 Mar 2024 07:19:31 GMT</pubDate>
            <atom:updated>2024-03-13T11:38:37.692Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong><em>How I Learned to Stop Worrying and Love Bash</em></strong></h3><h4>Efficient Streaming of TBs of data a day using AWS Lambda and Bash</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*exo0yrl0ps7H5cLrry1AgQ.jpeg" /></figure><p>Bash intimidates most developers. It is considered a complicated (eco)system with a steep learning curve.<br>Is it worth it? I am going to show you that learning bash will open a new world of possibilities with an endless collection of tools you wouldn’t have otherwise.<br>This is a story of a real case in which using Bash is actually <strong>simpler</strong> than any other option.<br>And perhaps, who knows, one day you will find that in your case it is the optimal solution as well.</p><h4>The Use Case</h4><p>The task is quite simple. Get files dropped to AWS S3 by the clients, split them to small equally sized splits and then put them in an internal bucket. Those splits will be processed later in a fair share manner (each split is treated the same, hence a big file won’t starve the small ones).</p><p>The HLD design is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R7yMIhuHMtrUGXJsG4I5Og.jpeg" /></figure><h4>The Original Design</h4><p>Files are not arriving all the time so having a machine that will run constantly, wait for files, do the splits and write them, is a waste of resources (and money). AWS Lambda is a function-as-a-service service. The decision was made to use AWS Lambda.</p><p>Some files are big and some are small (up to 30 GB). Some are compressed and some not. The solution must support all cases.</p><p>A few (wrong) assumptions were made:</p><ol><li>Getting the whole file from S3 to some storage, then decompressing it “locally” and then putting it back to S3 cannot be done in a timely manner due to AWS Lambda’s time limitation (up to 15 minutes)</li><li>When decompressing a file, all the decompressed data must be stored in a random access file system. You cannot work with the data “on the fly”</li><li>S3 supports accessing the middle of a file, but that cannot work for compressed files. There is no way to know when to put the break and also decompressing cannot be done from the middle (at least with the common GZIP format, and we cannot control the format sent by our partners)</li></ol><p>These assumptions led to the following design:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A1WJuAGAnwrXJYZ8ahYOfA.jpeg" /><figcaption>Original components design</figcaption></figure><p>The flow when getting a file is (numbers correspond to the number in the diagram):</p><ol><li>If it’s small enough → write it directly to the target S3 location</li><li>If it’s big and not compressed → for each desired split, spawn a new Lambda that will take the relevant part and write it to the target S3 location</li><li>If it’s big and compressed, start the unzipping process that in the end will put the uncompressed result in a special location. The big uncompressed file will now go back to the queue and will be processed as no. 2</li></ol><p>The unzipper part was actually an AWS Step Function that broke the whole process into several pieces. Lambda’s ephemeral storage is very limited and couldn’t be used for this purpose. Therefore, intermediate storage used was EFS (AWS’s Elastic File System that is practically unlimited in capacity).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*Yx5f3F5khIpjTiQ-RlgrBA.png" /><figcaption>original unzipper design</figcaption></figure><h4>The Breakdown</h4><p>Complicated as it is (100s of lines of code!), it actually worked. Until one day we were amazed to see this (We have a mechanism of reprocessing):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oCCNWqXQFsoyFfbn1ZU91A.png" /></figure><p>In late August 2023, something changed. Either the files became larger, or too many files arrived simultaneously, but EFS couldn’t process them quickly enough. As a result, the Lambda function timed out, and we lost those files.</p><h4>The Initial Fix — Bash Stream Instead of EFS</h4><p>We didn’t want stop using AWS Lambda but something had to be done. The original assumptions were reassessed and a study was conducted. These are the results:</p><ol><li>The most common format, GZIP, is streamable. When reading the file from the beginning, the data can be used even without having the whole file.</li><li>Which means, you can use the data without writing it to an intermediate storage.</li><li>The original Unzipper used the AWS CLI which is CPU intensive since it’s written in Python. Switching to native implementation (rclone, written in Go) used 1/3 of the CPU power. It’s being used twice (download and upload) so the benefit doubles itself.</li><li>We looked for a faster unzipping tool that will also support all compression formats (7zzs, written in C). We also took the latest version and not the one bundled in the Linux distribution (yes, it performed a bit better).</li></ol><p>These steps alone proved sufficient to address the issue. The entire unzipping process was streamlined into a single Bash command, utilizing the Bash pipeline to seamlessly stream the file from rclone (reading from S3), through 7z for decompression, and then back to rclone to upload it to S3 again. (Details have been omitted for brevity, but you can ask ChatGPT for an explanation.)</p><pre>rclone --config &quot;&quot;$LAMBDA_RUNTIME_DIR&quot;/rclone.conf&quot; cat s3:$1 | 7zzs x -si -tgzip -so | rclone --config &quot;&quot;$LAMBDA_RUNTIME_DIR&quot;/rclone.conf&quot; rcat --s3-chunk-size 10Mi s3:$2</pre><h3>The Full-Blown Solution — One Liner Bash Instead of Several Components</h3><p>That only solved the decompressing part. Can we do better? Can we remove the whole complicated pipeline and components? Can it run faster?</p><p>Let’s recall the goal we have. Take a big (5GB compressed, 30GB uncompressed) file and write it in chunks of 200,000 lines each. We already have a one-liner that writes the whole uncompressed file, but not chunked. The chunking is done in a different component. Furthermore, the chunks are not compressed which costs money for the networking and the storage. What if instead of writing the chunks uncompressed we will compress them? Can it be done using the same one-liner?</p><h4>The Compression Algorithm</h4><p>Several compression algorithms were tested; some were not fast enough, while others were limited to using only one CPU. Additionally, some algorithms required random access to the data, which was not feasible for us as we were streaming the data. Zstandard, a fast compression algorithm developed by Facebook, proved to be exactly what we needed. Although it produces files about 20% larger than those compressed with GZIP (yet still significantly smaller than uncompressed files), it offers high efficiency and can utilize multiple CPU cores.</p><p>Using some more of Bash magic to do the splitting, the final design is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QdjKtmdmWQf-e9g3BVmOg.jpeg" /><figcaption>the final design</figcaption></figure><p>Looks familiar? The final design is almost the HLD at the top. This is the best it can get having the low level design almost identical to the high level one. The core of the “ProcessingLambda” Lambda is the one-liner below. This command enriches the functionality of its predecessor by integrating the ‘split’ and ‘zstd’ tools for efficient splitting and compression.</p><pre>rclone --config &quot;$LAMBDA_RUNTIME_DIR/rclone.conf&quot; cat s3:$1 | 7zzs x -si -tgzip -so | split -l 200000 -a 4 -d --filter &#39;zstd -T2 | rclone --config &quot;$LAMBDA_RUNTIME_DIR/rclone.conf&quot; rcat s3:$2/$FILE.tsv.zstd &amp;&amp; echo $FILE.tsv.zstd &gt;&gt; /tmp/success.txt&#39; - $1_split</pre><p>This line might look complicated (try ChatGPT for an explanation) especially for one who is not used to Bash. But think of it, it’s instead of several components (100s of lines spread across several components), error handling in each of them, deployment etc. Ah! it’s also much faster. CPU is much faster than networking. Having only 1/6 of the data to send over the wire is way cheaper. Not to mention the storage price which is now a 1/6 of the previous one.</p><p>Another bonus is the simplicity of having less moving parts in the system. In our case, due to the timeout bug, the EFS wasn’t cleaned properly. It got bigger and cost us about 20,000$ a year. The new design doesn’t store any temporary intermediate files which might be left behind in case of failures.</p><h4>Takeaways</h4><ol><li>Challenge again what is considered to be “known”. (In the Lambda case the infrastructure changed and it can run for 15 minutes with 6 CPUs (previously it was only 5 minutes). What is considered impossible might become possible.)</li><li>Initial assumptions may be wrong. (The GZIP format is indeed streamable. You cannot check all your assumptions all the time but doing it in specific cases might be beneficial.)</li><li>Don’t be afraid to use “unorthodox” techniques. It might be much simpler. Bash is awesome!</li><li>The DRY principle applies not only to coding practices but also to the tooling you use. For instance, there are readily available tools designed for downloading and uploading from AWS S3, as well as for decompressing and compressing files.</li><li>When using a technology try to use it as simple as you can. (There is no need for fancy streaming libraries, a simple Bash pipe might work just as well.)</li></ol><h3>Summary</h3><p>The task definition was straightforward: take files and split them into equally sized segments. At first, it appeared that solving this would necessitate a complex solution. However, ultimately, the solution proved to be (almost) as straightforward as the task definition itself, thanks to the capabilities of the Bash ecosystem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ff2ab6c59b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>