<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eleni Kontou on Medium]]></title>
        <description><![CDATA[Stories by Eleni Kontou on Medium]]></description>
        <link>https://medium.com/@e.pkontou?source=rss-51243ce810d4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Eleni Kontou on Medium</title>
            <link>https://medium.com/@e.pkontou?source=rss-51243ce810d4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:53:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@e.pkontou/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Small Files Problem on S3]]></title>
            <link>https://medium.com/@e.pkontou/small-files-problem-on-s3-5a5ec7f19d0a?source=rss-51243ce810d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a5ec7f19d0a</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[json]]></category>
            <category><![CDATA[small-files]]></category>
            <dc:creator><![CDATA[Eleni Kontou]]></dc:creator>
            <pubDate>Wed, 06 Sep 2023 09:13:29 GMT</pubDate>
            <atom:updated>2023-09-06T09:13:29.103Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Small JSON Files Challenge on S3</h3><h4>Read and process numerous small JSON s3 files</h4><h3>Introduction</h3><p>In the era of the rapid technological change and the dominance of big data, various digital entities including servers, websites, applications generate an overwhelming volume of events/logs. Imagine thousands, millions, or even billions of logs generated every second, each stored in formats like CSV, XML, or JSON. Reading and managing such numerous minuscule files can be a nightmare for companies, given the substantial overhead it generates and the extended time required to complete. This constitutes a significant challenge known as the Small Files Problem.</p><p>This post delves into a specific facet of this challenge, focusing on having a lot of small JSON files on AWS S3. Since processing such a large number of small files is time-consuming, we will suggest an alternative approach to store such data, aiming to streamline our workflow.</p><h3>Small Files Problem</h3><p>The Small File Problem is the issue that arises when dealing with data stored in numerous small files. In the context of text files like CSV, XML, or JSON, a small file refers to a file that is a few KB or a few MB in size. But why can small files be a problem? Processing numerous files can significantly impact performance. Here are some reasons:</p><ul><li><strong>Opening, reading, and closing files </strong>requires time due to the overhead associated with this file operation. Reading small files might take milliseconds to be executed, which seems negligible on its own. However, dealing with thousands or even millions of files can escalate the time required to minutes or even hours. On the other hand, when you are dealing with a single large file, these overheads are incurred only once.</li><li>Each file has associated <strong>metadata</strong>, such as timestamps, that accompanies it. Storing many files translates to a significant amount of metadata, which in turn increases memory consumption, potentially straining the system’s resources.</li><li><strong>Big Data engines</strong>, like Spark or Hadoop, work perfectly for large files but <strong>struggle with handling a large number of small files</strong>. While these systems are designed for parallel data processing, when dealing with numerous small files, the overhead of managing individual files becomes significant, resulting in limit parallel processing opportunities and decreased performance.</li></ul><h3>Small JSON files on S3</h3><p>Amazon S3 is a cloud storage service designed to store and retrieve vast amounts of diverse data. It is widely used for data backup, archives, and creation of data lakes for analytical purposes.</p><p>At <a href="https://www.kpler.com/">Kpler</a>, we heavily utilize S3. For example, the <a href="https://www.marinetraffic.com/en/intransit">Visibility for Container Solution</a> that I am currently working on keeps a huge amount of data on S3 which are mainly used for analytics and machine learning. The data are in different formats and they are stored based on the medallion architecture.</p><p>This post focuses on the JSON files generated by our app, which serve as a valuable resource for our Data Science and Analytics team.</p><h4>The challenge</h4><p>One of our applications generates approximately half a million logs daily, and this number grows as the number of users increases. Each log is stored in a JSON file. Its size varies from bytes to kilobytes depending on the context. And that’s how we found ourselves in front of the small files problem.</p><p>These JSON files constitute our raw data which are then transformed based on the needs of the Data Science Team. The team is working on several Analytics and ML projects which involve processing large volume of historical data. Due to the diverse nature of these projects and the continuous additions to them, the requirements vary. Consequently, the JSON files must remain accessible at any time, ready for a new ETL process to consume them in order to serve the new needs. So, it is evident that the need for rapid reading and processing of numerous small files is imperative.</p><h4>The solution</h4><p>There are several approaches to tackle this problem. The one described here combines data aggregation with data compression.</p><ul><li><strong>Data Aggregation:</strong> Combining the data included in the small JSON files into larger files. Instead of storing just one log in each file, consider accumulating more data/logs per file, utilizing time-based intervals (e.g. hourly, daily) or size criteria (e.g. file size in MB or batch size, i.e. number of logs). By changing this data writing pattern and aggregating data before storing on s3 (or creating a new layer with larger files) can reduce the overhead of metadata and improve I/O efficiency.</li><li><strong>Data Compression: </strong>Compressing the files can reduce data transfer overhead and total storage size. Combining compression with a columnar storage format can improve data processing performance as well. For example, writing to Parquet files using a Snappy compression can decrease storage costs and speeds up data access and processing, as only relevant columns need to be accessed from disk.</li></ul><p>So, based on the aforementioned solution, we created an additional data layer between the raw JSON data and the data used for analytics. This layer incorporates Parquet files, which serve to aggregate and compress the JSON data. More specifically, the data layers are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*idmPJrFNE-y0dqaPQ54GFg.png" /><figcaption>Data Layer Architecture</figcaption></figure><ul><li>Layer 1: Application data is stored on S3 in the form of JSON files, with each file containing a single app log. These files are organized within daily S3 prefixes, primarily intended for data backup and archival of our historical data.</li><li>Layer 2: JSON data is then aggregated and compressed into Parquet files. Each Parquet file consists of 10K JSON data entries, corresponding to approximately 25ΜΒ. These files are also stored in daily s3 suffixes, but they are smaller in number and larger in size. This layer is used for easily retrieving the historical data in order to produce data models included in the next layer.</li><li>Layer 3: This layer consists of different data models, either stored in files or databases, all intended for utilization in Analytics and Data Science projects. Parquet files play a pivotal role in creating those data models efficiently.</li></ul><p>Certainly, the volume of Parquet data remains substantial and is not universally suited to all projects. For example, in a specific project, there is no need for full data granularity, so an ETL process which produces a reduced dataset has been created. This is achieved by filtering the Parquet data, subsampling, and keeping only the essential fields. This process generates a single CSV file per day, tailored for analytical purposes.</p><p>Regarding the number of JSON data in the Parquet files, writing 10K logs per file, rather than consolidating them into a single large Parquet file, offers practical advantages. It allows for parallel processing, as many tools can process files in parallel distributing the workload efficiently. Additionally, it ensures scalability, as the number of daily logs increases, without affecting the file size. This prevents the limitations and complexities associated with very large files.</p><p>Speaking in numbers, writing 10K JSON files into a single Parquet file using Snappy compression can reduce the reading time by 30 times, as presented below.</p><pre>import io<br>import boto3<br>import botocore<br>import pyarrow as pa<br>import pandas as pd<br>from concurrent.futures import ThreadPoolExecutor<br><br>class AwsS3:<br><br>    def __init__(self, s3_bucket, max_pool_con):<br>        self.bucket = s3_bucket<br>        self.s3 = boto3.client(&quot;s3&quot;, config=botocore.client.Config(max_pool_connections=max_pool_con))<br>    <br>    def read_json(self, file):<br>        &quot;&quot;&quot;Read json from s3&quot;&quot;&quot;<br>        obj = self.s3.get_object(Bucket=self.bucket, Key=file)<br>        data = obj[&#39;Body&#39;].read().decode(&#39;utf-8&#39;)<br>        return data<br>    <br>    def read_parquet(self, file):<br>        &quot;&quot;&quot;Read parquet from s3&quot;&quot;&quot;<br>        response = self.s3.get_object(Bucket=self.bucket, Key=file)<br>        parquet_file = io.BytesIO(response[&#39;Body&#39;].read())<br>        df = pd.read_parquet(parquet_file, engine=&#39;pyarrow&#39;) <br>        return df<br><br># Connect to AWS<br>bucket_name = &#39;YOUR_BUCKET&#39;<br>max_pool_con = 64<br>aws = AwsS3(bucket_name, max_pool_con)<br><br># Read json files<br># files = list with 10K filenames<br>with ThreadPoolExecutor(64) as pool:<br>     # Process JSON files concurrently using multithreading<br>    data = list(pool.map(aws.read_json, files))<br><br># Read parquet file<br># file = &#39;YOUR_PARQUET_S3_FILENAME&#39;<br>data = aws.read_parquet(file)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*m1Ch5XlMtSBfktH0AhgZEg.png" /><figcaption>Execution time of reading 10K json files(using Multithreading) vs 1 parquet file containing 10K json data</figcaption></figure><h4>Access JSON data</h4><p>When aggregating the data into one file, it’s important to take into account how the data is written. The data values should be written in correct data types in order to be easier to access the data after reading.</p><p>Our raw files contain columns with JSON nested data. The structure of the JSON is not the same in each file. Their context can change after new product releases. Consequently, defining a specific structure for our JSON columns is not feasible.</p><p>Many tools or packages convert JSON data columns into string type during the writing process. This is faster and easier, as there is no need to define the JSON structure. Nevertheless, it can result in additional overhead when subsequently attempting to convert the string column back to JSON to access the data.</p><p>In the following example, we assume that the <em>message </em>column in our data contains JSON nested data. From this data, we created 2 parquet files for testing, one applying the json.dumps() function to the <em>message</em> column when writing the file, and another one converting the <em>message </em>column into the data type string before writing the file. The code and the graph below presents the time required to read and access the JSON data.</p><pre>import io<br>import json<br>import ast<br>import boto3<br>import botocore<br>import pyarrow as pa<br>import pandas as pd<br>from concurrent.futures import ThreadPoolExecutor<br><br>class AwsS3:<br><br>    def __init__(self, s3_bucket, max_pool_con):<br>        self.bucket = s3_bucket<br>        self.s3 = boto3.client(&quot;s3&quot;, config=botocore.client.Config(max_pool_connections=max_pool_con))<br>    <br>    def read_json(self, file):<br>        &quot;&quot;&quot;Read json from s3&quot;&quot;&quot;<br>        obj = self.s3.get_object(Bucket=self.bucket, Key=file)<br>        data = obj[&#39;Body&#39;].read().decode(&#39;utf-8&#39;)<br>        data = json.loads(data)<br>        return data<br>    <br>    def read_parquet(self, file):<br>        &quot;&quot;&quot;Read parquet from s3&quot;&quot;&quot;<br>        response = self.s3.get_object(Bucket=self.bucket, Key=file)<br>        parquet_file = io.BytesIO(response[&#39;Body&#39;].read())<br>        df = pd.read_parquet(parquet_file, engine=&#39;pyarrow&#39;) <br>        df[&#39;message&#39;] = df[&#39;message&#39;].apply(lambda x: json.loads(x))<br>        return df<br><br>    def read_parquet_str(self,file):<br>        response = self.s3.get_object(Bucket=self.bucket, Key=file)<br>        parquet_file = io.BytesIO(response[&#39;Body&#39;].read())<br>        df = pd.read_parquet(parquet_file, engine=&#39;pyarrow&#39;) <br>        df[&#39;message&#39;] = df[&#39;message&#39;].apply(lambda x: ast.literal_eval(x))<br>        return df<br><br># Connect to AWS<br>bucket_name = &#39;YOUR_BUCKET&#39;<br>max_pool_con = 64<br>aws = AwsS3(bucket_name, max_pool_con)<br><br><br># Read json files &amp; access JSON data<br># files = list with 10K filenames<br>with ThreadPoolExecutor(64) as pool:<br>     # Process JSON files concurrently using multithreading<br>    dataframes = list(pool.map(aws.read_json, files)) <br><br># Read the parquet file where &#39;message&#39; column includes JSON data<br># file = &#39;YOUR_PARQUET_S3_FILENAME&#39;<br>data = aws.read_parquet(file)<br><br># Read the parquet file where &#39;message&#39; column is string<br># file2 = &#39;YOUR_PARQUET_S3_FILENAME2&#39;<br>data = aws.read_parquet_str(file2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*k6pdPINt8LFPUrGOt4XqLA.png" /><figcaption>Execution time of reading 10K json files(using Multithreading) vs 1 parquet file containing 10K json data (time for data type conversion is included in the total execution time)</figcaption></figure><ul><li>Blue(10K json files): Read 10K raw JSON files and convert the JSON string data to JSON</li><li>Green(1 parquet file): Read the parquet file(10K data included) where the message column was stored using the json.dumps() function, and convert JSON string data to JSON</li><li>Orange(1 parquet file with string data): Read the parquet file(10K data included) where the message column had data type string, and convert JSON string data to JSON</li></ul><p>It’s evident that the choice of data types during file writing significantly impacts the performance of accessing the data. Therefore, if programmatically implementing this solution, it is advisable to avoid converting JSON data into strings (str) while writing the data into a file, as this would add additional load afterwards , when processing the data. Consider using json.dumps() when writing JSON data to a file, and then json.loads() when reading such data.</p><h3>Conclusion</h3><p>A lot of small files can be an issue when having to read and process them. A simple way to save time and storage is to aggregate them into larger files and change the format to speed up the data access. This simple solution provides more flexibility if many complex transformations needed after the reading process. Of course, services like AWS Glue offer simplified S3 file handling and are being actively explored by our team as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a5ec7f19d0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>