<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rajagopal ParthaSarathi on Medium]]></title>
        <description><![CDATA[Stories by Rajagopal ParthaSarathi on Medium]]></description>
        <link>https://medium.com/@rajagopalparthasarathi?source=rss-ba2be13f769d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*oUxm28ct5uTKJyG4.</url>
            <title>Stories by Rajagopal ParthaSarathi on Medium</title>
            <link>https://medium.com/@rajagopalparthasarathi?source=rss-ba2be13f769d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 20:58:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rajagopalparthasarathi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evolution of data platform — Part 3]]></title>
            <link>https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-3-2f6ea2406439?source=rss-ba2be13f769d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f6ea2406439</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Rajagopal ParthaSarathi]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 09:28:09 GMT</pubDate>
            <atom:updated>2023-04-11T09:28:09.314Z</atom:updated>
            <content:encoded><![CDATA[<h3>Evolution of data platform — Part 3</h3><h3>Part 3: Beyond Analytics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RGjXI0ixgNvgoTaRNh7iVw.jpeg" /><figcaption>Data Lake</figcaption></figure><p>This is part 3 of the series <a href="https://psrlog.com/series/edp">Evolution of the data platform</a>. I would recommend reading <a href="https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-1-f9dd60f96791?source=your_stories_page-------------------------------------">parts 1</a> &amp; <a href="https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-2-7187d6280b19?source=your_stories_page-------------------------------------">2</a> before this post.</p><p>Ajar and the team established a good platform for analytics use cases. The final part was closing the loop of <strong>value delivery.</strong></p><h3>Reporting</h3><p>The initial use case of the platform was to generate valuable insights and report the KPI year on year. Write had already a thriving Microsoft suite and it was a no-brainer to use the reporting solution — PowerBI. The next step was to identify how to integrate Athena with PowerBI. This was already a solved problem and there was good material on the internet available. Ajar and the team used a PowerBI Gateway in the EC2 machine and installed the Athena ODBC driver for data connectivity to Athena.</p><p>This started as the first slice of value for the customer. Write was excited to see an initial set of KPIs based on user activity in PowerBI dashboards. As the reports and dashboards got built, Ajar and the team ran into a few problems.</p><p>Athena ODBC driver did not support interactive querying. The data had to be fully loaded into PowerBI and some of the business logic leaked into PowerBI. The loading of full data was always not feasible as PowerBI reports had huge performance problems.</p><p>To mitigate this Ajar and the team introduced an interim reporting layer that had pre-computations done on the original dataset. This is analogous to the gold zone of medallion architecture. This also led to moving the business logic upstream as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/0*9wJCKBUxpYrgRsDB.png" /></figure><p>But pre-computation is not always possible with changing requests and demands. At this juncture, the team was researching something like the metrics layer(<a href="https://metriql.com/">Metriql</a>). Any input on this particular area is welcome</p><h3>Operational Dashboards</h3><p>Currently, business teams and Data scientists had wealth of information to curate insights. This led to the possibility of creating an external data product. I.e using data not only to make decisions but give data back to end customers to make their decisions. This meant the team had to create data applications that would respond to rest API calls rather than reports</p><p>To facilitate this requirement, the team designed the following approach</p><ol><li>ETL to read data from the lake</li><li>Transform and enrich as needed</li><li>Feed the data into a document store like DynamoDB</li><li>Run a lambda on top of DynamoDB which would serve the insight.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*gQ9GQ9szupJV-dQk.png" /></figure><p>This would be consumed by external-facing consumers. The refresh cycle happened periodically.</p><h3>Streaming Architecture</h3><p>The architecture was batch oriented though events were ingested in real-time. As in every business setting, Write needed a real-time use case like Twitter feeds. Ajar and the team once again rolled their hands to figure out a solution to the problem. The team chose AWS Kinesis and Glue streaming for real-time use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/0*mYevfxUoXIn9srfU.png" /></figure><p>The team ran a micro-batch every 1 minute to get as real-time as possible. The team also explored Kinesis Firehose for a few use cases. But transformation logic was intensive leading to the above architecture.</p><h3>Takeaways</h3><ul><li>Use Kinesis if you don’t need low-level configurability</li><li>Athena is good for interactive querying but for reporting use cases presto/Redshift spectrum is better</li><li>Reevaluate tech choices based on the evolving ecosystem</li></ul><h3>References</h3><ol><li><a href="https://www.databricks.com/glossary/medallion-architecture">https://www.databricks.com/glossary/medallion-architecture</a></li><li><a href="https://docs.aws.amazon.com/athena/latest/ug/connect-with-odbc.html">https://docs.aws.amazon.com/athena/latest/ug/connect-with-odbc.html</a></li><li><a href="https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-personal-mode">https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-personal-mode</a></li></ol><p>Ajar has moved on to his next journey with the satisfaction of playing a good role in platform and value creation. Ajar is grateful for the opportunity and also to his wonderful journey companions(colleagues to friends). Though Ajar has moved on, the journey for Write continues into machine learning use cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f6ea2406439" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-3-2f6ea2406439">Evolution of data platform — Part 3</a> was originally published in <a href="https://medium.com/plumbersofdatascience">Plumbers Of Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Insights with MindsDB]]></title>
            <link>https://medium.com/@rajagopalparthasarathi/data-insights-with-mindsdb-20bfc02f1aac?source=rss-ba2be13f769d------2</link>
            <guid isPermaLink="false">https://medium.com/p/20bfc02f1aac</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Rajagopal ParthaSarathi]]></dc:creator>
            <pubDate>Sat, 17 Dec 2022 13:20:31 GMT</pubDate>
            <atom:updated>2022-12-18T16:11:52.628Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Insights with MindsDB</p><p>Recently I was discussing with a colleague that SQL is a pervasive part of a data stack. Their counterargument was SQL has not yet penetrated the ML world. While I was researching that idea, I encountered MindsDB, an AutoML platform that integrates SQL like interface on top of ML algorithms</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*L7FVdd7ZP0Lqb0PaTuxBcA.png" /><figcaption>MindsDB Logo</figcaption></figure><h3>Introduction</h3><p>MindsDB brings machine learning into databases by employing the concept of AI Tables. It provides connectors to integrate with existing databases. These integrated datasets are used to generate predictions by executing SQL Queries.</p><p>But any seasoned data scientist will assure you that before training the model, the first step is understanding the data. Luckily MindsDB also provides a data analysis tool as part of its product.</p><p>In this blog, we will look at the data insights component of MindsDB by exploring the Manufacturing Industries dataset.</p><h3>Running MindsDB</h3><p>MindsDB can be installed either locally or using a cloud offering. Cloud offering allows a free trial of 30 days. For this article, we will run mindsdb using docker in our local environment. To get started with mindsDB, run the below command</p><pre>docker run -p 47334:47334 -p 47335:47335 mindsdb/mindsdb</pre><p>This runs mindsdb on api port 47334. The Mysql instance which holds the database objects is running on port 47335. Once the docker container indicates the instance has started, visit http://localhost:47334/ to open the console. This opens the below page showing a SQL interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5EzBsoMwRaAf4kXJ.png" /><figcaption>Add Data</figcaption></figure><p>You can run the following command to look at the available data sources. show databases</p><h3>Importing Dataset</h3><p>Let’s start by importing the manufacturing dataset from <a href="https://www.kaggle.com/datasets/eishahassan/large-scale-manufacturing-industries-production">Kaggle</a>. Once the CSV is downloaded into our local machine, click on <strong>Add Data</strong> as highlighted in the previous section screenshot. This opens a page showing all possible data import mechanisms using connectors which are illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r7wkyRgpPyLh3w1R.png" /></figure><p>We will use file import and select the appropriate file from your downloaded folder. We should also provide a suitable name for the imported data so we can access the CSV dataset for further processing. Your screen should look like this during the import</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R92jWAlUMkX9LY00.png" /></figure><p>Click on <strong><em>save and continue</em></strong>. We are taken back to the console with convenient SQL commands showing how to access the data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iY2Sv8nCtoPHWptY.png" /></figure><h3>Exploring Data</h3><p>In this section, we will understand what the data is all about. We will look at a sample of the data by running the command SELECT * FROM files.large_scale_industries LIMIT 10;</p><p>This will show us 10 rows from the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SMrToWpBTqkKkUQB.png" /></figure><p>From the screenshot, we can see the following columns are present in the dataset</p><ul><li>Month-year</li><li>CPI</li><li>Interest Rates</li><li>Exchange Rates</li><li>Product</li><li>Unit of Quantity</li><li>Production</li></ul><p>Now if we run the query without the LIMIT, it will bring us the entire dataset</p><h3>Data Insights</h3><p>In the bottom part of the console(results pane), you will see <strong><em>the Data Insights</em></strong> label. Once a white dot appears on the green label, click on it to view the data analysis results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kN_G-UY3VQz3ra5O.png" /></figure><p>To perform a complete analysis of your data, you can either go to a full-screen mode or stay in a pane mode and click on the Full Data Analysis button.</p><p>Once the analysis is complete, you can see the distribution of the various columns. I have provided below output for two columns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XixbkXAIXj5y3hFI.png" /></figure><p>This is for the column <strong>CPI</strong>. The X-axis represents the actual value while Y-axis represents the count of the particular data point. The column is also flagged as Potential Bias which indicates that the dataset is not spread evenly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2z3CH_JvexLmRCG0.png" /></figure><p>This represents another column <strong>Exchange Rates.</strong> The column data type is indicated at the top as Float and this shows a spread similar to the other column. Hovering over the bar shows the count of values.</p><p>This helps analysts and scientists understand the data before proceeding with further tasks of prediction and classification.</p><p>MindsDB has potential. I would encourage fellow practitioners to give a spin. The community is also active and is responding to GitHub issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20bfc02f1aac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of data platform — Part 2]]></title>
            <link>https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-2-7187d6280b19?source=rss-ba2be13f769d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7187d6280b19</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Rajagopal ParthaSarathi]]></dc:creator>
            <pubDate>Sat, 09 Jul 2022 16:16:17 GMT</pubDate>
            <atom:updated>2022-07-09T16:16:17.475Z</atom:updated>
            <content:encoded><![CDATA[<h3>Evolution of data platform — Part 2</h3><h4>Part 2: Performance, Catalog and Access Control</h4><p>This is part 2 of the series <a href="https://psrlog.com/series/edp">Evolution of data platform</a>. I would recommend reading <a href="https://psrlog.com/evolution-of-data-platform-part-1">part 1</a> before this post</p><figure><img alt="Physical Lake representing data lake" src="https://cdn-images-1.medium.com/max/1024/0*uX8eBFjfE7diIeWk" /><figcaption>Photo by <a href="https://unsplash.com/@dvlw?utm_source=medium&amp;utm_medium=referral">dirk von loen-wagner</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*daiIz3pxJwFcxCL6" /><figcaption>Ingestion Pipeline</figcaption></figure><h3>Performance is a long game</h3><p>Ajar and team were basking in the success of the above architecture. After few weeks, the queries were again facing performance issues. Kriti and Kumar already found the reason for it. There was a daily batch which was generating the parquet file under the partition based on access pattern. This partition is a data element very specific to the business. But there was huge number of small files one for each day under the partitioned folder. The day based partitioning would have resulted in the same small file as daily volume was only few Megabytes</p><p>So Ajar and team looked at various file formats and databases to understand how this problem was generally solved. Ajar found a reference from Designing Data Intensive Applications by Martin Klepmann. (Refer <a href="https://psrlog.com/31-oltp-storage-and-retrieval-ddia">summary</a> from here). The answer was compaction of these small files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LFFW8R-Bsin6ARtE" /><figcaption>Compaction Pipeline</figcaption></figure><p>A new batch job was introduced at end of the day to compact these small parquet files into a bigger parquet file(128 MB in size). Once a file crosses 128MB in size, rest of the files are grouped to form the next file. This job operated on per partition basis. This ensured a good performance of queries on ongoing basis.</p><blockquote><em>Do we require all of this engineering complexity and maintenance ?</em></blockquote><h3>Friendly File formats</h3><p>In this approach, there is a lot of handcrafting with the compaction job at file level. Consider the problem of correcting a record in data lake or implementing RTBF(Right to be Forgotten) for a customer. This would entail the following steps</p><ul><li>Find the parquet file containing the records</li><li>Load the entire file and change/delete the required records</li><li>Write the new file back</li><li>Ensure no other job modifies the file during this time</li></ul><p>Should the Data Engineers focus on all of this heavy lifting ? This is in addition to all the business related pipelines which data team is responsible for building.</p><p>To summarize, the problems are</p><ul><li>Modify few records by key lookup efficiently</li><li>Have a good consistent file size for efficient query performance</li><li>Reduce overall maintenance effort</li></ul><p>Let us see few alternatives to the parquet file format which addresses these problems</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*x1uPXY52_7DIKti_9N_LkQ.png" /></figure><p>These file formats ensure development team focus on business concerns with reduced maintenance effort. Ajar would definitely pick one of these formats for projects beginning today.</p><blockquote><em>How do we get more people to use the data ?</em></blockquote><h3>Catalog and Access</h3><p>Team had grown to 2 developer pairs with a dedicated Quality Analyst and Business Analyst. So far the folks in team had been wearing many hats but now focused on specific areas. The success of the team also meant that there were growing consumers for this data lake. This also meant that consumers need an efficient way to access the data. We already saw that Athena is being used to access this data but Athena is just query engine. It needs a catalog to understand the information stored in data lake.</p><p>Ajar and team used the native AWS offering of Glue catalog. Glue supports both push based and pull based mechanism for catalog. The team used both approaches for populating the catalog.</p><h4>Pull Based</h4><p>This depends on glue crawlers. Crawlers scan through specified S3 folders and generate the table structure based on the parquet file metadata. It auto creates the partitions based on the folders in S3. Team mainly used it for click stream data as the data dimensions(secondary attributes) varied across each event. This allowed the team to flexibly capture events without restricting the schema.</p><ul><li>✅ Useful for sources with dynamic schema (Auto identify instead of data team intervention)</li><li>✅ Reduces effort as partitions are created automatically</li><li>❗ Corrupt data would create unnecessary partitions and need to be restored with care.</li><li>❗ Schema mismatch can sometimes occur as schema is inferred from file.</li><li>❗ Crawler runs on a schedule and data will not be visible in Athena, till the crawler has added the partitions to the catalog. Crawler schedule determines the freshness of your data on top of the ingestion schedule.</li><li>❗ It is slightly expensive service.</li></ul><p>Team scheduled crawler every week to detect schema changes and add new partitions.</p><h4>Push Based</h4><p>In this mechanism, the schema is defined upfront by running DDL scripts via Athena Query engine. This is suitable for fixed schema sources and evolution needs manual intervention</p><ul><li>✅ Greater control of schema.</li><li>✅ Bad data may make table non usable but can isolate the files and correct the problem quickly.</li><li>✅ Team ran add partition command as part of the ingestion job to keep partitions upto date.(Fresh data as soon as possible)</li><li>❗ To add partitions, needs manual effort.</li><li>❗ Would need manual intervention many times for widely changing schema</li></ul><p>Glue catalog suited the team purpose in their initial journey. The initial consumers were Business Analysts and developers who were comfortable with getting their hands dirty. But the team found few shortcomings as part of their usage in the wild</p><ul><li>It did not have lineage for transformed views. Much needed for data consumers</li><li>Data Discoverability was not great in Glue. Using another service with combination of glue is extra effort(eg: Amazon Kendra or elastic search)</li><li>Out of the box statistics needs to be improved</li><li>Common queries can help consumers a lot . Ability to provide detailed description at table and column level is necessary</li></ul><p>Team is currently looking at something like <a href="https://datahubproject.io/">DataHub</a> for robust capabilities</p><p>The team was so far dealing with non PII(<strong>P</strong>ersonally <strong>I</strong>dentifiable <strong>I</strong>nformation) data. But, a particular source system had personal data. At first, the team debated should it really bring PII data into the platform. The consumer usecase needed a personalization service to send mails based on user’s activity in the platform. Hence team decided to bring in the PII data.</p><p>This presented a problem of restricting access to consumers. PII data should be accessible to email service but not to other consumers. Krithi did research on this problem and suggested AWS <a href="https://aws.amazon.com/lake-formation">Lake formation</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PLAxzS0-kMTqLDtO" /><figcaption>Data Layout</figcaption></figure><p>Lakeformation works in conjuction with Glue Catalog and gives the flexibility to grant access at column level to IAM roles.</p><p>AWS has been doing some serious improvements to lakeformation. This <a href="https://aws.amazon.com/blogs/big-data/part-1-effective-data-lakes-using-aws-lake-formation-part-1-getting-started-with-governed-tables/">article</a> presents how lakeformation brings ACID compliance and row level security to data lake. Lakeformation governed tables is also a good alternative to the lake formats we discussed above.</p><blockquote><em>How do we choose an ETL platform ?</em></blockquote><h3>Glue vs EMR</h3><p>Ajar and team ran most of the workloads in AWS Glue. There was a spirited discussion of using AWS offering of Elastic map reduce. The team decided on AWS Glue against EMR for the following reasons</p><ul><li>Team had few small workloads which were predominantly python shell jobs. Porting them to Spark was unnecessary.</li><li>Having always on EMR cluster meant a fixed cost but it was expensive for few batch workloads(not 100s of jobs). Most of the jobs ran in 5–6 hour interval.</li><li>Provisioning an on demand EMR cluster meant we had to account the cluster provisioning time. EMR cluster also meant doing appropriate sizing and scaling of instances(Can be mitigated by using EKS and EMR). We were only running spark workloads and heavily using other AWS services. Though EMR had good configurability for Big data stack, Glue was satisfactory for team’s use case.</li><li>Glue was serverless and it also had an inbuilt scheduler. Team did not have nested/complex workflows. All this meant Glue was quite adequate for the use case.</li></ul><p>Glue incurs a slightly higher cost than on demand EMR cluster in terms of cpu per hour. But the ease of setup outweigh the cost. EMR is preferrable when volume is high and usecase involves variety of tools in Big data stack.</p><h3>Takeaways</h3><ul><li>Choose appropriate file format for the use case. Recommend use of lake file systems like Delta lake, Hudi or iceberg.</li><li>Data catalog is a key component in a data platform. This will be your gateway for Data Activation across the enterprise.</li><li>Role Based Access Control will definitely creep in your data platform with or without PII data. Lake formation is a robust offering in that regard. Apache ranger has better support if you are using AWS EMR.</li><li>AWS Glue provides a scheduler,catalog and serverless runtime for ETL. Definite recommendation unless your usecase outgrows Glue</li></ul><h3>References</h3><ol><li><a href="https://psrlog.com/31-oltp-storage-and-retrieval-ddia">DDIA Summary</a></li><li><a href="https://lakefs.io/hudi-iceberg-and-delta-lake-data-lake-table-formats-compared/">File formats compared</a></li><li><a href="https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html">Glue Crawlers</a></li><li><a href="https://aws.amazon.com/blogs/big-data/part-1-effective-data-lakes-using-aws-lake-formation-part-1-getting-started-with-governed-tables/">Lake formation</a></li><li><a href="https://www.dataengineeringweekly.com/p/the-curious-case-of-aws-emr-pricing">EMR costs</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7187d6280b19" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-2-7187d6280b19">Evolution of data platform — Part 2</a> was originally published in <a href="https://medium.com/plumbersofdatascience">Plumbers Of Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of data platform — Part 1]]></title>
            <link>https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-1-f9dd60f96791?source=rss-ba2be13f769d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9dd60f96791</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[parquet]]></category>
            <dc:creator><![CDATA[Rajagopal ParthaSarathi]]></dc:creator>
            <pubDate>Tue, 17 May 2022 07:59:02 GMT</pubDate>
            <atom:updated>2022-05-17T07:59:02.082Z</atom:updated>
            <content:encoded><![CDATA[<h3>Evolution of data platform</h3><p>Part 1: How it began and what we learned</p><figure><img alt="Lake Symbolic of Data lake" src="https://cdn-images-1.medium.com/max/1024/0*v_i4RyEwX-3tWevO" /><figcaption>Photo by <a href="https://unsplash.com/@dvlw?utm_source=medium&amp;utm_medium=referral">dirk von loen-wagner</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this post, we will look at how we built a data platform on AWS and our team’s learning while building it. I am writing this in business novel format similar to the <a href="https://www.amazon.com/Goal-Process-Ongoing-Improvement/dp/0884271951">Goal</a>.</p><h3>Introduction</h3><p>The story starts somewhere in December 2019. Client Write is in media business. Write has a terrific visitor facing site which people visit to understand about the business. One fine day, Karin(Product Owner) approached Krish(Tech Lead) with these questions</p><ul><li>Can we measure the impact of UX changes?</li><li>Can we optimize product based on user journey?</li><li>Can we do funnel Analysis to understand the drop off points?</li></ul><p>These lead to a series of initiatives which we will discuss. Krish brainstormed ideas with Ajar a developer in the team. The duo initially started with the plan of using google analytics free version. <a href="https://analytics.google.com/analytics/web/">Google Analytics</a> was pretty cool for few cases but was found gasping for certain advanced needs. In a nutshell, GA</p><ul><li>had Out of the box reporting with support for multiple visuals</li><li>was Easy to configure and no maintenance</li><li>had No access to raw data</li><li>Additional dimensions were paid feature</li></ul><p>Karin slowly evangelized the principles of data led business decisions within Write. This led to more raw data requirements for multiple use cases. Krish proposed an open source tool which can help to collect raw data. Ajar worked on an architecture to ingest this raw data for analytical use cases. Before we dive into the data lake, lets look at the open source tool : <a href="https://matomo.org/">Matomo</a></p><ul><li>had Access to raw data</li><li>was Open source and can run in own infrastructure</li><li>Needs Maintenance effort</li><li>Has limited Out of the box reporting</li><li>is Extensible</li></ul><h3>Data Platform</h3><p>Once Ajar and Krish proposed the data platform architecture, two more devs (Kriti and Kumar) were included to form the team. Ajar became the TL and Krish acted as the Data Analyst. Let’s look at the initial architecture below.</p><figure><img alt="Data Platform Architecture" src="https://cdn-images-1.medium.com/max/1024/1*wBB-z4vZ9b-pyJwNte9cWA.png" /><figcaption>Data Platform Architecture</figcaption></figure><ul><li>Architecture had both realtime and batch ingestion.</li><li>Each source system emits an event whenever a business change occur</li><li>This is fed to AWS SNS service which broadcasts to multiple consumers.</li><li>One of the consumer is data lambda which feeds the data as JSON to S3 bucket</li><li>There was also batch ingestion from few source systems. Jobs run on a periodic basis to fetch the data and put the JSON data in S3 bucket</li></ul><p>Two approaches for data persistence were discussed</p><ul><li>For batch data sources, data was overwritten during every run.</li><li>There are no historical data and the data available is always the latest.</li><li>This was suitable for cases when we don’t require the evolution of data.</li><li>For real time data sources, each event was captured separately and stored.</li><li>This helped in cases when we require historic evolution of state(time series)</li></ul><p>Data consumers and analyst access the data via SQL interface. AWS supported querying on top of S3 via Athena. Athena is AWS customized version of Presto. The main selling point is low latency interactive querying with faster results. This really was needed for the use cases outlined by Krish.</p><h3>Optimized Access</h3><p>Team released v1 of the architecture with energy and enthusiasm. After the release, Krish test drove the setup. He tried to answer some of the common questions business asked. He was nodding his head. The queries were taking long time to execute. Ajar suggested writing the joins differently but Krish’s argument was SQL is declarative. SQL Engine decides the order of execution.</p><p>Team had to hold off the celebration. Ajar, Kriti and Kumar went to find possible reasons. They read various articles, ran tests and hit upon the bottleneck: JSON. JSON was not giving the best performance with OLAP style queries. JSON, CSV are row oriented formats. What are row oriented formats ?</p><p>Lets see an example</p><figure><img alt="Row format containing 4 columns" src="https://cdn-images-1.medium.com/max/1024/1*R6RIQoy5T9bsbQcwpSDyog.png" /><figcaption>Row oriented Data</figcaption></figure><ul><li>Rows are stored as contiguous blocks in file</li><li>Good for single row operations i.e. find vehicle by license number</li><li>Not ideal when we want the number of vehicles per location(need only 1 column)</li><li>Application has to retrieve all the rows and all columns</li><li>Filter out the locations not matching</li><li>Count the rows</li><li>Now if we access only 2 out of 1000 columns, imagine the data transfer</li></ul><p>So whats the solution for these OLAP queries ? Column oriented formats are the answer. In Column oriented formats, only the needed columns are accessed. This means lesser data transfer and higher throughput.</p><figure><img alt="Columnar format" src="https://cdn-images-1.medium.com/max/1024/1*Ok3dW5qmYDuSXWZQQFjxuQ.png" /><figcaption>Columnar Format</figcaption></figure><p>Here, columns are stored together. If we need count of vehicles by locationid, just retrieve the single column location id. But these are not the only benefits of Parquet which is our column oriented format of choice</p><h4>Self Describing</h4><p>Parquet describes a well defined schema as part of its specification. Each column has a datatype and only values matching datatype can be stored. This provides a guarantee for the consumer on the datatype compared to CSV where everything is a text.</p><h4>Compression</h4><p>Since data stored in the column is of same datatype, it presents some good choices for compression.</p><h4>Dictionary Encoding</h4><p>Instead of having License number HV0003, we can map it to a integer 1. 1 occupies less size than string HV003. This would be a huge saving for millions of record file. Also for sparse datasets empty values are common and empty values can be represented as 0</p><figure><img alt="Dictionary Encoding" src="https://cdn-images-1.medium.com/max/1024/1*CwWQ0ykbZDMO8eQrr-c5ug.png" /><figcaption>Dictionary Encoding</figcaption></figure><h4>Run Length Encoding</h4><p>Lets take the compression to next step.</p><figure><img alt="Run Length Encoding" src="https://cdn-images-1.medium.com/max/1024/1*OWK6voBjFZLVZ-gLKBuPFw.png" /><figcaption>Run Length Encoding</figcaption></figure><p>As in the above image, if we know there are 500 records in sequence we can just say 500 0 1 1 to store efficiently.</p><h4>Optimize common access patterns</h4><p>When parquet files are written, common statistics like count of records, mean, sum at column level are stored in the footer of the file. Parquet file has the below structure</p><figure><img alt="Parquet format structure" src="https://cdn-images-1.medium.com/max/1004/1*KJHcRmPGpitrSqQt7zelQQ.jpeg" /><figcaption>Parquet format structure</figcaption></figure><p>Now when we look for count of records, Parquet returns from the footer and does not even read data blocks. The OLAP query we started with (Number of vehicles per location) will be answered efficiently by parquet. Parquet has predicate pushdown i.e. it will filter the location column in source and return records matching other location and count of only that rows happen. Very less data transfer resulting in quick speed.</p><p>Ajar went back to Krish with smile on his face. He showed the massive improvements with parquet. Query which timed out in 30 minutes was now running under a minute. Krish could see the potential already. This led to evolution of original architecture</p><figure><img alt="Evolved Architecture" src="https://cdn-images-1.medium.com/max/1024/1*8GMu1j0F0rqa13SoqBGj1w.png" /><figcaption>Evolved Architecture</figcaption></figure><p>This introduced a glue job which runs once per day to combine all the JSON files into parquet file based on predefined partitions.</p><blockquote>A good partitioning strategy based on access patterns saves a lot of query processing time and cost.</blockquote><p>Lets continue the story in next part and see what Ajar and his team are upto. Untill then :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9dd60f96791" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plumbersofdatascience/evolution-of-data-platform-part-1-f9dd60f96791">Evolution of data platform — Part 1</a> was originally published in <a href="https://medium.com/plumbersofdatascience">Plumbers Of Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>