<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Siddharth Mehta on Medium]]></title>
        <description><![CDATA[Stories by Siddharth Mehta on Medium]]></description>
        <link>https://medium.com/@siddharth-mehta?source=rss-f929f20f6311------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Siddharth Mehta on Medium</title>
            <link>https://medium.com/@siddharth-mehta?source=rss-f929f20f6311------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:23:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@siddharth-mehta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Journey from Databases to Data Lakehouse]]></title>
            <link>https://siddharth-mehta.medium.com/journey-from-databases-to-data-lakehouse-3e10e49b0d60?source=rss-f929f20f6311------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e10e49b0d60</guid>
            <category><![CDATA[data-virtualization]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[Siddharth Mehta]]></dc:creator>
            <pubDate>Wed, 14 Oct 2020 19:10:26 GMT</pubDate>
            <atom:updated>2020-10-14T19:10:26.526Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nO0ehR9hKW6p43y3" /></figure><h3>Evolution of Data Ecosystem</h3><p>Data ecosystem has evolved from the time’s of mainframes and punchcards to today’s IoT based data ecosystems like Smart Homes, Smart Cities, Connected Factories and even Smart Wearables. To get a sense of how large the data is growing in the world around us, here are some statistics. <em>As of 2020, 30+ Billion IoT devices generate 5 Exabytes i.e. 1 Billion GB of data “DAILY”. By 2025, total number of IoT devices may reach 42–45 Billion i.e. almost 50% increase in 5 years</em>. Data is the new currency in any and every enterprise today, and arguably every mature enterprise is aware of the value of data. As valuable the data keeps getting, probability is that the volume, variety, velocity and veracity (often known as 4 V’s) in data would keep increasing. These four V’s have directly or indirectly affected different areas related to data like Data Formats, Data Repositories, Data Consumption, Data Analytics, Data Management, etc.</p><h3>Database to Data Warehousing</h3><p>Between 1980’s — 1990’s (when many of us were in Kinder Garden and some of you were not even born) when the DBMS concept was getting popular in the industry, Ralph Kimball and Bill Inmon, two of the most famous and often known as the Father of Data Warehousing were arguing different school of thought for data warehouse modeling. Inmon’s methodology focused on centralizing all the data in a central repository in a normalized form, while Kimball’s data warehousing methodology focused on extending a data warehouse with data marts. These methods of dimensional modeling worked well when the major source of data generation was application driven or through manual methods, and consumption methods were limited to reporting and dashboarding.</p><p>Fast-forward from 1990’s to 2010 i.e. two decades later, came the age of big data when the four V’s started soaring. Data warehouse schemas were flattened, which posed a challenge a doing too many joins to consume the data. And Data Marts posed a challenge of one model has to fit many models of consumption. The traditional method of always moving data using Extract-Transform-Load (ETL) tools, relocating and replicating this data, and then re-modeling it in a data warehouse was not always the best option for the novel data applications. In fact, data started getting so huge and started originating so fast, that all the enterprise data can neither be modeled nor hosted in a data warehouse, in a timely or consumable manner. Enterprises wanted to let anyone bring in their data, to realize the potential from their data.</p><h3>Data Warehouses to Data Lakes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bK0pavYKgfAJdquD" /></figure><p>In late 2010, James Dixon of Pentaho coined the term “Data Lake” in his blog “<a href="https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/">Pentaho, Hadoop and Data Lakes</a>”. <em>Data Lakes hovered on the similar idea of data warehousing — which focused on centralized storage, but instead of transforming and moving all the data in a relational repository, Data Lakes focused on centralizing raw, unstructured semi-structured and structured data in a centralized logical layer</em>. To deal with big data that Data Lake would cater, and to be in-line with the scale-out approach, inspired from Hadoop, Data Lake incorporated the concept of Schema-On-Read as one of its principle. The concept of Schema-On-Read enabled data consumption without the need (or lesser need) for ETL jobs, as continuous ETL of big data is neither efficient nor feasible after a certain extent. This ideology gave way to enterprises to bring previously untapped data into the data ecosystem.</p><p>To enable this idea into implementation, cloud (which was in it’s infancy at the same time) provided the required fuel. Cloud vendors like AWS started introducing object storage like <a href="https://aws.amazon.com/about-aws/whats-new/2006/03/13/announcing-amazon-s3---simple-storage-service/">AWS S3 in Mar 2006</a>. And since then there has been continuous growth in different types of purpose-built data stores on cloud till date. Below is my humble attempt to cover different data stores on some of the major clouds, and I am sure that even after my best efforts, I am still missing to represent at least 30% or more of data stores as the clouds are always on the move and it’s beyond comprehension on a single person to keep track of all these ever-evolving repositories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vEGuZ_LcfeuXam06" /></figure><h3>Data Lakes to Data Swamps</h3><p>As novel as this idea of Data Lake is (or was), it brought it’s own set of challenges.</p><ol><li>As innovation in hardware space has neither been at the same pace as software nor has been economic, the scale-up methods were no longer efficient, and cloud brought scale-out methods to the data space.</li><li>Cloud introduced the notion of migrating every piece of data from on-premise to cloud, due to obvious elastic and managed services benefits on cloud. But those who were not able to migrate data, were / are stuck with those legacy systems and the data stuck in those systems is still awaiting to be brought into the cloud-mix till they migrate.</li><li>Mushroom-growth of purpose built data repositories on-premise and on-cloud, gave new life to ETL in the form of Enterprise Service Bus (ESB) or Enterprise Information Integration (EII). Data practitioners started building data pipelines from data lakes to data processing repositories, effectively replicating and relocating data all over the data landscape.</li><li>The philosophy of separate compute from storage has been ideally aimed to support scale-out approach. Data professionals started dumping data from relational databases, IoT, data files, mainframes, and every other possible nook and corner into object stores like AWS S3 or Azure Blob Storage. And then the same data is channeled back into aggregation and analytical repositories, irrespective of whether it’s hot data (frequently used) or cold data (occasionally used).</li><li>As the popularity of columnar compressed data formats like Parquet, ORC, Avro, Apache Arrow etc increased, data hosted on object storage formats, started getting almost cloned — one into native format (like CSV, JSON etc) for the golden record, and another into compressed data formats.</li></ol><p>One of the issues that most enterprises started experiencing considering the above points, which is still the case till date, is a loosely controlled Data Lake. <em>Cloud migration and variety of purpose-built data stores as well as processing systems on the cloud, resulted in anyone bringing in any type of data, often without any profiling, often same data being brought multiple times by multiple groups, at most places for hosting, aggregation, search, analytics etc. Data Lake eventually started becoming a data hoarding ground, which is also known as “Data Swamp”.</em></p><p>When enterprises spend most of their time in below mentioned activities, than consuming their data effectively from data lake and realizing maximum value out of it, it’s a signal that their Data Lake is more of a Data Swamp.</p><ol><li>Data life-cycle management is uncertain, as no one knows which data is required by whom at what time for any given purpose.</li><li>Searching for the data of interest or value is the hardest part of the job</li><li>Massive customized curation is required to make the data usable</li><li>Maximum data budget is spent in re-modeling and integration efforts to consume the data</li><li>Governance requires individually plumbing each data repository and data objects to achieve cell level (row + column) security.</li></ol><h3>Data Swamp to Data Virtualization</h3><p>In 2013 <a href="https://www.facebook.com/notes/facebook-engineering/presto-interacting-with-petabytes-of-data-at-facebook/10151786197628920/">Facebook introduced Presto</a> to the world. The scale of data at which Facebook operates, ETL is no longer an option. Hadoop based data warehouses were the norm during those times and even today suited for the type of data that Facebook operates on. In 2014, <a href="https://netflixtechblog.com/using-presto-in-our-big-data-platform-on-aws-938035909fd4">Netflix cemented data virtualization applications</a> by announcing that they use Presto for querying 10 peta-byte data on AWS S3. Around 2016–2017, Gartner gave data virtualization it’s highest ranking in the hype-cycle, signaling it’s ready to gain maturity and main-stream adoption in less than two years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/0*DHLDVzY62_PAceCE" /></figure><p><em>The concept of data virtualization is based on the theory of providing a uniform and abstract data layer, that can reduce the data integration burden, enable consumers to access data as if they were dealing with a single source of data</em>. The benefit is that consumers can access data without need to have multiple database credentials, without knowing multiple query languages, without waiting for ETL teams to bring the data of interest to data extraction points, without the need to replicate and relocate data for every frequent and infrequent data needs, and without the risk of accessing stale data as the data extraction point was not refreshed with ETL jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FMDDvFACyXZJ7paQ" /></figure><p>Think of a scenario where one has data spread in below repositories.</p><p>1) Blob stores on cloud (csv, tsv, json, gzip, bzip, parqet, avro, orc etc)</p><p>2) Relational databases on cloud (SQL Server, Oracle, Postgres, MySQL etc)</p><p>3) NoSQL databases on cloud (Graph, Document, Key-Value, TimeSeries, Lucene Indexes, Spatial, Ledger etc)</p><p>4) On-premise commercial DB’s or systems (Teradata, Netezza, Mainframe, Exadata, DB2 etc)</p><p>5) Data on 3rd party SaaS cloud vendors like Snowflake, MongoDB, multiple public clouds etc.</p><p>If one wants to query this data together to extract valuable business insights, the immediate solution that comes to mind is ETL. In the modern data architectures, the answer can be Data Virtualization, where data is queried in place from all these sources, mashed up in the data virtualization layer and then served to the client. And the consumer is abstracted from the details of all these sources and submits a single query to the data virtualization layer, which handles the heavy-lifting of integrating with corporate data catalog to find the address of the data, extracting this data from relevant repositories, optimizes query for each type of engine, caches data, applies role based security, converts data in the format of choice before serving etc.</p><p>Examples of data virtualization tool providers are Denodo, Actifio, Atscale, IBM Cloud Pak for Data, and other similar players. Public clouds have been offering services like AWS Presto on EMR, AWS Redshift Spectrum, AWS Athena, Google BigQuery, Databricks Delta Lake, Apache Drill, Informatica Intelligent Cloud Services etc to implement data virtualization for addressing the above mentioned challenges. But <em>data virtualization by itself is a practice area in itself, whereas data warehouses or data lakes are modeling techniques</em>. There has not been any similar successor of data lakes (born out of the concept of scale-out and separation of storage and compute) to the theory of data virtualization. Even cloud providers did not have / had an out-of-box data virtualization service. Query federation and other similar concepts started getting introduced this year (2020) itself.</p><h3>Data Virtualization to Data Lakehouse</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vVnxpYj2daetE84s" /></figure><p><em>In simplest words (not to take away any credit from the concept), a unified product or a set of services working together, to address the diverse needs of data consumption, in various formats from various sources, is known as Data Virtualization.</em> Earlier this year, Data Bricks team started the concept of Data Lakehouse via a <a href="https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html">blog</a>, which got extended in their <a href="https://databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf">VLDB whitepaper</a>. Informatica has a <a href="https://blogs.informatica.com/2020/05/12/3-key-essentials-for-cloud-lakehouse-data-management/">Cloud Lakehouse certification</a> as well, which shows how the popularity of this term is picking up. The key features of Data Lakehouse as per the Data Bricks blog, and my thoughts on each points are mentioned below.</p><ol><li>ACID Transaction support. Data Virtualization delegates transactions to actual data repositories. So a virtualization layer is just a mediator for passing client transactions, whereas data repositories manage actual transactions. For transactions on data lake storage repositories like AWS S3 or Azure Data Lake Storage, layers like Apache Hudi or Delta Lake offers ACID Transactional support, which can be virtualized / abstracted with Data Virtualization.</li><li>Schema enforcement and governance. Data Virtualization tools / layers support it.</li><li>Support for variety of data, Storage is decouple from compute, Openness as well as BI Support. This is leveraging the Data Lake Storage, and virtualization plays to role of data aggregator.</li><li>Support for diverse workloads and end-to-end streaming. This is data virtualization as the ingestion end as well. When you have data virtualization at ingestion as well as consumption end, which is generally the case, this capability comes in organically.</li></ol><p>Each present and future provider of Data Lakehouse may offer peculiar set of features, but the enabler (IMO) is data virtualization. It’s not as novel a concept as it is as a product offering.</p><h3>Conclusion</h3><p><em>Data Virtualization is not a replacement for ETL, and it’s use is relevant depending upon use-cases</em>. In my humble opinion, Data Lakehouses have been existent on cloud with data virtualization implementation through 3rd party providers, though it was not named so. Public clouds started realizing the value of this, and have started virtualization capabilities either natively (like AWS Athena Query Federation) or by collaborating with providers in this space (like Azure Data Bricks Delta Lake). <em>Data Lakehouse in my own personal opinion, is a successor of Data Lakes. Data Virtualization is an area of practice, and the way it gets modeled will be seen or called as Data Lakehouse. And Data Lakes as well as Data Warehouses will still continue to exist, either independently or in abstracted form in the data virtualization products being used</em>. Data Lakes are here to stay, the way we consume has definitely changed, and for all the right purposes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e10e49b0d60" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>