<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravi Chikkam on Medium]]></title>
        <description><![CDATA[Stories by Ravi Chikkam on Medium]]></description>
        <link>https://medium.com/@ravibabu.chikkam?source=rss-1ffc6457e5da------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8LQZZz2fTn-gRkJX</url>
            <title>Stories by Ravi Chikkam on Medium</title>
            <link>https://medium.com/@ravibabu.chikkam?source=rss-1ffc6457e5da------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:53:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ravibabu.chikkam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Strategy vs Tactics]]></title>
            <link>https://medium.com/@ravibabu.chikkam/data-strategy-vs-tactics-52db3d73b230?source=rss-1ffc6457e5da------2</link>
            <guid isPermaLink="false">https://medium.com/p/52db3d73b230</guid>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[modern-data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ravi Chikkam]]></dc:creator>
            <pubDate>Mon, 24 Oct 2022 16:18:56 GMT</pubDate>
            <atom:updated>2022-11-11T18:16:11.616Z</atom:updated>
            <content:encoded><![CDATA[<p>A quote from ancient wisdom still rings true in current times. About 2500 years ago Sun Tzu said:</p><p><em>“Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat”.</em></p><p>This article is a follow-up to my short note on “<a href="https://www.linkedin.com/pulse/digital-transformation-ravi-chikkam/">Digital Transformation</a>” on LinkedIn.</p><p>As data is exploding beyond big data, many tools, technologies, and services are being offered by myriad vendors to address the problem of storing and processing it. (Data?) Architects and Engineers are deploying these in many combinations (stacks?). It is essential to think of these technologies in stacks as it simplifies communication and serves to rally skills and capabilities to deliver meaningful products and services just as LAMP, MEAN, etc., did for modern application development. What is new in these stacks is that they are actively being promoted as a panacea for the data problem.</p><p>With the rise of AI and Machine Learning, there is a sudden realization of the importance of data. It is now general knowledge that “Most modern apps and services consume hand-crafted algorithms and data to serve users&#39; needs as well as some more data. Whereas the AI &amp; ML consume data to generate a model/algorithm to serve users&#39; needs”. So more the merry become the norm these days about data. There are even billion+ dollar businesses built on processing application logs for various specialized purposes including cybersecurity.</p><p>Logs are the only easily accessible portion, but more valuable data is generated and stored inside the persistence layers of these apps and services. This data is either structured or semi-structured in the form of tables or documents. Since these data stores are applications specific the data in these stores is siloed.</p><p>Despite the advances and ease of use of tools and technologies handling siloed data, executive or customer-facing dashboards are always fragile. Most managers responsible for building and maintaining these dashboards regularly end up explaining themselves to angry stakeholders.</p><p>This fragility is due to the way data is acquired and believing that tactics (tools and technologies) make up for the strategy. Yes, It just takes a day or two to weave together an ELT (maybe another day to connect data in silos via ETL) using Snowflake and Fivetran for example. Throw dbt into the mix you can create aggregations and analytics the next day. These tools are remarkably stable and not the source of fragility despite their relative recency. While these are an essential part of the data landscape, they are just tactics that speed up a solution. A clear and viable data strategy is still needed.</p><p>In a rush to build the data lakes, most architects and senior engineers ignore the application life cycle of the applications that are generating this data. For a larger enterprise, the teams that build and change applications are always outside the control of the data team. This can be remedied by (human) communication mechanisms with their imperfections.</p><p>Yet, the biggest source for the fragility comes from mistaking data sourced directly from these application-specific databases as a dataset. For example, if these applications have published (and curated ) API end-points, the data is a dataset. This is because a lot of application code is written outside the persistence layer to interpret this data. In the case of the “schema on the read” paradigm of the NoSQL, the conceptual gap between the raw data and the actual dataset is further. This is because the application is interpreting the data in the context where it is created and serves the API queries. If you take this a step forward the applications can choose to publish these datasets out to Kafka or S3 as the changes happen in the form of events.</p><p>If this sounds similar to Domain Driven Design (Eric Evans’ book), you are on the mark. All of this requires a lot of work at the Enterprise level but the payoffs will be significant as the data acquired into the data lake is no longer raw, it is a dataset. In a micro-services world, the API is hosted at the <a href="https://martinfowler.com/bliki/DDD_Aggregate.html">aggregate root </a>of a set of entities serving a business purpose within an <a href="https://martinfowler.com/bliki/BoundedContext.html">application context</a>.</p><p>Whether you use a direct ODBC connection to an application database or use a cloud-based Fivetran connector or consume CDC (change data capture) piped to a Kafka topic the fragility is still there. This is because of the tight coupling of the data lake to the applications’ databases. In the lingo of Martin Fowler’s refactoring let’s call this coupling <a href="https://wiki.c2.com/?InappropriateIntimacy">inappropriate intimacy</a>. This is a fact even if you got the credentials to connect to the applications’ databases transparently.</p><p>So the appeal is to recognize the direct acquisition of data from applications’ databases is a tactic, not a strategy. Even if you use ELT powered by the so-called modern data stacks (Snowflake/dbt + Fivetran) you still have the data silo problem due to the lack of an Enterprise-wide data strategy. Such a strategy still needs to be informed and supported by the best architectural advice appropriate to the enterprise&#39;s needs so the datasets published and stored in the data lakes are curated by application-informed logic before these are fed into analytical pipelines. If you are in the business of feeding these datasets to a machine-learning model the need for quality (and control) is far higher as the cognitive distance from the application programmer to a data scientist is far greater than that of an application programmer to a data engineer (who is also an application programmer with more SQL or Python savvy!).</p><p>This article does not address the problem of handling the changes (versions?) to the datasets engendered by the SDLC of the applications that generate these datasets. That need to be solved at data lakes. There are two kinds of changes: schema changes, and data changes. Savvy architects and data leaders have already recognized the need to go beyond the approach of building data lakes as a set of files with a chosen format (organized into a directory structure)! Instead, the attention is on the table format specifications rather than the specific file formats.</p><p>In brief, this approach standardizes the data lake using table format specifications such as Delta Lake, Apache Hudi, and Apache Iceberg. These formats handle data changes (without needing a large-scale x-copy to maintain versions), schema changes, and partition evolution. Some of these features are termed attractively enough to catch attention, “time travel” for example. Apart from this the ability to adapt and store continuously changing datasets, these specifications also allow different computing engines such as Apache Spark, Apache Flink, and even Snowflake to operate on the same underlying dataset. If there is a flexible way of scheduling the compute containers (embedding Spark, Flink, etc.,) on cheap AWS spot instances, you found a way to limit your compute cost.</p><p>While the “time travel” may help to query versioned data in tables, <a href="https://projectnessie.org/">Project Nesse</a> brings git-like commits to the data lake to ensure consistency across multiple tables. This enables a <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow">gitflow-like</a> workflow to mutate data in the data lake in a methodical way. Now we are talking about quality datasets!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52db3d73b230" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>