<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hung Dang on Medium]]></title>
        <description><![CDATA[Stories by Hung Dang on Medium]]></description>
        <link>https://medium.com/@hung-dang?source=rss-72d7b6f6b8c7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*g76SPhT4PTy8i96PtGZ4Lg.jpeg</url>
            <title>Stories by Hung Dang on Medium</title>
            <link>https://medium.com/@hung-dang?source=rss-72d7b6f6b8c7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:11:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hung-dang/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[3 workflow improvements I wish dbt announced at Coalesce 2023]]></title>
            <link>https://blog.dataengineerthings.org/3-workflow-improvements-i-wish-dbt-announced-at-coalesce-2023-fb9216e36785?source=rss-72d7b6f6b8c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb9216e36785</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt-labs]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Hung Dang]]></dc:creator>
            <pubDate>Wed, 29 Nov 2023 17:08:05 GMT</pubDate>
            <atom:updated>2023-11-29T17:08:05.416Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bv0a4kgz_QQYCu7VsTNsdQ.png" /></figure><p>Last month, the data world paused to catch <a href="https://coalesce.getdbt.com/">dbt’s annual Coalesce event</a>. The main theme was “dealing with complexity”. The solution? An architectural mesh that splits a monolithic dbt project into smaller, interconnected ones. Contracts, versions, and access control guide these projects, fostering speed, collaboration, and trust on a large scale. New features like dbt Explorer, cloud-based CLI, and CI/CD capabilities rolled out, but only for paying Cloud customers.</p><p>Meanwhile, dbt-core, the open-source lifeline of dbt Labs, <strong>stands still</strong>. The same dbt-core that people rallied behind for years to build the largest data community and establish it as the go-to for data transformation. Yet, this time, no significant investments were announced for it other than minor incremental fixes. With each new dbt release, the gap between dbt-core and dbt Cloud continues to widen.</p><p><a href="https://www.getdbt.com/dbt-labs/about-us">Of the 30,000 companies using dbt, only 3,600 are dbt Cloud customers</a> — <strong>that’s just 12%</strong>. This reveals that a vast 88% of dbt’s customer base is still relying on dbt-core, the open-source platform. Such a major reliance on dbt-core by the community raises concerns about the long-term effects of its neglect. As dbt Labs continues to roll out updates tailored mainly for its premium service — dbt Cloud, one can’t help but wonder about the potential disconnect that may arise from overlooking the needs of the majority.</p><h3>What’s new from dbt?</h3><p>dbt Cloud’s product updates reveal a commitment to scalability and workflow optimization, including:</p><ul><li><a href="https://www.getdbt.com/product/dbt-mesh"><strong>dbt Mesh</strong></a>: break down large dbt projects into smaller, collaborative components.</li><li><a href="https://docs.getdbt.com/docs/collaborate/govern/project-dependencies"><strong>Cross-project ref &amp; orchestration</strong></a>: reference models and orchestrate pipelines across multiple dbt projects.</li><li><a href="https://docs.getdbt.com/docs/collaborate/govern/model-contracts"><strong>Contracts</strong></a><strong>, versions, and access control</strong>: provide governance through contracts and model versions while boosting security via granular access control.</li><li><a href="https://www.getdbt.com/product/dbt-explorer"><strong>dbt Explorer</strong></a>: an enhanced version of dbt docs designed <a href="https://open.substack.com/pub/analyticsengineeringroundup/p/and-thats-a-wrap?selection=43926e40-a5d7-43f7-bf32-db9ebccbede0&amp;utm_campaign=post-share-selection&amp;utm_medium=web">to scale beyond the 1200–1400 models mark</a>, with the addition of some more metadata attributes specific to each object type.</li><li><strong>(New) </strong><a href="https://www.getdbt.com/product/semantic-layer"><strong>dbt Semantic Layer</strong></a>: introducing adaptable joins and a wider range of metric types.</li><li><a href="https://docs.getdbt.com/docs/deploy/ci-jobs"><strong>CI Jobs</strong></a>: automatic deferral to a different environment and ability to rerun jobs from the point of failure.</li><li>and the <a href="https://docs.getdbt.com/docs/cloud/dbt-cloud-ide/develop-in-the-cloud"><strong>dbt Cloud CLI</strong></a></li></ul><p>These developments are aimed at teams scaling their data practices, with an emphasis on collaboration, control, and comprehensive documentation/data catalog. We can categorize these new capabilities into two overarching themes designed for teams that work at scale: <strong>enhancing visibility</strong> and <strong>workflow improvements</strong>.</p><p>The mesh architecture required cross-project references, contracts, versions, and access control to govern information and have it flow securely from one project to another. All wrapped with a new, more performant docs service, dbt Explorer.</p><p>In terms of workflow improvements, the CI jobs improvements and the Cloud CLI (simplifying python dependencies, credentials management and local setup) only scratch the surface of what could have been done in terms of developer workflows and ops management.<strong> This is where I think folks expected true innovation coming from dbt Labs.</strong></p><h3>Development workflow improvements I wish dbt announced</h3><p>Looking back, not much has really changed in data handling over the last decade. We’ve moved from those drag-and-drop ETL tools to code-based systems that are version-controlled, more flexible, and easier to work with. Add on top automated docs, tests, CI/CD, observability and we are rapidly getting on par with the software engineering best practices — as we aimed so.</p><p>Yet, the fundamental way we manage data assets — both data and metadata — remains the same. Working with data involves more than changing code. It involves managing the codebase, database schema, infrastructure, and data. Keeping them all in sync is a constant challenge. If we don’t address this, there’s a real risk of your codebase drifting away from what’s actually happening in your data warehouse.<strong> So, what if we could handle everything from one place? Imagine having a system where your code is the single source of truth, always in sync with the latest state of your data warehouse </strong>— just like in software development, having a single system dictating the state of our app.</p><p><strong>This is the kind of game-changing shift we expected dbt to announce at Coalesce. Better workflows designed for data specifically. It’s about carving out our own path as data practitioners, not just borrowing ideas from software engineering.</strong></p><h3>Step 1: Making our intent possible</h3><p>To get our data warehouses to truly reflect our code, we need a system where changes in the codebase directly update the data warehouse. It’s simple: change your code, and your data warehouse changes too. Deploy something new or roll back a change, and it should instantly show up in your data, schema, everything. This way, what you see in your code is what’s really happening in the data warehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YaK3JFlmMXim17y9Wy-9XQ.png" /></figure><h3>Step 2: Doing it without wasting compute resources</h3><p>Sure, tracking every code change by adding new tables in the data warehouse sounds cool. But that could get messy and costly fast. We should be efficient and reuse tables where possible. For example, what we already built in dev before creating the PR could be reused when moving in prod if no other changes or refreshes happen in the meantime. But for this to work, we need a setup where we’re always working with the latest data from prod in dev, making sure we could perform the swap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmISsdFwLBA8_bvIF0YPkQ.png" /></figure><h3>Step 3: Making it safe for everyone to develop.</h3><p>Efficiency is important, but what’s crucial is creating an environment where data is accessible and safe for everyone to use and develop. <a href="https://www.y42.com/blog/how-to-improve-data-quality-with-better-data-quality-tools/">We need to break down the barriers that limit who can work with data</a>, ensuring it’s not just a few gatekeepers holding the keys. This means equipping our tools with safety features that prevent most common incidents, while still offering the flexibility to customize those safety measures. Additionally, we should be able to quickly revert to a previous state if things don’t go as planned. By doing this, we empower more people in an organization to confidently use and experiment with data, democratizing its use and preventing bottlenecks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uYHaXuB2qgBmEqdB0mwFSw.png" /></figure><p>But dbt hasn’t moved in this direction yet. No big updates for dbt-core or its paying users, nothing that really changes the game for how we manage data pipelines.</p><h3>A deeper problem with the Modern Data Stack (MDS)</h3><p>There’s a fundamental issue in the MDS that many have hinted at but few have addressed head-on. As developers, we’re still juggling multiple disjointed tools to run an end-to-end pipeline. This means integrating various systems, investing in specialized training, managing access across different services, and understanding the full performance of our pipelines. It’s a constant hop from one tool to another. <a href="https://open.substack.com/pub/analyticsengineeringroundup/p/and-thats-a-wrap?selection=c02ea235-8ffa-4fb9-8553-3bb310061d66&amp;utm_campaign=post-share-selection&amp;utm_medium=web">While dbt acknowledges these challenges</a>, we haven’t yet seen all-encompassing great solutions tailored for the MDS.</p><p>Data leaders want fewer vendor relationships. I’ve talked about this at length before and won’t belabor it here, but it came out very clearly this week. Relatedly, the boundaries between product categories like observability, quality, governance, cataloging, discovery, and lineage are becoming less and less clear. They were never that clear to begin with, and vendors are now increasingly overlapping with one another’s functionality.</p><p>A few years ago, I would’ve said dbt, central to our modern data stacks, could drive this consolidation. Fewer tools and more integration would lead to a more coherent development experience, lower billing and management overhead, less time onboarding and hiring individuals, and better all round products that solve also some of the real-world challenges teams face today, challenges often overlooked that fall between the cracks of integrating so many vendors.</p><p>For instance, when focusing solely on transformation, changes in upstream sources can cause significant downstream issues. Keeping load and transform coupled could improve debugging and give better control over the pipeline.</p><p>Another area is monitoring. A lineage view of dbt assets is useful for visibility, but what if this view also showed each asset health status? Imagine a color-coded system indicating the status of each asset, allowing you to jump right into it and fix problems as they arise. Simple, yet so powerful.</p><p>There are also more advanced topics to explore: how pull requests would look like in the context of data assets: rather than analyzing file by file, maybe zoom out at a conceptual level, see the downstream impact of your changes in a visual manner, determine which tests to add/remove based on the impact, and select which assets to reuse rather than rebuild. That might be an option. Similarly, why stick to manually generating YAML files for your models, when a platform could translate user’s actions into YAMLs or generate new ones based on common patterns, such as creating staging tables? What’s the ideal IDE experience for data practitioners? How can we further leverage git in our workflows?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb9216e36785" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/3-workflow-improvements-i-wish-dbt-announced-at-coalesce-2023-fb9216e36785">3 workflow improvements I wish dbt announced at Coalesce 2023</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing Data Warehouse Environments]]></title>
            <link>https://blog.dataengineerthings.org/managing-data-warehouse-environments-7fcf20254fdd?source=rss-72d7b6f6b8c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fcf20254fdd</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Hung Dang]]></dc:creator>
            <pubDate>Fri, 13 Oct 2023 18:02:39 GMT</pubDate>
            <atom:updated>2023-10-13T18:02:39.118Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Solution Overview for Data Engineers</h4><h4>Moving from siloed approaches to unified solutions</h4><p>As data or analytics engineers, our work is often guided by three distinct systems: the tracking of logical changes through Git version control, the management of materialization changes resulting from code alterations, and the navigation of operational changes via various environments. But what if these disparate systems could be merged into a single unified framework? In this article, we will explore the past, present, and a new concept to manage data warehouse environments efficiently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4qdsUdSlxOlzZSe8bc-AQ.png" /><figcaption>In the current state of managing your data project, only code-level changes are tracked via Git, but not the state of your data assets and environments.</figcaption></figure><h3>What are Data Warehouse Environments?</h3><p>A <a href="https://www.oracle.com/database/what-is-a-data-warehouse/">data warehouse (DWH) environment</a> refers to a specific setup or instance of a data warehouse, with its associated data, transformations, connections, and configurations. This can be thought of as the context in which your data warehouse operates.</p><p>In a typical software development process, multiple environments may be used to manage changes safely — such as development, staging, and production environments. Similarly, data warehouse environments are used to manage and test changes to the data warehouse setup — including changes to the structure of the data, the transformations applied, the connections used, and so on — before those changes are deployed to the production environment. Each environment, in effect, operates as a separate instance of the data warehouse, with different data and configurations.</p><h3>The Challenges with Managing DWH environments</h3><p>Managing DWH environments can be a complex task, as it involves ensuring that data, schema, transformations, and configurations are correctly synchronized across environments, and that changes are safely tested and deployed, often requiring deploying manually configurations, long-running CI pipelines, and code to physical schema synchronization.</p><p>This is particularly challenging when working with large volumes of data, as creating separate physical copies of the data for each environment can be resource-intensive.</p><h4>Traditional Database Environment Management</h4><p>Managing database environments involves tasks such as asset materialization, schema management and code versioning. Tools like <a href="https://flywaydb.org/">Flyway</a> and <a href="https://www.liquibase.com/">Liquibase</a> have allowed controlled, versioned changes to database schemas, often alongside application changes. Meanwhile, Git has been the go-to solution for code versioning. However, synchronizing changes between the codebase and the database schema often involves rerunning data pipelines for backfilling data. This is especially true for cases like new column additions. The process is both resource-intensive and time-consuming.</p><h4>LakeFS for Data Lake Environments</h4><p>In the domain of data lakes, <a href="https://lakefs.io/">LakeFS</a>, an open-source platform, tackles the challenge of data versioning. It brings Git-like capabilities to data lakes, allowing for branching, isolated changes, and version commits. However, LakeFS mainly addresses raw data versioning and does not extend its management and synchronization capabilities to transformed data across different environments.</p><h4>Snowflake and BigQuery’s Cloning Features</h4><p>Cloud-based data warehouse providers such as <a href="https://www.snowflake.com/en/">Snowflake</a> and <a href="https://cloud.google.com/bigquery">Google’s BigQuery</a> have stepped up to the challenge with features such as <a href="https://mindmajix.com/snowflake-cloning">zero-copy cloning</a> and <a href="https://cloud.google.com/bigquery/docs/table-clones-intro">table clones</a>. These features allow instant duplication of data without any actual data replication — solely referencing assets. Despite their utility, they need to be carefully integrated with deployment activities.</p><p><strong>Pros</strong>: Cost-effective</p><p><strong>Cons: </strong>Doesn’t address the management of environments and assets materialization.</p><h4>dbt for Data Warehouse Environments</h4><p><a href="https://www.getdbt.com/">dbt</a> has transformed data warehousing by enabling SQL-driven transformations and tests. However, managing environments in dbt presents challenges. The — <a href="https://docs.getdbt.com/reference/node-selection/defer">defer</a> flag, which allows mixing of production and development assets based on a snapshot of the manifest.json, may work for testing in lower environments but has limitations for production deployments.</p><p>Asset management in dbt is not straightforward. Suppose an asset was updated in production while being deferred in development. This introduces complexities around which version to reference in production pipelines and how to efficiently manage subsequent deferrals.</p><p>Moreover, promoting changes with dbt often involves long rebuilding cycles for each changed asset. This process can be resource-intensive, slowing down data operations. Despite dbt’s significant contributions, these challenges underline the need for enhanced environment management solutions in data warehousing.</p><p><strong>Pros</strong>: Code changes version controlled. Allows experimentation using the — defer command in lower environments. With the new dbt clone command you can lower your costs of creating new environments as replicas of prod.</p><p><strong>Cons</strong>: When it comes to asset deployment and environments, these are still managed separately from your code changes.</p><h4>SQLMesh for Data Warehouse Environments</h4><p>While <a href="https://sqlmesh.com/">SQLMesh’s</a> Virtual Environments improve data warehouse management, they introduce a new tool with a unique syntax to manage environments (SQLMesh plan) instead of using an already-established system, such as Git, to manage operational changes. As a user, you have to run SQLMesh commands to manage operational changes alongside managing your code changes.</p><p><strong>Pros</strong>: Cost-effective (using view pointer swaps, reducing the storage and compute costs)</p><p><strong>Cons</strong>: Some overhead as the environment and asset materialization are not highly coupled wit</p><h3>A data warehouse environment for every Git commit</h3><p>Looking ahead, it’s easy to envision a future where every single commit in your Git repository generates its own isolated data warehouse environment. In this future, data engineers and analysts can focus on creating and refining their models, transformations, and visualizations without the overhead of managing underlying environments.</p><p>Ideally, a data warehouse (DWH) environment for each Git commit would mean that each branch in your Git repository would correspond to an isolated DWH environment. These environments, ephemeral in nature, would mimic production environments, allowing you to test your changes extensively before deploying them. They could be spun up when needed and torn down when not, without duplicating data and thereby saving storage and compute resources.</p><p>The system would track your Git actions such as creating branches, committing, merging, or reverting your code changes and automatically handle the creation of environments.</p><h4>How would it work?</h4><p>Such a system would require two schemas: one internal, containing all physical materializations of your data assets based on your git activity, and one public, which would cherry-pick from the internal schema the variations of the data asset that align with your current state in Git. The cherry-picking process would be instant and could potentially be facilitated by using view pointer swaps.</p><p>In such a system, the following would be possible:</p><ul><li><strong>Committing Changes</strong>: Upon committing changes to one or more data assets, a new physical variation of the asset(s) would be created. The view pointers in the public schema would automatically update to reflect the new state of your codebase, referencing the latest materializations of the assets.</li><li><strong>Reverting Changes</strong>: When you revert changes (at the commit or individual file level), the public schema view would point to the previous materialization of the asset, eliminating the need to recreate tables.</li><li><strong>Creating New Branches</strong>: This action should incur no additional storage or compute cost, as new logical replicas simply point to the same underlying data in the internal schema.</li><li><strong>Merging Branches</strong>: The public schema view(s) would reference the existing physical table(s) from the branch you just merged, again, avoiding the need for table recreation.</li><li><strong>Upstream Changes</strong>: Made a change affecting other assets downstream? The system should alert you to recreate all impacted assets downstream to keep the state of your codebase in sync with the DWH assets. When changes are pushed to the main branch, the operation simply involves a pointer swap in the public-facing view definition.</li><li>Furthermore, this system would facilitate parallel testing of multiple changes without the risk of them interfering with each other. It would eliminate the tedious task of synchronizing different environments manually and reduce the risk of errors due to misconfigurations or missed synchronizations.</li></ul><h3>Conclusions</h3><p>As data complexity and volume grow, effective management of data environments is becoming increasingly essential. The challenges with traditional methods of managing data environments are clear, whether it’s a relational database with Flyway and Liquibase, a data lake with lakeFS, or a data warehouse using solutions like Snowflake, BigQuery, or dbt.</p><p>However, the landscape is changing rapidly, and new solutions are emerging that aim to address these challenges and simplify the management of data environments. One such promising direction is the concept of creating a separate data warehouse environment for every Git commit, enabling isolation and precise control over data changes.</p><p>Companies like SQLMesh and Y42 are pioneering this concept with <a href="https://tobikodata.com/virtual-data-environments.html">Virtual Data Environments</a> and <a href="https://www.y42.com/blog/virtual-data-builds-one-data-warehouse-environment-for-every-git-commit/">Virtual Data Builds</a>, making the process seamless and more efficient. This approach doesn’t just simplify data environment management but also opens up new possibilities for testing and deploying data changes.</p><p>In the end, the choice of tool will depend on the specific needs of your organization and your data pipeline. However, as we move into this new era of data management, it’s clear that the future is not just about managing data but managing it effectively and efficiently, with clear version control and environment management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fcf20254fdd" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/managing-data-warehouse-environments-7fcf20254fdd">Managing Data Warehouse Environments</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Y42 built the first Modern DataOps Cloud]]></title>
            <link>https://hung-dang.medium.com/why-y42-built-the-first-modern-dataops-cloud-d9b87f4f1231?source=rss-72d7b6f6b8c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9b87f4f1231</guid>
            <category><![CDATA[data-tools]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[dataops]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[modern-dataops-cloud]]></category>
            <dc:creator><![CDATA[Hung Dang]]></dc:creator>
            <pubDate>Tue, 08 Nov 2022 16:49:26 GMT</pubDate>
            <atom:updated>2022-11-08T16:49:26.658Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dx6utD-6YnxX7OFp.png" /></figure><p>We at Y42 are confident that every data team’s mission should be to produce reliable data tables efficiently (at a low cost and high performance) and make them consistently accessible to downstream users and applications to help ensure strong decision-making and further automate critical processes.</p><p>To achieve this goal, data teams need to have coherent, reliable, and <strong>efficient processes </strong>to generate production-ready data pipelines. However, as we <a href="https://www.y42.com/blog/data-industry-problems">outlined in our previous article</a>, the modern data stack (with all its point-to-point tools) is not really accessible. And product management best practices, such as collaboration and governance, can’t just be “bolted onto” the stack, as they have to apply to all downstream and upstream processes to enhance DataOps in a meaningful way.</p><p>What’s more, the data industry is severely lacking a crucial asset that’s needed to help enable DataOps in any company: engineering power. Due to the high demand and low availability of data engineers, companies are often left with limited resources and fail to carry out comprehensive DataOps practices.</p><p>Therefore, it has been Y42’s core mission to fill this gap in the data industry with an efficient and scalable data solution to bring the world one step closer to taming the complexity that working with data entails. That’s why we set out to build the first <strong>Modern DataOps Cloud</strong>. Y42 now enables companies with the ambition to go beyond simple ad-hoc reporting to build production-ready data pipelines with a <strong>quick turnaround time,without heavy investments in data engineering.</strong></p><p>The <strong>Modern DataOps Cloud </strong>serves asmission control for a company’s data pipelines. Running on top of Snowflake and BigQuery, it enables organizations to deliver reliable and high-quality data to every downstream user or application. With Y42, companies are finally able to upgrade their often stovepiped data stack to an easily accessible and fully managed DataOps environment.</p><p>Y42 leverages the world’s most advanced existing open-source tools and technologies and embraces software engineering and product management best practices, making <strong>accessibility</strong>, <strong>collaboration</strong>, and <strong>governance</strong> the three guiding principles of our product.</p><p>Here’s how the Modern DataOps Cloud tackles the core problems currently present in the data tooling landscape, one by one.</p><h3>Accessibility</h3><p>Y42 natively integrates many of the modern data stack’s best-of-breed tools, resulting in one unified webapp, API, and CLI. This approach saves organizations precious data engineering hours stitching together and maintaining countless tools:</p><ul><li><strong>Integrations: </strong>Managed Airbyte with 150+ integrations</li><li><strong>Orchestration: </strong>DAG-based workflow scheduler — interoperable and extensible with any other orchestrator and cloud tool such as Airflow</li><li><strong>SQL Model: </strong>Managed dbt Core overcoming some issues, such as the need to create a dataset for every branch in order for Git to work properly</li><li><strong>No-Code/Low-Code Model: </strong>Proprietary drag-and-drop SQL query builder to enable less technical users to build pipelines and explore data ad hoc</li><li><strong>Change Management:</strong> Managed Git server with features like pull requests and branch protection</li><li><strong>Headless BI: </strong>Managed Cube Dev providing a reliable metrics layer used across BI, data science, and operational applications — accessible through APIs</li><li><strong>Pipeline Visualization: </strong>Managed ECharts which enables users to quickly create powerful charts in Y42’s visualization layer</li><li><strong>API &amp; CLI:</strong> Fully programmable pipeline through APIs, CLI, and webhooks</li></ul><h3>Governance</h3><p>Data governance refers to the enforcement of rules that reduce the margin of error for working processes, transforming a team’s working style from reactive to proactive. This is possible due to the native integration approach in Y42, which spans across the whole data stack:</p><ul><li><strong>Access Control: </strong>Role-based, multi-level access allows organizations to have full internal and external control of every data asset</li><li><strong>Asset Ownership: </strong>A system of ownership for better change management and accountability</li><li><strong>Data Contracts: </strong>Enforce semantic and relationship standards between data tables</li><li><strong>Data Tests:</strong> Eliminate errors and ensure high data quality by running data tests throughout the entire pipeline</li><li><strong>Templates:</strong> End-to-end templating to enforce standards across different pipelines</li><li><strong>Observability &amp; Monitoring: </strong>Complete observability and monitoring of the whole pipeline through anomaly detection and alerts</li><li><strong>Data Lineage:</strong> Column-level data lineage that helps visualize the flow of data across the entire pipeline, identifying the root causes of performance and quality issues</li></ul><h3>Collaboration</h3><p>Introducing collaboration best practices fosters knowledge sharing and alignment, supporting the co-creation of high-quality data pipelines. Y42 makes collaboration possible with the following features:</p><ul><li><strong>Figma-like Canvas: </strong>Commenting function as well as sticky notes, text, and drawing capabilities available across the app</li><li><strong>Version Control: </strong>The world’s first Git engine that works with local changes in the browser and on the local machine</li><li><strong>Data Catalog:</strong> Enables users to quickly discover data assets, definitions, and metrics</li></ul><h3>The future of the data industry is now</h3><p>Solving the core challenges currently present in the data space is Y42’s main goal. We’re here to revolutionize the data industry and the Modern DataOps Cloud is our first <em>big bang</em>.</p><p>We envision a future where every organization — whether it has one single data engineer, data analyst, or a whole data team — is able to deploy production-grade data pipelines efficiently and consume data in any downstream application to make better business decisions.</p><p>Since no one else in the data space is currently tackling the DataOps issues of inaccessibility, broken collaboration, and lack of data governance through the simultaneous implementation of software engineering and product management best practices, we at Y42 see it as our duty to help our customers bridge this gap.</p><p>The technical innovations we’ve outlined here are just the beginning. We can’t wait to keep expanding the capabilities of our product and continuously enable organizations to work with data in the most efficient and streamlined way possible.</p><p><a href="https://www.y42.com/demo?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=why_first_dataops_article">But for now, see for yourself what the Modern DataOps Cloud is capable of.</a></p><p><em>Originally published at </em><a href="https://www.y42.com/blog/why-first-modern-dataops-cloud"><em>https://www.y42.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9b87f4f1231" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything that’s wrong with the current data landscape]]></title>
            <link>https://hung-dang.medium.com/everything-thats-wrong-with-the-current-data-landscape-9b2f4b4690ee?source=rss-72d7b6f6b8c7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b2f4b4690ee</guid>
            <category><![CDATA[industry-data]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[data-tooling]]></category>
            <category><![CDATA[dataops]]></category>
            <category><![CDATA[data-landscape]]></category>
            <dc:creator><![CDATA[Hung Dang]]></dc:creator>
            <pubDate>Tue, 08 Nov 2022 16:43:53 GMT</pubDate>
            <atom:updated>2022-11-08T16:44:20.727Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KJgAszwNidwiL4SV.png" /></figure><p>The recent advent of cloud data warehousing (such as <a href="https://cloud.google.com/bigquery">BigQuery</a> and <a href="https://www.snowflake.com/en/">Snowflake</a>) has democratized working with large volumes of data for all types of companies. What used to be possible only for big corporations is now accessible to a wider audience thanks to <em>pay-as-you-need </em>cloud computing and storage solutions. These solutions tend to be more cost-effective and performant than the alternative of purchasing and maintaining an in-house data warehouse infrastructure.</p><p>As an early adopter of data warehouses myself and having already implemented BigQuery in my previous data analytics venture, I’ve always been excited to see the industry move toward more<strong> democratized</strong> and <strong>efficient</strong> data use. I was convinced that these innovations would help data teams move closer to the ultimate goal:</p><p>Having <strong>access </strong>to <strong>reliable</strong>, <strong>high-quality</strong>, and <strong>high-performing </strong>data (tables) at a <strong>low cost</strong> ensures that organizations make the best possible <strong>business decisions.</strong></p><p>But as time progressed, the initial hype turned into a sober realization that this new wave of disruption was still in its infancy. Implementing a data warehouse wouldn’t bridge the gap to make data pipelines accessible, reliable, and production-ready. At least not on its own.</p><p>Just to clarify, a data pipeline is production-ready when it has been fully tested for errors, continuously monitored, and optimized for cost. If you can tick those boxes, then your pipeline is ready to be deployed and used by any user or tool.</p><p>Why are production-ready data pipelines a necessity though? Well, the use case for data has moved beyond ad-hoc reporting (i.e., creating reports as needed) to become a company’s lifeblood, which today includes key decision-making, business-critical machine learning, and operational applications. Data pipelines built ad-hoc inevitably break over time and lead to an overflow of fire-fighting requests and, ultimately, mistrust in data.</p><p>In an attempt to tackle this issue, there has been a surge in specialized data tooling in recent years — tooling that caters to each of the data pipeline’s stages (integration, transformation, orchestration, visualization, etc.). It works in conjunction with the data warehouse — the so-called modern data stack (MDS).</p><p>However, after interviewing and working with hundreds of data teams, from single data analysts starting out on their data journey to established teams at big corporations, we identified three core pitfalls with the current state of the data industry when it comes to building production-ready data pipelines: <strong>accessibility</strong> of the modern data stack, lack of data <strong>governance</strong>, and broken <strong>collaboration</strong>.</p><h3>The accessibility problem</h3><p>To build a decent data infrastructure and an automated data pipeline with the modern data stack, companies need to stitch together at least five different data tools. A typical data stack could look like this:</p><ul><li>Integration: Fivetran</li><li>Modeling and transformation: dbt</li><li>Orchestration: Airflow</li><li>Monitoring and observability: Monte Carlo</li><li>Visualization: Tableau</li></ul><p>While these tools are truly best-in-class within their respective disciplines, gluing and maintaining them results in serious costs for each organization:</p><ul><li>Companies need to hire highly sought-after (i.e., expensive and hard-to-find) data engineers to build and maintain the entire stack.</li><li>The process of researching, testing, purchasing, and eventually learning how to use each of the tools is a big time investment for data teams.</li><li>Paying for the managed version of each individual tool adds up.</li><li>Silos are inevitable, as each individual tool is operated by different teams or users.</li><li>Best practices are not baked into the tools, so you need to build workarounds.</li></ul><p>This means that only highly tech-savvy companies with enough resources can leverage the full power of the MDS, which excludes the majority of the market and makes it highly inaccessible.</p><h3>The data governance problem</h3><p>Data governance refers to the enforcement of rules that reduce the margin of error for working processes, transforming a team’s working style from reactive to proactive.</p><p>Without implementing proper data governance processes, the whole data architecture becomes sub-optimal at best and completely unmaintainable at worst. With a dispersed data stack like the MDS, data governance to keep the whole tooling landscape in sync and accounted for becomes extremely difficult. Here are some of the resulting problems:</p><ul><li><strong>Lack of clear ownership: </strong>This leads to issues not getting fixed, blocking the entire organization.</li><li><strong>A stack out of sync: </strong>The version control, asset ownership, access control, and docs for different stack parts are managed separately, often leading to contradictory, invalid states.</li><li><strong>Expensive, unmaintainable code: </strong>A chain of thousands of inefficient SQL files increases your warehouse costs.</li><li><strong>Undiscoverable and duplicated data: </strong>Lack of discoverability across the whole stack leads to duplicate metrics, business logic, and data across tools.</li><li><strong>Access control unsolved: </strong>Data assets cannot be protected across the entire stack, which leads to serious downstream compliance and productivity issues.</li><li><strong>Unreliable and useless data: </strong>This occurs due to disjointed data workflows.</li></ul><h3>The collaboration problem</h3><p>Data has become indispensable for better decision-making in organizations. Introducing collaboration best practices fosters knowledge sharing and deep alignment, supporting the co-creation of high-quality data pipelines.</p><p>However, there’s still a huge gap when it comes to how cross-functional teams work together to deliver business value using data. Here are the natural issues we’ve witnessed firsthand that have truly crippled an organization’s data function:</p><ul><li><strong>Data professionals lack business context:</strong> This leads to the slow turnaround of data products and multiple iteration cycles due to constant misalignment with business users.</li><li><strong>Business users lack technical data context: </strong>This leads to unreasonable requests and friction with the data team.</li><li><strong>Lack of access for end-users: </strong>Data needs to be accessible for every business function, such as marketing, sales, product, and HR. This is not currently possible with the MDS.</li><li><strong>Shaky collaboration among data professionals: </strong>Collaboration among data engineers, data analysts, data scientists, and analytics engineers is still a somewhat unsolved problem. This is reflected in the lack of documentation and enforcement of documentation standards, among other things.</li></ul><h3>The exponential curve of complexity</h3><p>Tooling accessibility, data governance, and collaboration surface as problems for companies when data use cases and their respective complexity increase. Naturally, the business value for these use cases needs to be higher than the cost of designing and maintaining the processes to execute them. However, the process <strong>complexity</strong> and <strong>cost</strong> increase exponentially the more data use cases companies have. And the more use cases they want to tackle, the more data professionals they need to hire.</p><p>So, as soon as data teams grow from one team member to two, three, or more, organizations are faced with convoluted lines of communication (due to Brooke’s Law) and complex process dependencies.</p><p>In order to overcome these problems, as well as the accessibility, governance, and collaboration issues, implementing <strong>software and product management best practices</strong> become a <strong>necessity </strong>for companies with more than two people in their data team.</p><p>But why should we follow software engineering best practices? Simply put, they help create extremely high-quality and maintainable processes by minimizing failure, automating systems, and following certain failure recovery steps. Software engineering best practices include:</p><ul><li>Git — version control</li><li>Audit</li><li>Change management</li><li>Abstraction</li><li>Reusability</li><li>DRY (don’t repeat yourself) philosophy</li><li>Tests (minimize the chance of failure)</li><li>Observability (monitoring, alerts, rollbacks)</li></ul><p>However, data management efficiency is also a matter of architecture, and that’s where product management best practices come in. They help to center processes around human collaboration, enabling goal-setting alignment between cross-functional teams. These activities result in a scalable architecture, as goals and expected business value are not subject to constant change. The learnings to take from product management are:</p><ul><li>Goal alignment through collaboration enhancement (ideation canvas, chat communication, code / no code, etc.)</li><li>Knowledge sharing (documentation, catalog)</li><li>Governance (access control, audit, lineage)</li></ul><p>How are current data management solutions navigating these efficiency dilemmas? The modern data stack only covers software engineering best practices, like data tests or monitoring<em>. </em>But it’s falling behind in terms of enabling collaboration, implementing governance practices, and offering things like data catalogs or data contracts<em>. </em>This means no modern data solution out there is taking advantage of the power of product management.</p><p>So, in order to achieve the desired goal of building <strong>production-ready data pipelines that deliver reliable and high-quality data to every downstream user or application</strong>, the data industry needs to embrace software, collaboration, and governance best practices <em>simultaneously, as first-class citizens. </em>A world where data, software, and product collide.</p><p>The question that remains is, <em>who’s going to take the data industry there?</em></p><p><em>Originally published at </em><a href="https://www.y42.com/blog/data-industry-problems"><em>https://www.y42.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b2f4b4690ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>