<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Data Nerd on Medium]]></title>
        <description><![CDATA[Stories by Data Nerd on Medium]]></description>
        <link>https://medium.com/@dev.studio.ua?source=rss-212b1cc38cd6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QvtNOgJRXP-w-O2of7oiag.png</url>
            <title>Stories by Data Nerd on Medium</title>
            <link>https://medium.com/@dev.studio.ua?source=rss-212b1cc38cd6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:02:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dev.studio.ua/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Council 2025 Unpacked: Key Insights & Future Trends for Engineers]]></title>
            <link>https://medium.com/@dev.studio.ua/data-council-2025-unpacked-key-insights-future-trends-for-engineers-9e607113e89d?source=rss-212b1cc38cd6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e607113e89d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-council]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Data Nerd]]></dc:creator>
            <pubDate>Wed, 11 Jun 2025 18:20:39 GMT</pubDate>
            <atom:updated>2025-06-11T18:20:39.568Z</atom:updated>
            <content:encoded><![CDATA[<h3>I. Introduction: The Pulse of Modern Data</h3><p>Data Council ’25, held in Oakland, CA from April 22–24 , once again solidified its reputation as a premier technical event. It serves as a crucial gathering for those at the forefront of artificial intelligence and the data stacks that power it, including AI engineers, founders, CTOs, AI researchers, Heads of Data, and investors. This post aims to distill key takeaways from the conference, highlighting novel advancements and offering deeper findings for software and data engineers looking to stay ahead. The conference’s “NO BS DATA CONF” ethos resonates with a clear industry understanding: every significant AI breakthrough indeed starts with robust, well-engineered data. This foundational role of data engineering, often the unsung hero of the AI revolution, was a recurring undercurrent in many discussions. Major themes that emerged include the tightening symbiosis of AI and data infrastructure, the relentless push for real-time systems, the evolution of database architectures, practical advancements in AI/ML engineering, and the ongoing challenge of orchestrating complex systems at scale.</p><h3>II. Theme 1: When Data Meets Intelligence — The AI-Infra Symbiosis</h3><p>The convergence of data infrastructure and artificial intelligence is no longer a futuristic concept but a present-day reality shaping enterprise technology.</p><h4>A. Keynote: Data Meets Intelligence: Where the Data Infra &amp; AI Stack Converge</h4><p>Naveen Rao, VP of AI at Databricks, delivered a keynote emphasizing that the traditional boundaries between data infrastructure and AI systems are rapidly blurring. The session explored this critical convergence, examining its profound impact on enterprise technology stacks. Key discussion points revolved around the technical and strategic considerations for building modern, integrated data and AI platforms. It was highlighted how this convergence is pivotal for enabling scalable AI adoption and fostering new architectural patterns across industries. For engineers, this signals a fundamental shift: designing data systems in isolation from AI workloads, or vice-versa, is an increasingly outdated approach. The “blurring boundaries” suggest a move towards deeply integrated platforms, demanding engineers who possess, if not hybrid skill sets, then at least a strong reciprocal understanding of data management principles and AI operational needs. This trend is likely to spur the development of new tools and platforms that are natively data-and-AI aware.</p><h4>B. Deep Dive: The Future of Data Engineering in the Agent Era</h4><p>A panel featuring Nuno Campos (LangChain), Gleb Mezhanskiy (Datafold), Madison Faulkner (NEA), and Diptanu Gon Choudhury (Tensorlake) explored “The Future of Data Engineering in the Agent Era”. Their discussion underscored how AI agents and foundation models are set to revolutionize data engineering, particularly through advancements in real-time data processing, sophisticated agent memory systems, and novel knowledge representation techniques like semantic layers, vector embeddings, and graph RAG systems.</p><p>This isn’t merely about employing AI; it’s about how AI fundamentally reshapes the discipline of data engineering. The concept of “dynamic knowledge graphs,” for instance, points to a departure from rigid, static schemas towards more fluid, interconnected data representations that AI agents can autonomously navigate and utilize. The development of “agent memory” architectures introduces new components that data engineers will be tasked with building and maintaining. This “Agent Era” necessitates a paradigm shift where data transforms from a passive repository into an active, intelligent participant in automated systems. Consequently, data modeling must evolve to ensure data is not just queryable but also “understandable” and “actionable” by AI agents, blurring the lines with knowledge engineering. This heightened reliance on AI agents places an even greater premium on data quality and observability — areas of expertise for panelists like Gleb Mezhanskiy — as autonomous actions driven by flawed data can have significant negative consequences.</p><h3>III. Theme 2: Building for Now and Next — Real-Time Systems &amp; Evolving Data Architectures</h3><p>The demand for immediate insights and adaptable data platforms continues to drive innovation in real-time systems and foundational data architectures.</p><h4>A. Keynote: Real-Time Data Infrastructure and AI: Powering the Next Generation of Analytics</h4><p>Aaron Katz, Co-Founder &amp; CEO of Clickhouse, and Martin Casado, General Partner at a16z, engaged in a fireside chat designed to cut through industry hype surrounding real-time data infrastructure. They delved into the technical hurdles of processing data at scale in real-time and the critical architectural decisions that enable performance breakthroughs. The discussion covered the evolution of data processing, bottlenecks in distributed systems, innovations in observability, and the delicate balance between consistency and throughput. Martin Casado’s background in pioneering software-defined networking brought a deep distributed systems perspective to the conversation.</p><p>The emphasis on “honest assessments of implementation challenges” was particularly noteworthy. It suggests that despite significant advancements, constructing robust, scalable real-time data infrastructure, especially when integrated with AI, remains a formidable engineering challenge requiring careful trade-offs. Achieving genuine real-time performance at scale involves complex architectural decisions, reinforcing that there are no universal solutions and that profound system design expertise is indispensable.</p><h4>B. Deep Dive: The Deconstructed Database and the Advent of the Open Data Lake</h4><p>Julien Le Dem, Principal Engineer at Datadog and co-creator of pivotal open-source projects like Parquet, Arrow, and OpenLineage , revisited his influential 2018 “Deconstructed Database” concept. His talk, “The Deconstructed Database and the Advent of the Open Data Lake,” detailed how the big data ecosystem has matured from a collection of competing projects into a more composable system organized around open-source standards such as Parquet, Arrow, Iceberg, Calcite, Substrait, and OpenLineage. These tools provide an interoperability layer, enabling data to be harnessed for many purposes without creating silos.</p><p>The critical insight is the <em>realization</em> of this vision: modern data platforms are increasingly built on these principles. Le Dem highlighted the impact of cloud computing, which enables the efficient decoupling of compute and storage, fostering a common storage layer. This, in turn, supports a vibrant ecosystem of specialized, on-demand tools, crucially helping organizations avoid vendor lock-in. The “contracts” that ensure these components remain decoupled and composable are key for engineers designing flexible and future-proof systems. This shift empowers engineers to select best-of-breed components, building more specialized, efficient, and cost-effective data stacks. Understanding these core standards (Arrow, Iceberg, etc.) is rapidly becoming essential for data engineers.</p><h4>C. Deep Dive: Building InfluxDB 3 Core: A Real-Time Columnar DB and Data Processor on Object Storage</h4><p>Paul Dix, Founder &amp; CTO of InfluxData, presented “Building InfluxDB 3 Core: A Real-Time Columnar DB and Data Processor on Object Storage”. This talk offered a concrete example of the “Deconstructed Database” principles applied to the demanding domain of time-series data. InfluxDB 3 Core represents a ground-up rewrite in Rust, leveraging Apache Arrow, DataFusion, and Parquet. It features an innovative diskless architecture that utilizes object storage for persistence, complemented by a sophisticated caching system to enable real-time data ingestion and querying.</p><p>Further architectural details from Dix’s related blog post on the InfluxDB 3.0 GA reveal a commitment to open standards and modern cloud-native patterns. The “FDAP Stack” (Apache Flight, DataFusion, Arrow, Parquet), separation of compute and storage, stateless operation for easier scaling, and multi-AZ durability are key features. A particularly novel aspect is the embedded Python VM, which transforms the database into a comprehensive data collector, monitoring agent, and data transformation platform, capable of processing data via WAL Flush, scheduled triggers, or on-demand HTTP requests. This architecture demonstrates how mature open-source components can be used to build highly specialized, performant systems, accelerating innovation. The embedded processing capability also signals a trend towards pushing computation closer to the data, reducing movement and enabling more reactive systems.</p><h3>IV. Theme 3: The Engineer’s Edge — Cutting-Edge AI/ML Applications</h3><p>Practical applications and the underlying engineering of AI and machine learning systems were also key focal points.</p><h4>A. Deep Dive: RAG In 2025: State Of The Art And The Road Forward</h4><p>Tengyu Ma, Co-Founder &amp; CEO of Voyage AI, delivered a forward-looking talk on “RAG In 2025: State Of The Art And The Road Forward”. The session explored the future trajectory of Retrieval-Augmented Generation (RAG) technology, focusing on its evolution beyond current limitations. Key to this evolution are advancements in embedding models and rerankers, which are paving the way for more robust and scalable RAG solutions. A significant theme was the transition from heuristic-heavy, often brittle, RAG pipelines to more automated solutions capable of delivering higher-quality, more relevant results with reduced manual optimization.</p><p>This anticipated maturation of RAG, driven by improvements in its underlying components, suggests that RAG is moving from an experimental technique to a more engineered, dependable component in the AI toolkit. For engineers building or considering RAG systems, this talk offered a glimpse into how these systems will become more sophisticated, scalable, and maintainable. As RAG systems mature, the engineering focus will likely shift from basic pipeline construction to the nuanced optimization and evaluation of these increasingly automated systems.</p><h4>B. Highlight: What Every Data Scientist Needs To Know About GPUs</h4><p>Charles Frye, Developer Advocate at Modal Labs, presented “What Every Data Scientist Needs To Know About GPUs”. This session provided a comprehensive guide to GPU architecture and optimization tailored for modern machine learning workloads. It covered critical concepts from hardware fundamentals to high-level frameworks, with a practical focus on performance tuning for neural networks and optimizing system latency and throughput in popular ML frameworks like PyTorch, vLLM, and RAPIDS. Frye’s broader expertise also covers various compute options (CPUs, GPUs, TPUs, LPUs) and inference server technologies.</p><p>The very necessity of such a talk highlights a common scenario: many practitioners utilize GPU-accelerated libraries without a deep understanding of how to maximize their efficiency, potentially leading to suboptimal resource utilization. As models grow larger and computational costs escalate, the ability to efficiently leverage GPU capabilities becomes a significant economic and performance differentiator. This session aimed to empower data scientists and ML engineers to look “under the hood,” enabling them to write more efficient ML code and make more informed infrastructure decisions.</p><h3>V. Theme 4: Taming Complexity — Orchestration at Scale &amp; Responsible AI</h3><p>Managing the inherent complexity of large-scale data systems and ensuring responsible AI development remain paramount.</p><h4>A. Deep Dive: Orchestrating at Scale: How Instacart Manages 20M+ Daily Workflows</h4><p>Anant Agarwal from Instacart shared invaluable insights into “Orchestrating at Scale: How Instacart Manages 20M+ Daily Workflows”. The core of Instacart’s strategy involves a YAML-defined abstraction layer built on top of powerful workflow engines like Temporal and Airflow. This abstraction enables them to manage a staggering volume of over 20 million daily workflows. The key takeaway was that investing in such abstractions and adopting opinionated configurations yields significant dividends when operating large-scale platforms. This approach empowers internal teams (tenants) to scale out their diverse workflows effectively and autonomously.</p><p>Instacart’s YAML abstraction layer serves as a compelling architectural pattern for managing complexity at scale. It allows various teams to define their workflows in a standardized, simplified manner, while a central platform team handles the intricacies of the underlying execution engines. This effectively democratizes workflow creation while maintaining crucial central control, platform stability, and operational consistency — a strong endorsement for the value of internal developer platforms.</p><h4>B. Brief Mention: Guardrails for the Future: AI Safety and Responsible AI in Practice</h4><p>Jake Brill, Head of Product — Integrity at OpenAI, delivered a keynote on “Guardrails for the Future: AI Safety and Responsible AI in Practice”. This session aimed to move beyond purely theoretical discussions of AI ethics to explore the practical realities and challenges of implementing responsible AI safeguards in real-world systems. The presence of such a keynote from a leading AI organization signals an industry-wide shift from discussing <em>why</em> responsible AI is important to focusing on <em>how</em> it can be tangibly implemented. This indicates a growing expectation for data and software engineers to integrate these practical safeguards into their development lifecycles.</p><h3>VI. Spotlight on Novelty: Most Promising Takeaways for Engineers</h3><p>Several particularly novel and promising themes for software and data engineers emerged from Data Council ‘25:</p><ol><li><strong>The Maturing Composable Data Stack:</strong> Julien Le Dem’s presentation powerfully confirmed that the vision of a deconstructed, composable data stack — built upon robust open standards like Apache Arrow, Parquet, and Apache Iceberg — is no longer just a theoretical ideal but a practical reality. This paradigm empowers engineers to construct flexible, best-of-breed systems, effectively mitigating vendor lock-in and fostering innovation.</li><li><strong>Next-Generation Database Architectures:</strong> Paul Dix’s work on InfluxDB 3 Core exemplifies a modern, cloud-native approach to database design. The strategic use of Rust for performance, the adoption of the Arrow/Parquet/DataFusion ecosystem, the leveraging of object storage for persistence, and the inclusion of embedded processing capabilities signal a significant trend. Databases are evolving into highly performant, specialized engines that offer far more than just data storage, increasingly built from and contributing to open-source building blocks.</li><li><strong>Pragmatic AI Engineering:</strong> Talks such as Tengyu Ma’s on the future of RAG and Anant Agarwal’s on Instacart’s large-scale workflow orchestration highlight a clear industry movement. There’s a strong shift towards more robust, scalable, and manageable AI and data workflow solutions, with an emphasis on automation, practical abstractions, and engineering discipline over purely experimental or ad-hoc setups.</li></ol><p>Collectively, these takeaways point towards an overarching trend: the increasing industrialization and application of rigorous engineering principles to both data infrastructure and AI application development. This maturation phase emphasizes reliability, scalability, and efficiency in production environments.</p><h3>VII. Conclusion: Engineering the Future with Data</h3><p>Data Council ’25 provided a compelling snapshot of a field in dynamic evolution. It’s clear that data engineering sits at the absolute heart of AI innovation and is itself being rapidly reshaped by its symbiotic relationship with AI, the emergence of new architectural paradigms like composability and real-time responsiveness, and the persistent need for scalable, responsible solutions.</p><p>The discussions and technologies showcased offer a valuable window into the trends that will define the next generation of data and AI systems. For software and data engineers, the overarching message is one of continuous learning and adaptation. The skills and knowledge in this domain have a relatively short half-life, demanding proactive engagement with new concepts and tools. Engineers are encouraged to explore the talks further as they become available on the(<a href="https://www.youtube.com/playlist?list=PLAesBe-zAQmFUeS0gMFSII4m-Zw4CoOoE">https://www.youtube.com/playlist?list=PLAesBe-zAQmFUeS0gMFSII4m-Zw4CoOoE</a>) and to critically consider how these insights can inform their own projects, drive innovation, and shape their careers in this exciting and pivotal field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e607113e89d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Summit 2024: Lakeflow Takes Center Stage in Data Orchestration Evolution]]></title>
            <link>https://medium.com/@dev.studio.ua/databricks-summit-2024-lakeflow-takes-center-stage-in-data-orchestration-evolution-0b37c735a78e?source=rss-212b1cc38cd6------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b37c735a78e</guid>
            <category><![CDATA[data-orchestration]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[analytics-platforms]]></category>
            <dc:creator><![CDATA[Data Nerd]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 21:47:37 GMT</pubDate>
            <atom:updated>2025-06-05T21:47:37.487Z</atom:updated>
            <content:encoded><![CDATA[<p>The imperative for robust, scalable, and efficient data orchestration has never been more critical as organizations increasingly rely on data and AI to drive innovation. However, many data teams grapple with fragmented tooling, operational complexities, and the challenge of ensuring data reliability across the entire data lifecycle. The <a href="https://dataaisummit.databricks.com/flow/db/dais2024/landing/page/home">Data and AI symmit</a>, held in San Francisco from June 9–12 , served as a pivotal platform where Databricks unveiled significant advancements in its data engineering capabilities, with a pronounced focus on unifying and simplifying data orchestration. Featuring over 700 sessions and insightful keynotes (), the summit underscored Databricks’ commitment to evolving its <a href="https://www.databricks.com/">unified platform for data, analytics, and AI</a>.</p><p>A clear theme emerging from the summit was the market’s strong appetite for consolidated solutions that reduce the “tool sprawl” often associated with complex data pipelines. Databricks’ strategic announcements, particularly around data orchestration, directly address this need, aiming to provide a more cohesive and streamlined experience for data practitioners.</p><h3>Unveiling Databricks Lakeflow: A New Era for Data Engineering</h3><p>The centerpiece of Databricks’ data orchestration announcements at the summit was the introduction of <strong>Databricks Lakeflow</strong>. Positioned as a unified, intelligent solution for the entire data engineering lifecycle, Lakeflow is designed to help users ingest data from diverse sources, transform it for both batch and real-time streaming, and confidently deploy and manage data pipelines in production. This comprehensive approach was detailed in various summit keynotes and sessions (see keynotes <a href="https://www.youtube.com/playlist?list=PLTPXxbhUt-YXh57uL3QUQiKLwTf39IoT1">here</a> and a preview <a href="https://www.databricks.com/product/data-engineering/data-ingestion">here</a> ).</p><p>Lakeflow represents a strategic consolidation and evolution of existing Databricks strengths, rather than an entirely new set of disparate tools. It integrates and enhances the capabilities of established features like Delta Live Tables and Databricks Workflows under a single, more comprehensive umbrella. This move is indicative of Databricks’ focus on creating an integrated environment that simplifies development and operations for data teams. Furthermore, the infusion of AI-powered intelligence into Lakeflow, with features like an AI assistant for pipeline discovery, authoring, and monitoring, signals a trend towards leveraging AI not just as an output of data pipelines but as an integral part of their creation and management.</p><p>Lakeflow is structured around three core components :</p><h3>A. Lakeflow Connect: Ingesting Data with Ease</h3><p>The critical first step in any data pipeline is ingestion, and Lakeflow Connect aims to simplify this process significantly. It provides native, scalable connectors for a wide array of sources, including databases (like MySQL, PostgreSQL, SQL Server, Oracle), enterprise applications (such as Salesforce, Workday, Google Analytics), and cloud storage.</p><p>Key features of Lakeflow Connect include a simple UI and API, enabling even users with limited coding experience to set up ingestion pipelines, thereby saving valuable programming hours. It supports incremental reads and writes for efficiency, operates as a fully managed service to reduce infrastructure overhead, and integrates deeply with Unity Catalog for unified governance from the point of ingestion. This approach effectively lowers the barrier to data ingestion, allowing for faster onboarding of new data sources into the lakehouse.</p><h3>B. Lakeflow Pipelines: Building Intelligent ETL (Leveraging DLT principles)</h3><p>For data transformation, Lakeflow Pipelines offers an efficient, declarative approach. This component is fundamentally built upon the robust and proven <strong>Delta Live Tables (DLT)</strong> framework. Users can define their business logic using SQL or Python, and Lakeflow Pipelines automates the underlying orchestration, compute scaling, and data quality monitoring.</p><p>It supports both batch and streaming data, facilitates efficient incremental updates, and includes built-in data quality monitoring. A notable feature is the Real-Time Mode, designed for low-latency delivery of time-sensitive datasets. For existing DLT users, Lakeflow Pipelines offers a familiar yet more integrated experience, while new users benefit from a mature declarative ETL engine embedded within the broader Lakeflow ecosystem.</p><h3>C. Lakeflow Jobs: Orchestrating the Modern Data Workflow (Evolution of Databricks Workflows)</h3><p>To manage and monitor production workloads, Lakeflow Jobs provides reliable orchestration for a wide range of tasks, including notebooks, SQL queries, machine learning model training, and dashboard updates. This component is an evolution of <strong>Databricks Workflows</strong>, inheriting and expanding upon its existing capabilities.</p><p>Lakeflow Jobs offers full observability into pipeline execution, advanced control flow capabilities such as data-driven triggers (reacting to table and file arrivals), branching, and looping, and mechanisms for prompt issue detection and resolution. Recent enhancements to Workflows, now part of Lakeflow Jobs, include AI-assisted workflow creation, scalability to 1,000 tasks per job, and integrated cost tracking through system tables. This signifies a maturation of Databricks’ orchestration capabilities, moving towards more event-driven, operationally aware, and efficient pipeline management.</p><p><strong>Databricks Lakeflow: Components Overview</strong></p><p>ComponentCore FunctionKey Aspects Inherited/Evolved FromKey New Aspects/IntegrationsLakeflow ConnectData IngestionAuto Loader, COPY INTO principles UI-driven, broad enterprise connectors, fully managed service Lakeflow PipelinesDeclarative Data Transformation (ETL/ELT)Delta Live Tables (DLT) Tighter integration with Connect &amp; Jobs, Real-Time Mode Lakeflow JobsWorkflow Orchestration &amp; MonitoringDatabricks Workflows Enhanced observability within Lakeflow, AI-assisted features</p><h3>Powering the Pipelines: Foundational Enhancements</h3><p>These advancements in orchestration are underpinned by significant updates to Databricks’ core technologies:</p><ul><li><strong>Delta Lake 4.0:</strong> This release brings improved performance, Liquid Clustering for advanced data organization, expanded data type support including an open Variant type, and enhanced governance features. Delta UniForm continues to drive interoperability with Apache Iceberg.</li><li><strong>Apache Spark 4.0:</strong> The underlying engine gets a boost with features like ANSI mode by default, a new VARIANT data type, improved streaming capabilities, and the general availability of Spark Connect for more flexible remote connectivity.</li><li><strong>Unity Catalog:</strong> Its role in providing unified governance — covering access control, data lineage, and metadata management across all Lakeflow-managed assets — is crucial. The summit also highlighted the full open-sourcing of Unity Catalog, promoting a more collaborative and secure data environment. This deep integration ensures that governance is an intrinsic part of the data engineering lifecycle, not an afterthought.</li></ul><h3>Impact and Benefits for Data Teams</h3><p>The introduction of Lakeflow and related enhancements translates into tangible benefits:</p><ul><li><strong>Simplified Development:</strong> A unified environment, declarative approaches, and AI assistance reduce complexity and development time.</li><li><strong>Improved Reliability:</strong> Built-in monitoring, data quality rules, and automated error handling contribute to more robust pipelines.</li><li><strong>Enhanced Collaboration:</strong> A shared platform fosters seamless teamwork among data engineers.</li><li><strong>Scalability and Performance:</strong> Leveraging serverless compute, Spark 4.0, and Delta Lake 4.0 ensures pipelines can handle growing data volumes and processing demands.</li><li><strong>Faster Time-to-Insight:</strong> Streamlining the data engineering lifecycle accelerates the delivery of actionable insights.</li><li><strong>Cost Efficiency:</strong> Serverless options and optimized resource utilization can lead to better cost management.</li></ul><p>The emphasis on declarative and managed services within Lakeflow signifies a shift towards abstracting infrastructure complexity, allowing data teams to focus more on delivering business value.</p><h3>Conclusion: Orchestrating a Data-Intelligent Future</h3><p>The Databricks Data &amp; AI Summit 2024 marked a significant step forward in the evolution of data orchestration. With Lakeflow, Databricks is clearly aiming to provide a unified, intelligent, and simplified experience for data engineering, built upon an increasingly powerful and open foundational platform. These advancements are poised to empower organizations to build and manage their data and AI workflows with greater efficiency, reliability, and governance, accelerating their journey towards a data-intelligent future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b37c735a78e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airflow Summit 2024: Powering Next-Gen Analytics Platforms & Data Infrastructure]]></title>
            <link>https://medium.com/@dev.studio.ua/airflow-summit-2024-powering-next-gen-analytics-platforms-data-infrastructure-9852482358ed?source=rss-212b1cc38cd6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9852482358ed</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-orchestration]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Data Nerd]]></dc:creator>
            <pubDate>Fri, 30 May 2025 22:14:36 GMT</pubDate>
            <atom:updated>2025-05-30T22:14:36.915Z</atom:updated>
            <content:encoded><![CDATA[<p>The 2024 Airflow Summit marked a significant juncture for Apache Airflow, celebrating a decade of innovation and community growth. This milestone underscores a period of extensive learning and real-world application that has solidified Airflow’s position as the industry standard for data orchestration. The summit showcased not just an evolution, but a transformation, with Airflow poised to extend its reach beyond traditional data engineering into broader application and infrastructure orchestration. This shift suggests that engineers should increasingly view Airflow as a strategic platform component, capable of unifying diverse operational workflows across the entire data and analytics stack. This distillation of the summit’s key announcements and discussions focuses on the developments most pertinent to software engineers building and maintaining today’s complex analytics platforms and data infrastructure.</p><h3>The Main Event: Airflow 3.0 Unveiled — A Paradigm Shift for Engineers</h3><p>The centerpiece of the summit was the unveiling of Apache Airflow 3.0, heralded as the most impactful release in the project’s history. This major version introduces a suite of features designed to address long-standing community requests and propel Airflow into new frontiers of orchestration. The community’s involvement is actively sought, with calls for beta testers and contributors to help shape this next generation of Airflow.</p><p>The architectural enhancements in Airflow 3.0 are substantial. The introduction of the Task Execution Interface (AIP-72), for instance, signals a move towards a client-server model. This not only facilitates running tasks in multiple languages but, when combined with inherent event-driven capabilities (AIP-82), positions Airflow as a potential backend engine for dynamic, polyglot data applications and services. This could lead to new architectural patterns where Airflow is embedded more deeply within application stacks, moving beyond its traditional role as a separate batch processing system, making talks like (<a href="https://www.youtube.com/watch?v=ounsKvoGeUY"> An Intro to Apache Airflow</a>) particularly relevant.</p><p>Key Airflow 3.0 features particularly relevant for data and analytics engineers include several advancements. <strong>DAG Versioning</strong> (related to AIP-65 and AIP-66) brings improved change tracking, history, rollback capabilities, and easier debugging of DAGs. The ability to <strong>Run Anywhere, Any Language</strong> (related to AIP-72) allows engineers to leverage non-Python code and libraries, fostering broader team skill integration through a client-server architecture. <strong>Event-Driven capabilities and Data Assets</strong> (related to AIP-74, AIP-75, and AIP-82) enable more reactive and efficient pipelines, better data lineage and awareness, and reduced latency. <strong>Scheduler-Managed Backfills</strong> (related to AIP-78) offer more scalable, reliable, and diagnosable data reprocessing. The <strong>Task Execution Interface</strong> (AIP-72) is foundational for remote execution and enables multi-cloud, multi-language operations. Finally, a <strong>Revamped UI</strong> promises improved usability, a modern interface, and a better debugging experience.</p><h3>A. DAG Versioning: Finally, Robust Change Management</h3><p>One of the most anticipated features, DAG versioning, is now a reality in Airflow 3.0. Previously, structural changes to DAGs, such as removing a task, could lead to a loss of historical run information in the UI. DAG versioning, underpinned by AIP-65 (Improve DAG history) and AIP-66 (DAG Bundles and Parsing), addresses this by allowing users to maintain a full history of DAG changes. This is particularly powerful when combined with Git DAG bundles, enabling the ability to rerun older versions of a DAG. For engineering teams managing complex analytics platforms, this feature is critical for stability, auditability, and simplified debugging, especially in environments with multiple developers contributing to DAGs. While complexities around clearing tasks with versioning are still being refined, as discussed in (<a href="https://www.youtube.com/watch?v=Di1nhrr4-VM">AIP 63: DAG versioning, where are we?</a>) the overall improvement to change management is substantial.</p><h3>B. “Run Anywhere, In Any Language”: Expanding Horizons</h3><p>Airflow 3.0 breaks the Python-only barrier for task execution. Spearheaded by the Task Execution Interface (AIP-72), Airflow can now orchestrate tasks written in other languages, with Golang being an early example, as detailed in (<a href="https://www.youtube.com/watch?v=qN0BJNR_GW0">Running Airflow Tasks Anywhere, in any Language</a>). This represents one of the most significant architectural shifts in Airflow’s history, enabling a client-server architecture that supports multi-cloud deployments and allows teams to leverage existing codebases or the specific strengths of other programming languages without cumbersome workarounds. This extensibility broadens Airflow’s applicability, allowing analytics platform teams to more seamlessly integrate specialized tools and libraries, regardless of their native language. The community is actively encouraged to contribute support for additional languages.</p><h3>C. Event-Driven Architecture &amp; Data Assets: Towards Reactive Pipelines</h3><p>A major leap in Airflow 3.0 is its inherent understanding of event-driven workflows, a topic explored in (<a href="https://www.youtube.com/watch?v=4EU7M_E5pMw">Airflow Datasets and Pub/Sub for Dynamic DAG Triggering — Airflow Summit 2024</a>). This is built upon the evolution of Datasets into more comprehensive “Data Assets” (AIP-74) and mechanisms for external event-driven scheduling (AIP-82). Data Assets are designed to represent any data product of significance, be it datasets, ML models, reports, or dashboards, bringing clearer visibility into Airflow, as highlighted in (<a href="https://www.youtube.com/watch?v=2suK4F6JHLE"> Seeing Clearly with Airflow: The shift to data-aware orchestration</a>). Combined with a new asset-centric syntax (AIP-75), this allows pipelines to become more reactive, triggering tasks based on actual data events rather than relying solely on fixed time-based schedules. For analytics platforms, this translates to more responsive data processing, reduced latency in delivering insights, and potentially lower operational costs by optimizing resource usage.</p><h3>D. Enhanced Backfills and UI Improvements</h3><p>Airflow 3.0 also delivers significant improvements to backfill operations. Scheduler-managed backfills (AIP-78) integrate backfill processing directly with the scheduler, leading to enhanced scalability and better diagnostic capabilities. This is crucial for data reprocessing and historical analysis. Alongside these backend improvements, the Airflow UI is undergoing a substantial rewrite, aiming to provide a more modern and intuitive user experience, with details shared in the (<a href="https://www.youtube.com/watch?v=nq_FiCWuLfM"> Airflow UI Roadmap</a>). Community feedback is actively being solicited to shape this new interface, including through initiatives like the Airflow debugging improvement survey mentioned during the (<a href="https://www.youtube.com/watch?v=GYJL1kOKFz0">Airflow 3 Roadmap Discussion</a>).</p><h3>Engineering for Scale: Performance, Efficiency, and Robust Operations</h3><p>The summit highlighted a strong focus on enabling Airflow to operate efficiently at massive scale, a critical concern for engineers managing enterprise-grade analytics platforms.</p><h3>A. Mastering Scalability and Configuration</h3><p>Sessions delved into the intricacies of Airflow configuration for optimal scalability, such as in (<a href="https://www.youtube.com/watch?v=cd37h7bm8cc">A deep dive into Airflow configuration options for scalability</a>). Discussions emphasized the importance of tuning parameters related to DAG parsing, scheduler operations (<a href="https://www.youtube.com/watch?v=BjWx2r3P3GA">Investigating the Many Loops of the Airflow Scheduler</a>), and worker management to prevent bottlenecks and optimize resource utilization in large-scale deployments. For instance, inefficient DAG parsing can lead to increased CPU usage, impacting overall performance. The experiences of companies like Bosch, which scaled its Airflow environment to handle 1.2 million DAG runs per day, as detailed in (<a href="https://www.youtube.com/watch?v=2wJrsPaPLSo">How we Tuned our Airflow to Make 1.2 Million DAG runs per day</a> ), underscored the necessity of meticulous tuning of active DAG runs, task concurrency, scheduler and worker counts, and database performance. Bosch even contributed patches back to the Airflow project to address specific scaling challenges. Similarly, Coinbase shared its success in running millions of tasks daily with a relatively modest hardware footprint in (<a href="https://airflowsummit.org/sessions/2024/airflow-unleashed-making-hundreds-of-deployments-a-day-at-coinbase/">Airflow Unleashed: Making Hundreds of Deployments A Day at Coinbase</a> ), achieved through optimized event processing and careful configuration of schedulers and workers. These examples demonstrate that while Airflow can scale significantly, it requires a deep understanding of its internal mechanics and configuration landscape.</p><h3>B. Executor Evolution: Hybrid Models and Custom Environments</h3><p>The evolution of Airflow executors continues to provide engineers with more flexibility in managing task execution. Talks like “<a href="https://www.youtube.com/watch?v=mRuPETgCQwc">Comparing Airflow Executors and Custom Environments</a>” aimed to update common understanding beyond outdated rules of thumb. A significant development is the introduction of Hybrid Executors, allowing different tasks within the same DAG to utilize different executors, as explained in (<a href="https://www.youtube.com/watch?v=3iKM28Ww73w">Hybrid Executors: Have your cake and it eat too</a>). This means, for example, that lightweight tasks could run on a LocalExecutor while resource-intensive computations are dispatched to a KubernetesExecutor, all within a single workflow. This feature, available even before the full Airflow 3.0 release, offers a powerful mechanism for optimizing resource usage and cost. Future enhancements, discussed in talks like (<a href="https://www.youtube.com/watch?v=VK-2OO5-MNA">The Essentials of Custom Executor Development</a>), are planned to allow even more granular, task-level executor configuration, further empowering engineers to tailor execution environments precisely to task requirements.</p><h3>C. Real-World Scaling: Lessons from the Trenches</h3><p>Numerous organizations shared their journeys and strategies for running Airflow at impressive scales.</p><p>Bosch’s experience scaling to 1.2 million DAG runs per day, detailed in (<a href="https://www.youtube.com/watch?v=2wJrsPaPLSo">How we Tuned our Airflow to Make 1.2 Million DAG runs per day</a>), involved tackling “event storms” by tuning schedulers, workers, and database configurations, alongside implementing custom policies for DAG event archiving and event filtering.</p><p>Coinbase showcased “AirAgent,” a bespoke system for autonomous, high-frequency deployments, handling hundreds of updates daily to their Airflow environment, which processes millions of tasks, as detailed in (<a href="https://airflowsummit.org/sessions/2024/airflow-unleashed-making-hundreds-of-deployments-a-day-at-coinbase/">Airflow Unleashed: Making Hundreds of Deployments A Day at Coinbase</a> ).</p><p>Instacart manages a centralized Airflow cluster running over 2,200 DAGs and 16 million tasks per month, utilizing Terraform for infrastructure management, YAML-based DAG definitions for abstraction, and remote executors for flexibility, as shared in (<a href="https://www.youtube.com/watch?v=QlCT3h6AxW0">Optimizing Airflow Performance: Strategies, Techniques, and Best Practices</a> ).</p><p>Uber discussed the evolution of its internal orchestrator, Piper, and its plans to converge with open-source Airflow in “Evolution of Airflow at Uber,” particularly leveraging upcoming Airflow 3.0 features for performance and event-driven scheduling.</p><p>Booking.com shared insights into their multi-tenant Airflow setup in (<a href="https://www.youtube.com/watch?v=hGhrmma9mtg">How we use Airflow at Booking to Orchestrate Big Data workflows</a> ), emphasizing the ease of using Astro’s API for infrastructure-as-code practices.</p><p>These case studies reveal a maturing landscape for Airflow operations. The trend is towards more sophisticated deployment patterns, such as centralized platforms with remote or hybrid executors (as seen with Robinhood’s architecture where central schedulers manage workloads for team-owned workers, described in (<a href="https://airflowsummit.org/sessions/2024/optimizing-critical-operations-enhancing-robinhood-s-workflow-journey-with-airflow/">Optimizing Critical Operations: Enhancing Robinhood’s Workflow Journey with Airflow</a>)), and a greater emphasis on robust CI/CD and automated management. This suggests that organizations are increasingly treating Airflow as a managed platform service, rather than deploying numerous siloed instances. Engineers building analytics platforms should consider these platform-centric approaches early, focusing on automation, abstraction, and standardized deployments to prepare for future scale and complexity. Such strategies make the prospect of managing hundreds of Airflow environments, as described by Michael Juster in (<a href="https://www.youtube.com/watch?v=GKkUtH9H57E">How we run 100 Airflow environments and millions of tasks as a part time job using Kubernete</a>), a more feasible endeavor.</p><h3>Modernizing DAGs: Development, Operations, and Intelligence</h3><p>The summit also brought forth innovative approaches to DAG development and management, aiming to enhance developer productivity and pipeline reliability.</p><h3>A. Evolving DAG Design and Best Practices</h3><p>New paradigms for DAG design are emerging. The (<a href="https://www.youtube.com/watch?v=1cqFkbKftzI">A New DAG Paradigm: Less Airflow more DAGs</a>) concept, presented by Marion Azoulai &amp; Maggie Stark from Astronomer, advocates for a shift towards dataset scheduling, micro-pipelines, and the use of “Control DAGs” for managing complex dependencies and providing end-to-end pipeline visibility. This approach aims to minimize failures and increase reliability by breaking down monolithic DAGs into smaller, more manageable, and independently schedulable units, moving away from problematic sensor-dependent setups. Other sessions emphasized DAG parameterization using the Param class for improved efficiency, type checking, and UI form generation, as detailed in (<a href="https://www.youtube.com/watch?v=HUfdzkiYYg8">Optimize Your DAGs: Embrace Dag Params for Efficiency and Simplicity — Airflow Summit 2024</a>). Foundational best practices such as task idempotency, focused task design, and patterns like fan-in/fan-out were reiterated in talks like (<a href="https://www.youtube.com/watch?v=tzW-sWllpGk">Exploring DAG Design Patterns in Apache Airflow</a>). The talk (<a href="https://www.youtube.com/watch?v=W0YDfUAQNRU">Refactoring DAGs: From duplication to delightful efficiency with a centralized library</a>) also highlighted creating centralized libraries as a means to reduce code duplication and improve maintainability.</p><h3>B. Streamlining with CI/CD, Testing, and YAML</h3><p>The critical role of robust software engineering practices in data pipeline development was a recurring theme. Sessions covered (<a href="https://www.youtube.com/watch?v=Me94KjSK8bw">Automated Testing and Deployment of DAGs</a>), the importance of building comprehensive CI systems for provider packages as shown in (<a href="https://www.youtube.com/watch?v=lWCp-r-bQEI">Hello Quality: Building CIs to run providers packages system tests</a>), and the continuous effort required to keep CI/CD pipelines and developer tools well-maintained and efficient, discussed in (<a href="https://www.youtube.com/watch?v=U4n7tuK9pGs">The Silent Symphony: Keeping Airflow’s CI/CD and Dev Tools in Tune</a>). Stripe’s approach to (<a href="https://www.youtube.com/watch?v=kKUY2Vx76-0">Stress Free Airflow Development: From dev to prod at Stripe</a>) involves using container images for different DAG versions within a single cluster, enabling isolated testing and promotion. For simplifying DAG creation, particularly in large organizations or for users less familiar with Python, the use of YAML-based DSLs was discussed in (<a href="https://www.youtube.com/watch?v=4UO_1cKpqsc">Streamlining DAG Creation with YAML in Large Organizations</a>) as a way to provide flexibility, reproducibility, and simplicity, even in an era of AI-assisted DAG generation.</p><h3>C. AI-Assisted DAG Development</h3><p>Artificial intelligence is beginning to make its mark on Airflow development. Talks like (<a href="https://www.youtube.com/watch?v=XokcRts8XTY">From Oops to Ops: Smart Task Failure Diagnosis with OpenAI</a>) explored the use of LLMs and tools like OpenAI for intelligent diagnosis of task failures, potentially speeding up troubleshooting. Furthermore, AI-driven automation is being applied to streamline the DAG development process itself, with coding companions offering suggestions, generating boilerplate, and even helping to identify security vulnerabilities, as presented in (<a href="https://www.youtube.com/watch?v=MAACq8drYuc">Unleash the Power of AI: Streamlining Airflow DAG Development with AI-Driven Automation</a>). The emphasis here is on quantifying the impact of these AI tools, ensuring the security of coding companions, and enabling custom code recommendations that leverage existing organizational libraries rather than just generating generic code.</p><p>Collectively, these trends in DAG modernization point towards an effort to abstract the complexities of Airflow from the end-users. Whether through YAML DSLs, AI-assisted development, new DAG paradigms focusing on dataset-driven micro-pipelines, or centralized task libraries, the objective is to allow data practitioners to concentrate on their business logic rather than the intricacies of Airflow itself. This implies that platform engineering teams will increasingly build internal developer platforms (IDPs) or frameworks on top of Airflow, providing a curated and simplified experience. This aligns with the “Airflow-as-an-Engine” concept, where Airflow’s core power is harnessed beneath user-friendly abstraction layers, requiring platform engineers to be adept at both building these abstractions and managing the underlying robust Airflow infrastructure.</p><h3>Airflow’s Expanding Universe: Key Integrations &amp; the AI/ML Frontier</h3><p>Airflow’s utility is significantly amplified by its rich ecosystem and its growing applicability in orchestrating cutting-edge AI and ML workloads.</p><h3>A. Deepening Ecosystem Integrations: dbt, OpenLineage, and More</h3><p>Seamless integration with other best-of-breed tools in the modern data stack is paramount. The summit featured prominent discussions on Astronomer Cosmos, a tool designed to simplify running dbt Core projects as Airflow DAGs and Task Groups, highlighted in <a href="https://www.youtube.com/watch?v=si6UCKpurmM">Building on Cosmos: Making dbt on Airflow Easy</a>. Balyasny Asset Management (BAM), for example, uses Cosmos to translate dbt manifests into Airflow task groups, leveraging dynamic manifest caching and Kubernetes for execution, thereby simplifying dbt operations in production. Another key integration is with OpenLineage, an open standard for collecting metadata and lineage from running jobs, as discussed in <a href="https://www.youtube.com/watch?v=z_j3UFIyj9o">Activating operational metadata with Airflow, Atlan and OpenLineage</a>. Activating operational metadata with OpenLineage provides enhanced observability, data governance, and a clearer understanding of data flows within and beyond Airflow pipelines. The OpenLineage integration itself is evolving to align with Airflow 3.0’s Task Execution Interface (AIP-72) for more efficient, asynchronous event emission, a topic explored in <a href="https://www.youtube.com/watch?v=LoyIvKWXGuM">OpenLineage: From Operators to Hooks</a>.</p><h3>B. Airflow as the Backbone for GenAI, LLMOps, and MLOps</h3><p>Airflow is rapidly becoming the orchestrator of choice for the end-to-end AI/ML lifecycle. Sessions detailed visions for <a href="https://www.youtube.com/watch?v=0djdrF91n5s">Gen AI using Airflow 3: A vision for Airflow RAGs</a>, including the creation of Airflow RAGs (Retrieval Augmented Generation) leveraging common LLM interfaces and the task execution interface for data privacy. Airflow 3.0’s support for inference execution, especially when combined with event-driven scheduling (as described in talks like <a href="https://www.youtube.com/watch?v=bk9NStntfi0">Event-driven Data Pipelines with Apache Airflow — Airflow Summit 2024</a> and “<a href="https://www.youtube.com/watch?v=xgX5i7gMxqo">Airflow 3.0: Inference Execution</a>”), allows DAGs to serve as backends for AI applications. The platform’s dynamic, extensible, and scalable nature makes it well-suited for FMOps (Foundation Model Ops) and LLMOps, managing everything from data ingestion and preprocessing to model training, evaluation, and deployment, as outlined in <a href="https://www.youtube.com/watch?v=DPx-MVsXcns">Unlocking FMOps/LLMOps with Airflow: A guide to operationalizing and managing Large Language Models</a>. Companies like Ford (with Mach1ML, showcased in <a href="https://www.youtube.com/watch?v=s5Hxx5ux2rc">Unlocking the Power of AI at Ford</a> ) and ASAPP (detailed in <a href="https://www.youtube.com/watch?v=kGe4-Z24a7w">Airflow, Spark, and LLMs: Turbocharging MLOps at ASAPP</a>) are already using Airflow to orchestrate complex MLOps pipelines. The integration with tools like Ray further enhances Airflow’s capability to manage scalable AI deployments, including LLM fine-tuning and serving, as presented in <a href="https://www.youtube.com/watch?v=n4nXsmUMb-A">Streamlining AI Workflows with Apache Airflow and Ray | Ray Summit 2024</a>.</p><h3>C. Managing Infrastructure with Airflow</h3><p>The versatility of Airflow extends beyond data and ML into infrastructure orchestration. Jack Cusick’s talk <a href="https://www.youtube.com/watch?v=pWvdP1ukcXs">Bronco: Managing Terraform at Scale with Aiflow</a> demonstrated how Airflow can manage Terraform at scale, treating infrastructure provisioning as another type of orchestrated workflow. LinkedIn provided a compelling case study of using Airflow for the continuous deployment of its services in <a href="https://www.youtube.com/watch?v=GW7n3E3c3Gg">LinkedIn’s Continuous Deployment</a>, where Airflow DAGs orchestrate complex deployment pipelines and validation steps for over 7,000 services, handling approximately 1 million deployments per month. This showcases Airflow’s potential as a general-purpose orchestrator for platform teams.</p><p>These expanding use cases and integrations suggest Airflow is evolving into a “meta-orchestrator.” Its strength lies not only in its native capabilities but also in its capacity to integrate and manage a diverse array of specialized tools like dbt, Ray, MLflow, and Terraform. With Airflow 3.0’s multi-language support and event-driven architecture, its ability to connect disparate systems is further amplified. For engineers designing analytics platforms, Airflow can serve as the central nervous system, coordinating various best-of-breed tools. This can simplify the overall architecture by reducing the need for multiple, siloed scheduling systems, but it also elevates the importance of Airflow’s own reliability, scalability, and security (a topic touched upon in <a href="https://www.youtube.com/watch?v=f6gfoVJXWEE">Security United: ecosystem with Alpha-Omega, PSF &amp; ASF — Airflow Summit 2024</a>, especially as it gains access to more critical systems.</p><h3>The Road Ahead: Airflow’s Trajectory in Data Orchestration</h3><p>The 2024 Airflow Summit painted a clear picture of a platform undergoing significant transformation while solidifying its foundational strengths. Key themes revolve around the transformative potential of Airflow 3.0, a relentless pursuit of scalability and operational excellence, and a decisive expansion into AI/ML and broader infrastructure orchestration.</p><p>Airflow continues to be a dynamic and critical tool for software engineers in the data and analytics landscape. The future appears to involve increasing abstraction and specialization. As the core Airflow engine becomes more powerful and flexible with releases like 3.0, many users will interact with it through higher-level tools, platforms, and providers — such as Cosmos for dbt integration, YAML-based DAG definitions, AI-assisted development tools, and bespoke internal company platforms. This suggests a two-tiered evolution: a robust, feature-rich core for platform builders and more straightforward, abstracted interfaces for data practitioners and application developers. The vibrancy of the community and the provider ecosystem will be crucial in enabling these abstractions and ensuring Airflow’s continued success. The calls for contribution, whether through code, documentation, or feedback (echoed in talks like <a href="https://www.youtube.com/watch?v=R_80D254oN0">10 years of Airflow: history, insights, and looking forward</a> and <a href="https://www.youtube.com/watch?v=2g1nBbHgZbY">The road ahead: What’s coming in Airflow 3 and beyond? — Airflow Summit 2024</a>), remains a testament to Airflow’s community-driven ethos.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9852482358ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>