Sagar_Soni·4d agoApache Airflow DAGs: Why Task Dependencies Matter More Than Task OrderLearn how Apache Airflow schedules tasks, manages dependencies, handles parallel execution, and helps you build reliable ETL pipelines.
InOrchestra’s Data Release Pipeline BlogbyG the Sheep·Nov 6Terraforming Orchestra pipelines, YAML and allKeep the YAML you already have, and let Terraform do the nagging…A response icon2A response icon2
Nicolas Pogeant·Nov 5dbt Knows the Order, Not the ScheduleOrdering models is dbt’s job. Deciding when anything runs, and what must finish first, belongs to the layer around it
Alexander Daniel Pratama·Nov 5Orchestrate your Arcpy in AirflowFor years, all my ETL scripts did its job well. Then one day a colleague asked me to explain how it worked, and I froze because it will…
In47BillionbyPankaj Jaiswal·Sep 25Choosing a Data Pipeline Orchestrator: Cron Jobs to DagsterIt started with a single cron job, then grew to a dozen, across different schedules, servers, and parameters, with no reliable way to…
Sudeep Hazra·Sep 17Your Orchestrator Is an Operating Model, Not a Feature ListData orchestration became interesting again, although I am not sure it ever stopped being complicated.
InDevOps.devbySendoa Moronta·Aug 27Why Your Airflow Metadata Database Keeps GrowingAirflow metadata is operational state, not application data. At scale, treating it like an unlimited database eventually turns scheduling…
Nafiu Shaibu·Aug 27Beyond JMESPath: How We Reduced Partner Onboarding Time by 40% with a Custom JSON EngineA few years ago, I stumbled across an article describing how enterprise software systems in the early 2000s used XML to encode complex…
InOrchestra’s Data Release Pipeline BlogbyG the Sheep·Aug 26SFTP to S3, and back again, without the dramaTwo get-file tasks, two put-file tasks, and no SSH config in sight
Andrew Tan (layline.io)·Aug 25The News Everyone Saw ComingDagster joining Prefect signals the end of standalone orchestrators. The winners will be unified platforms that combine orchestration and…
