<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mehmet Akturk on Medium]]></title>
        <description><![CDATA[Stories by Mehmet Akturk on Medium]]></description>
        <link>https://medium.com/@mathchi?source=rss-2b2acb66658e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DucpVvQVhBRIAqkGdKs-qA.jpeg</url>
            <title>Stories by Mehmet Akturk on Medium</title>
            <link>https://medium.com/@mathchi?source=rss-2b2acb66658e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:24:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mathchi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[3. Masters of Speed and Flow in Azure Databricks: Cluster, Unity, and Stream]]></title>
            <link>https://mathchi.medium.com/3-masters-of-speed-and-flow-in-azure-databricks-cluster-unity-and-stream-b46c203b91d4?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b46c203b91d4</guid>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Wed, 26 Aug 2026 10:01:04 GMT</pubDate>
            <atom:updated>2026-08-26T10:01:04.138Z</atom:updated>
            <content:encoded><![CDATA[<p>In the data universe, speed and flexibility are everything.</p><p><strong>The Cluster Commander⚡</strong> manages high-performance infrastructure, <strong>The Unity Sage🔮</strong> is the wise guide for data integration, and when joined by <strong>The Stream Whisperer🌊</strong>, real-time data streaming becomes an art form. <br>These three heroes are essential for performance and streaming in the Azure Databricks ecosystem.</p><p>Let’s discover how to leverage their strengths for maximum impact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Obj16Xsa_cQke1bAO-PHmA.png" /></figure><h3>⚡ The Cluster Commander</h3><p><em>“Balances compute power, keeps the system’s pulse.”</em></p><p><strong>Who?</strong><br> The Cluster Commander is the general managing compute resources in Databricks. Scales according to workloads, dynamically starts and stops resources. Finds the perfect balance between performance and cost. ⚙️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBtqimSWJLj8gWWwt9Wi8w.png" /></figure><p><strong>What does it do?</strong></p><ul><li>Manages interactive, job, and autoscaling clusters</li><li>Sets node size, spot instance, and runtime versions</li><li>Optimizes high-cost jobs</li></ul><p><strong>When to use?</strong></p><ul><li>When compute is needed for notebooks or jobs</li><li>When allocating resources across different environments (dev/test/prod)</li><li>For parallel processing of large datasets</li></ul><blockquote>💡 <em>Tip: Keep autoscaling on but set “minWorkers” low. An unnecessarily large cluster starts slowly and costs more 💸</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*O3tikg009M8HHpT5Sx9g6g.png" /></figure><p>💬 <em>But managing power isn’t enough… You need to know who can access what and how. Next hero: </em><strong><em>The Unity Sage</em></strong> 🔎</p><h3>🔎 The Unity Sage</h3><p><em>“Organizes data, shares knowledge.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nlc7GYoYlyoh0EvSdHVnJQ.png" /></figure><p><strong>Who?</strong><br> The Unity Sage is the security sage of Databricks. Represents the spirit of Unity Catalog: organizes data access, classification, and governance. Brings order instead of chaos to the data universe. 🧙</p><p><strong>What does it do?</strong></p><ul><li>Manages data access across all Databricks workspaces</li><li>Controls permissions on tables, views, and schemas</li><li>Provides auditing, lineage, and metadata management</li></ul><p><strong>When to use?</strong></p><ul><li>When creating access control and data security strategies</li><li>For implementing corporate Data Governance policies</li><li>In projects where different teams share the same Databricks environment</li></ul><p><strong>Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*_V7pw5sKkxOqwGDzUC6DUA.png" /></figure><blockquote>💡 <em>Tip: If using Unity Catalog, always reference your tables as “catalog.schema.table”.</em></blockquote><p><strong>Situation Table:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*l-TZ0ICeuCQ1_h-1KdExDw.png" /></figure><p>💬 <em>But above all this order, there’s one more hero… Not only stores data, but manages it as it flows. Next hero: </em><strong><em>The Stream Whisperer</em></strong> 🌊</p><h3>🌊 The Stream Whisperer</h3><p><em>“Hears data streams, captures time.”</em></p><p><strong>Who?</strong><br> The Stream Whisperer is the master of the real-time data world. Using Structured Streaming, manages real-time data flows. Enables data to be analyzed as it streams — manipulating time itself. ⏱️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_fDDGQO62x09XzNokw9yQ.png" /></figure><p><strong>What does it do?</strong></p><ul><li>Pulls data from streaming sources (Kafka, Event Hub, IoT Hub)</li><li>Manages real-time analytics, alerting, or dashboard streams</li><li>Works in “trigger once” or “continuous” modes</li></ul><p><strong>When to use?</strong></p><ul><li>If you have IoT, sensor, log, or event-based data sources</li><li>When real-time operations (e.g., fraud detection) are needed</li><li>If batch jobs cause unacceptable delays</li></ul><p><strong>Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*eNxTj6LKjBgzdraM8voqfQ.png" /></figure><blockquote>💡 <em>Tip: Don’t forget to specify the “checkpoint location” — otherwise, your streaming job can’t resume from where it left off.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*p-gs5etNqQX1KaBW_8Teyw.png" /></figure><p>💬 <em>And so, the Databricks Heroes Universe </em><strong><em>is complete</em></strong><em>… From Spark to Stream, this chain makes data meaningful.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b46c203b91d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2. AI and Delta Power in Azure Databricks: ML Alchemist & Delta Guardian]]></title>
            <link>https://mathchi.medium.com/2-ai-and-delta-power-in-azure-databricks-ml-alchemist-delta-guardian-163c5e01f3b1?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/163c5e01f3b1</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Wed, 22 Jul 2026 09:56:01 GMT</pubDate>
            <atom:updated>2026-07-22T09:56:01.553Z</atom:updated>
            <content:encoded><![CDATA[<p>In modern data platforms, processing alone isn’t enough — you need to learn and protect as well.</p><p><strong>The ML Alchemist 🤖</strong> brings the magic of machine learning to your data, while <strong>The Delta Guardian 🛡️</strong> ensures integrity and governance. This duo plays a critical role in both innovation and compliance. Unlocking their power in Azure Databricks can elevate your data strategy to the next level.</p><p>Ready to explore where AI meets Delta Lake? Let’s begin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dpouIiWgLB4dehScjy6H2Q.png" /></figure><h3>🤖 The ML Alchemist</h3><p><em>“Turns data into knowledge, knowledge into intelligence.”</em></p><p><strong>Who?</strong><br> The ML Alchemist is the wizard in Databricks’ AI lab. Processes raw data, extracts features, trains models — and brings them to life. Model success depends on their magic, and equation accuracy on their intuition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j3dbCnCXQtBNWUTibWvnMg.png" /></figure><p><strong>What does it do?</strong></p><ul><li>Develops Machine Learning and Deep Learning models</li><li>Manages experiment, model, and metric tracking with MLflow</li><li>Deploys models to production (model registry)</li></ul><p><strong>When to use?</strong></p><ul><li>For model training or evaluation in the Data Science process</li><li>When building notebook-based ML pipelines</li><li>When model versioning, traceability, and MLOps are needed</li></ul><p><strong>Example:</strong></p><pre>import mlflow.sklearn<br>with mlflow.start_run():<br>    model = RandomForestClassifier().fit(X_train, y_train)<br>    mlflow.sklearn.log_model(model, &quot;rf_model&quot;)</pre><p>💡 <em>Tip: Don’t forget to specify the “conda_env” parameter when saving your model — environment dependencies are critical.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*6bCqIAwgu6zJXKc3sc4O1g.png" /></figure><p>💬 <em>But every spell needs an orderly space… Data also needs order over time. Next hero: </em><strong><em>The Delta Guardian</em></strong> 🛡️</p><h3>🛡️ The Delta Guardian</h3><p><em>“Keeps data consistent, brings order over time.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YY7mS63qG-N9ToDB8vtvtA.png" /></figure><p><strong>Who?</strong><br> The Delta Guardian is the sentinel of the Databricks data universe. With the ACID power of Delta Lake, it protects data and stores historical versions. Thanks to this hero, no data is ever lost — it only evolves.</p><p><strong>What does it do?</strong></p><ul><li>Stores data in Delta Lake format (ACID-compliant)</li><li>Enables Time Travel to access historical data versions</li><li>Makes Merge, Update, Delete operations safe</li></ul><p><strong>When to use?</strong></p><ul><li>When data security, accuracy, and rollback are important</li><li>For error-free data in ETL processes</li><li>In Slowly Changing Dimension (SCD) or audit trail scenarios</li></ul><p><strong>Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*44T_GgiabrA7qwVxSL10bg.png" /></figure><blockquote>💡 <em>Tip: Use “Optimize” and “Vacuum” commands regularly to balance performance and storage cost.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*1FwYHRLCjUt_l9mFUgUiOw.png" /></figure><p>💬 <em>But protecting data isn’t enough… You need to manage the right power to process it. Next hero in the third article: </em><strong><em>The Cluster Commander</em></strong> ⚡</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6qHNxZahr6VLEuYmrsdQjA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=163c5e01f3b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[1. The Three Musketeers of Azure Databricks: Spark, Notebook, and Orchestration]]></title>
            <link>https://mathchi.medium.com/1-the-three-musketeers-of-azure-databricks-spark-notebook-and-orchestration-d117b40be277?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d117b40be277</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[azure-data-engineer]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-databricks]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Wed, 24 Jun 2026 08:53:25 GMT</pubDate>
            <atom:updated>2026-06-24T08:53:25.075Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of Azure Databricks, three heroes stand out:</p><p><strong>The Spark Master 🔥</strong>, <br><strong>The Notebook Architect 🧠</strong>, <br>and <strong>The Job Orchestrator ⚙️</strong>.</p><p>Spark Master is the powerhouse that processes massive datasets at lightning speed; Notebook Architect is the creative mind that blends analytics with code; and Job Orchestrator is the conductor turning complex workflows into a flawless symphony. Together, they form the foundation of data processing and automation.</p><p>So, how can you harness their full potential in Azure? Let’s dive in…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zk9RRqNXADPUgmD28f1zGQ.png" /></figure><h3>🔥 The Spark Master</h3><p><em>“Every data transformation begins with a small spark.”</em></p><p><strong>Who?</strong><br> The Spark Master is the very engine at the core of Databricks. Harnessing the power of Apache Spark, it processes and transforms data as an energy hub. It can process hundreds of millions of rows in seconds — performance is its superpower. ⚡</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vFTTixYyr7iJR2OyZ4mcWw.png" /></figure><p><strong>What does it do?</strong></p><ul><li>Processes large datasets in parallel (batch or streaming)</li><li>Supports DataFrame, SQL, R, Scala, and Python</li><li>Forms the foundation for ETL, analytics, and machine learning tasks</li></ul><p><strong>When to use?</strong></p><ul><li>When you have petabyte-scale data on a Data Lake</li><li>For real-time or scheduled big data jobs</li><li>When high-performance transformations are needed</li></ul><p><strong>Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*1P_dWxZZ5hKbmNMuQzypPw.png" /></figure><blockquote>💡 <em>Tip: Use </em><em>cache() or </em><em>persist() to keep your data in memory for faster reuse.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*-jtm05pDhtiWQ26iM-t4Zw.png" /></figure><p>💬 <em>But processing raw data isn’t enough… Sometimes you need to tell its story and visualize it. Next hero: </em><strong><em>The Notebook Architect</em></strong> 🧠</p><h3>🧠 The Notebook Architect</h3><p><em>“Every line is an idea, every cell an action.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ne4qINP-vUaaijw-Yps5Ow.png" /></figure><p><strong>Who?</strong><br> The Notebook Architect is the creator within Databricks. It brings code and thought together on the same canvas — creating visual, documented, and shareable data experiences. In these notebooks, there are not just codes, but stories. 📓</p><p><strong>What does it do?</strong></p><ul><li>Runs Python, SQL, Scala, and R in the same notebook</li><li>Creates visualizations and interactive outputs (display(), matplotlib, plotly).</li><li>Provides collaborative analysis environments</li></ul><p><strong>When to use?</strong></p><ul><li>During data exploration, prototyping, or model development</li><li>When sharing analysis within a team</li><li>When building integrated data stories with ADF, Synapse, or Power BI</li></ul><p><strong>Example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*BpBz79YA-jctuV9uJB2JDQ.png" /></figure><p>💡 <em>Tip: Create “widgets” for parameterized notebooks to make them dynamic.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*miHN8lxdPrs8hQg5M5qxcg.png" /></figure><p>💬 <em>But the universe doesn’t spin with just one notebook… Sometimes you need a leader to manage and sequence all the work. Next hero: </em><strong><em>The Job Orchestrator</em></strong> ⚙️</p><h3>⚙️ The Job Orchestrator</h3><p><em>“Every task has its order, but this hero keeps everything in sync.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6fX1orYPcBytRz4P4uF2Q.png" /></figure><p><strong>Who?</strong><br> The Job Orchestrator is the strategic commander of Databricks. It organizes notebooks, scripts, and tasks; running them all in the right sequence at the right time. It brings the “Execute Pipeline” spirit of #ADF to Databricks.</p><p><strong>What does it do?</strong></p><ul><li>Triggers notebooks, JARs, or Python scripts</li><li>Manages scheduling and dependencies</li><li>Can retry failed tasks and send notifications</li></ul><p><strong>When to use?</strong></p><ul><li>To automate daily batch jobs</li><li>When step-by-step management of ETL flows is needed</li><li>To create traceable data processes in production</li></ul><blockquote><strong>Example:</strong><br> “Job → Tasks → Notebook Sequence”</blockquote><blockquote>💡 <em>Tip: Design with job parameters so you can manage Dev/Test(Acc)/Prod environments with the same code.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*HSnKjqQ0fZEnqq0Rs0NRcQ.png" /></figure><p>💬 <em>And sometimes, even a commander can’t work miracles… For miracles, you need a wizard who teaches data. Next hero in the second article: </em><strong><em>The ML Alchemist</em></strong> 🤖</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4yfEciwdSSvVuzqMJvejw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d117b40be277" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validation Activity — “The Inspector”]]></title>
            <link>https://mathchi.medium.com/validation-activity-the-inspector-0a195fe6b627?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a195fe6b627</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Mon, 01 Jun 2026 05:36:00 GMT</pubDate>
            <atom:updated>2026-06-01T05:36:00.447Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The guardian of truth and order.</strong></p><blockquote><em>“I do not seek fault…</em></blockquote><blockquote><em>Mistakes fear me.”— The Inspector 🔍</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAL-8yVZ1OeszMR6iK2XoQ.png" /></figure><p><strong>🧠 Who is “The Inspector”?</strong></p><p><strong>The Validation Activity</strong> is used in Azure Data Factory to ensure that the pipeline or dataset <strong>is correctly defined</strong> and <strong>accessible</strong>.</p><p>In other words, “The Inspector” is the quality control department of the data world.</p><p>It performs <strong>preliminary checks</strong> before running the pipeline, thus catching potential future errors before they even begin. 🧩</p><p><strong>⚙️ What Does It Do?</strong></p><p>The Validation Activity’s role is to check that the components within ADF:</p><blockquote>• <strong>Dataset connections</strong> (e.g., Blob, SQL, etc.) are correctly defined,</blockquote><blockquote>• <strong>Dependent activities</strong> are ready,</blockquote><blockquote>• <strong>The pipeline structure</strong> is complete.</blockquote><p><strong>📘 Example:</strong></p><blockquote>Before using “Copy Data” in the pipeline, check if the “Source Dataset” is defined.</blockquote><blockquote>If it’s missing, the pipeline is stopped immediately — saving unnecessary computation.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kVsRKG5wSr46IigaQypnZQ.png" /></figure><blockquote><strong>🧠 Tip: </strong>Using the Validation Activity as the <strong>first step</strong> in large, multi-tiered pipelines is the most effective way to catch errors early.</blockquote><p><strong>🧩 When to Use It?</strong></p><p>“The Inspector” is often a friend of <strong>proactive</strong> teams.</p><p>It comes into play in complex pipelines or when there are many dependent datasets.</p><p><strong>Use Cases:</strong></p><blockquote>• For testing after new pipelines are deployed</blockquote><blockquote>• When datasets, linked services, or mappings change</blockquote><blockquote>• To prevent unnecessary runs in large data streams</blockquote><blockquote>• Automated post-CI/CD pipeline validation</blockquote><p><strong>📘 Example:</strong></p><blockquote>“Validation Activity” → “If Condition” → “Execute Pipeline”</blockquote><blockquote>If validation fails, “The Judge” (If Condition) stops the pipeline.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AXiYnOGC1aqiCVFRk3tx_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*jk8v7u5yevvnV3o-7-CbPA.png" /></figure><p><strong>🧩 Sample Scenario</strong></p><p><strong>Scenario:</strong></p><p>You’ve added a new dataset and want to make sure everything is in order before running the pipeline.</p><p><strong>🪄 ADF Flow:</strong></p><blockquote>1. <strong>Validation Activity</strong> → Verifies that all datasets are defined and accessible.</blockquote><blockquote>2. <strong>If Condition Activity (The Judge)</strong> → If the validation result is Success, continue.</blockquote><blockquote>3. <strong>Execute Pipeline Activity (The Commander)</strong> → Calls the actual data loading pipeline.</blockquote><blockquote><strong>📘 Tip: </strong>If the Validation Activity fails, the ADF logs will clearly show you which dataset or connection is missing.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIAVYwzIGYv07SBGn2aHHQ.png" /></figure><p><strong>🧠 Why?</strong></p><p>“The Inspector” is the <strong>symbol of preventative justice</strong> in the data world.</p><p>He steps in before problems can stop the pipeline.</p><p>He detects errors, maintains order, and creates a safe data flow.</p><p>Behind every successful pipeline is an “Inspector.”</p><blockquote><em>“I’m invisible, but I’m confident everything is going well.” — The Inspector 🔍</em></blockquote><p><strong>Beyond the Azure Data Factory universe lies a new world powered by seamless integration and future-ready capabilities: Azure Databricks!</strong><br> In this article series, we’ll explore this game-changing platform for data processing and analytics, guided by our new superheroes. Get ready — <strong>Azure Databricks</strong> isn’t just a technology; it’s a revolutionary step in the data universe!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*eqfZC2GhGqKmmosrMelIbQ.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a195fe6b627" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Execute Data Flow — “The Transformer”]]></title>
            <link>https://mathchi.medium.com/execute-data-flow-the-transformer-20f029d24b21?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/20f029d24b21</guid>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Mon, 18 May 2026 05:31:00 GMT</pubDate>
            <atom:updated>2026-05-18T05:31:00.727Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>A master of creating meaning from raw data.</strong></p><blockquote><em>“I don’t just carry data… I rebirth it.”</em></blockquote><blockquote><em>— The Transformer ⚙️</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9HoYWZ08WQUs-niu9vHA3Q.png" /></figure><p><strong>🧠 Who is “The Transformer”?</strong></p><p><strong>Execute Data Flow Activity</strong> is Azure Data Factory’s <strong>data transformation engine</strong>.</p><p>It’s the “visual ETL” component of ADF; <strong>it ingests data, processes it, transforms it, cleans it, enriches it, and sends</strong> it to the target.</p><p>So, the smooth data you see in Power BI reports — has actually been through <em>The Transformer</em>’s hands. 💪</p><p><strong>⚙️ What Does It Do?</strong></p><blockquote>• Performs data cleansing, joining, filtering, sorting, and calculations.</blockquote><blockquote>• Provides high performance on large data volumes thanks to Spark-based parallel processing.</blockquote><blockquote>• Allows you to visually design data flows — you can perform complex transformations without writing SQL.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9v9WEqd5ykb8pXsaRSAqtw.png" /></figure><p><strong>📘 Example:</strong></p><blockquote>Read the “sales_raw.csv” file from Blob Storage →</blockquote><blockquote>Purge null rows →</blockquote><blockquote>Group by “region” →</blockquote><blockquote>Load into Azure SQL.</blockquote><blockquote><strong>🧠 Tip:</strong></blockquote><blockquote>Within Data Flow, you can create dynamic columns with Derived Column and branch data with Conditional Split.</blockquote><p><strong>🌐 When to Use:</strong></p><p>“The Transformer” comes into play:</p><blockquote>• When data should not only be moved, but also processed and transformed.</blockquote><blockquote>• When a visual transformation design is preferred over SQL or Python code.</blockquote><blockquote>• When data cleansing, mapping, formatting, or ETL steps are required.</blockquote><blockquote>• When data from different sources needs to be combined into a single format.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3uvmzrVfYvjssZaeBqozPg.png" /></figure><p><strong>📘 Example:</strong></p><blockquote>Extract raw data with “Copy Activity,”</blockquote><blockquote>then standardize it with “Execute Data Flow.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*fGEhbaM_7uGgzEibbjUaxw.png" /></figure><p><strong>🧩 Sample Scenario</strong></p><p><strong>Scenario: </strong>Consolidate daily sales data and convert it into a reporting format.</p><p><strong>🪄 ADF Flow:</strong></p><p>1. <strong>Copy Activity</strong>: Move raw data from Blob to Data Lake.</p><p>2. <strong>Execute Data Flow</strong>:<br>- Remove null values<br>- Group by Region<br>- Calculate “total_sales”<br>- Sort by Date</p><p>3. <strong>Sink</strong>: Write clean data to Azure SQL.</p><p>4. <strong>Power BI</strong>: Now the report is ready 🎯</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*781u-fpklmtC0fdf3Bkn3A.png" /></figure><blockquote><strong>📘 Tip: </strong>Keep each step in the Data Flow small.</blockquote><blockquote>Test only with sample data during debugging — don’t waste computing hours.</blockquote><p><strong>🧠 Why?</strong></p><p>The power of ADF comes from<strong> processing</strong> data, not just moving it.</p><p>“The Transformer” is the heart of this transformation.</p><p>It transforms raw data into information and chaos into order.</p><p>A kind of digital blacksmith — he shapes data, purifies it, and prepares it for analysis.</p><blockquote><em>“Data becomes meaningful in my hands.”</em></blockquote><blockquote><em>— The Transformer ⚙️</em></blockquote><p>🔍 Every hero fights, but one watches.</p><p><strong>The Inspector</strong>, the eye that makes sure everything goes right, is in the next episode! 👁️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PS2NlvS8v81tiHLh9A_59w.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20f029d24b21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Switch Activity — “The Strategist”]]></title>
            <link>https://mathchi.medium.com/switch-activity-the-strategist-c15d0dbbc8eb?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c15d0dbbc8eb</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Sun, 03 May 2026 05:26:01 GMT</pubDate>
            <atom:updated>2026-05-03T05:26:01.239Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Master of Decisive Moments</strong></p><p>Every pipeline is a story.</p><p>But every story has multiple paths.</p><p>This is where the master tactician of the ADF universe enters the picture:</p><p><strong>“The Strategist.”</strong></p><blockquote><em>“There is no one right path…</em></blockquote><blockquote><em>But I make the right decision at the right time.”— The Strategist 🔄</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nBhZCHra4LhOnyEHb70WXw.png" /></figure><p><strong>🧠 Who is “The Strategist”?</strong></p><p><strong>Switch Activity</strong> is Azure Data Factory’s <strong>multi-decision mechanism</strong>.</p><p>It works just like the switch-case structure in programming:</p><p>It evaluates an expression and executes different activities based on its result.</p><p>In short, <strong>it’s the tactical control center of the pipeline.</strong></p><p>It simplifies decision trees and manages conditional branches.</p><p><strong>⚙️ What Does It Do?</strong></p><p>Switch Activity generates an <strong>expression</strong> and, based on this value, executes one of the defined case blocks.</p><p>If none match, the <strong>default case</strong> is invoked.</p><p><strong>📘 Example:</strong></p><p>Call different data sources based on the @pipeline().parameters.region value:</p><pre>expression: @pipeline().parameters.region  <br>cases:<br>  - case: &quot;EU&quot;<br>    activities: [Copy_EU_Data]<br>  - case: &quot;US&quot;<br>    activities: [Copy_US_Data]<br>default:<br>  - activities: [Notify_Admin]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y51K5FWh8101wXS3M5dlXQ.png" /></figure><p><strong>🧩 When to Use:</strong></p><p>“The Strategist” comes into play, especially in situations where <strong>multiple possible scenarios</strong> exist.</p><p>It is the <strong>decision cente</strong>r in pipelines where a single condition is not sufficient and different flows are required.</p><p><strong>Usage Scenarios:</strong></p><blockquote>• Separate streams for different <strong>regions</strong> or <strong>environments (dev/test/prod)</strong></blockquote><blockquote>• Different processing steps for different <strong>file types</strong></blockquote><blockquote>• Lookup or parameter-based branching</blockquote><blockquote>• Different recovery scenarios based on error conditions</blockquote><p><strong>📘 Example:</strong></p><p>If the Lookup Activity results in dataType = &quot;csv&quot; → run the CSV stream;</p><p>if the dataType = &quot;json&quot; → run the JSON stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*inhPPbz4aRT-tGhpwTiVmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*B0LPRN4-DCBNIcS_hgO6MA.png" /></figure><p><strong>🧩 Sample Scenario</strong></p><p><strong>Scenario: </strong>A pipeline must pull data from different sources based on its region.</p><p><strong>🪄 ADF Flow:</strong></p><blockquote><strong>1. Pipeline Parameter</strong>: region (e.g., “EU”, “US”, “ASIA”)</blockquote><blockquote>2. <strong>Switch Activity</strong>:</blockquote><pre>expression: @pipeline().parameters.region<br>cases:<br>  - case: &quot;EU&quot;<br>    activities: [Copy_EU_Data]<br>  - case: &quot;US&quot;<br>    activities: [Copy_US_Data]<br>default:<br>    activities: [Send_Alert]</pre><blockquote>3. <strong>Copy Activity</strong> → Data appropriate to the region is transferred.</blockquote><blockquote>4. <strong>Web Activity</strong> → A success message is sent.</blockquote><blockquote><strong>📘 Tip: </strong>If there are many cases, using a combination of a Lookup table and an If Condition instead of a Switch is easier for maintenance.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_o7KDvwrgQrtRrSbBjgqg.png" /></figure><p><strong>🧠 Why?</strong></p><p>Data operations do not follow a single path.</p><p>Sometimes circumstances change, plans diverge, and strategy is needed.</p><p>It’s at those moments that “The Strategist” takes control, determining the path to the right scenario.</p><blockquote><em>“I’m not in the flow…</em></blockquote><blockquote><em>I’m the one driving it.”</em></blockquote><blockquote><em>— The Strategist 🔄</em></blockquote><p>⚙️ Raw data becomes meaningful in his hands.</p><p>The master of transformation is coming — <strong>The Transformer</strong> takes the stage!⚙️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4yfEciwdSSvVuzqMJvejw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c15d0dbbc8eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Filter Activity — “The Guardian”]]></title>
            <link>https://mathchi.medium.com/filter-activity-the-guardian-aa5d4af6f7f2?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa5d4af6f7f2</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Sun, 19 Apr 2026 05:21:01 GMT</pubDate>
            <atom:updated>2026-04-19T05:21:01.206Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Selective Shield of the Data Stream</strong></p><p>Not everything in the data universe should be allowed in.</p><p>Some are dirty, some are incomplete, some are unnecessary.</p><p>That’s why there’s a hero standing at the edge of the ADF:</p><p><strong>“The Guardian.”</strong></p><blockquote><em>“I don’t let everyone through.</em></blockquote><blockquote><em>I only let in the right ones.” — The Guardian 🧩</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5dttY58ayzHVrzWEatjPw.png" /></figure><p><strong>🧠 Who is “The Guardian”?</strong></p><p><strong>The Filter Activity</strong> is Azure Data Factory’s <strong>data filtering expert</strong>.</p><p>It filters elements within a collection (usually an array) <strong>based on a specific condition, ensuring </strong>only those that match are passed.</p><p>In short:</p><p>It’s <strong>a “data security checkpoint”</strong> in an ADF stream.</p><p>Data arrives, it checks, passes the appropriate elements, and stops the rest.</p><p><strong>⚙️ What Does It Do?</strong></p><p>The Filter Activity <strong>takes an array as input</strong>, evaluates each element against a specific condition, and <strong>returns a new array of elements that meet the condition.</strong></p><p><strong>Structure:</strong></p><blockquote>• <strong>Items</strong>: Array to filter</blockquote><blockquote>• <strong>Condition</strong>: Boolean expression (e.g.,@equals(item().type, &#39;csv&#39;))</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JorbDjLGq0--25Xduhm2_A.png" /></figure><p><strong>📘 Example:</strong></p><p>“Select only .csv files in the blob folder.”</p><pre>items: @activity(&#39;Get Metadata&#39;).output.childItems  <br>condition: @endsWith(item().name, &#39;.csv&#39;)</pre><p><strong>⚖️ When to Use:</strong></p><p>“The Guardian” comes into play when you need to narrow down the data list in a pipeline or process only data that meets certain conditions.</p><p><strong>Usage Scenarios:</strong></p><blockquote>• Selecting specific file types (.csv, .json)</blockquote><blockquote>• Filtering based on date or size</blockquote><blockquote>• Cleaning the dataset before Lookup or Data Flow</blockquote><blockquote>• Excluding incorrect or missing data</blockquote><blockquote><strong>📘 Example:</strong></blockquote><blockquote><strong>Get Metadata</strong> → retrieve all files</blockquote><blockquote><strong>Filter Activity</strong> → select only those starting with “2025_”</blockquote><blockquote><strong>ForEach </strong>→ process selected files</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pxB96EzkwyAbuikn9EV_lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*npvBZ--n8D6Xp8pyRcY_Hg.png" /></figure><p><strong>🧩 Sample Scenario</strong></p><p><strong>Scenario: </strong>A Blob folder contains many different files.</p><p>Only .csv files starting with “report_” should be processed.</p><p><strong>🪄 ADF Flow:</strong></p><blockquote>1. <strong>Get Metadata</strong> → List files</blockquote><blockquote>2. <strong>Filter Activity</strong> →</blockquote><pre>items: @activity(&#39;Get Metadata&#39;).output.childItems  <br>condition: @and(endsWith(item().name, &#39;.csv&#39;), startsWith(item().name, &#39;report_&#39;))</pre><blockquote>3. <strong>ForEach Activity</strong> → Process the filtered list</blockquote><blockquote>4. <strong>Copy Activity</strong> → Load into SQL</blockquote><blockquote><strong>📘 Tip: </strong>You can access the filter results with @activity(&#39;Filter&#39;).output.Value.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*80p6dW7Gooj4HFadWrnnkg.png" /></figure><p><strong>🧠 Why?</strong></p><p><strong>Not everything</strong> in the data world <strong>should be processed.</strong></p><p>ADF pipelines are safe, consistent, and efficient when they work with accurate data.</p><p>“The Guardian” maintains this integrity —</p><p>it doesn’t report everything, <strong>only what’s right</strong>.</p><blockquote><em>“My mission is simple.</em></blockquote><blockquote><em>I don’t trust everyone.”</em></blockquote><blockquote><em>— The Guardian 🧩</em></blockquote><p>🔄 Everything may not go according to plan…</p><p>But he finds the right strategy in every scenario.</p><p>Our next hero: <strong>The Strategist</strong> 🎯</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*4u86vPfFpBoAt1Zfb5y9mA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa5d4af6f7f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Append Variable Activity — “The Scribe]]></title>
            <link>https://mathchi.medium.com/append-variable-activity-the-scribe-c3bfe16d5eaa?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3bfe16d5eaa</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 06:36:01 GMT</pubDate>
            <atom:updated>2026-04-06T06:36:01.292Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Archivist of the Data Stream</strong></p><p>Some heroes move data, some transform it…</p><p>But “The Scribe” <strong>writes </strong>it all down.</p><p>He silently records every event, every file name, every result that occurs in the ADF universe in a notebook.</p><blockquote><em>“I don’t change variables…</em></blockquote><blockquote><em>I magnify them.”— The Scribe 📜</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A6jpA3QU5Du-5RzyOGafsQ.png" /></figure><p><strong>🧩 Who is “The Scribe”?</strong></p><p><strong>The Append Variable Activity</strong> is the <strong>accumulator</strong> hero of Azure Data Factory.</p><p>It <strong>adds a new element</strong> to an existing <strong>array variable</strong>.</p><p>It enables dynamic data collection in ADF — much like a printer keeps a log.</p><p>This activity is used for <strong>accumulating data</strong>, <strong>creating lists, and collecting logs or outputs</strong>.</p><p><strong>⚙️ What Does It Do?</strong></p><p>The Append Variable Activity adds <strong>new value(s</strong>) to <strong>an array variable</strong> previously defined at the pipeline level.</p><p>With each execution, the number of elements in the list increases.</p><p><strong>📘 Example:</strong></p><p>Accumulating file names:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*kT7qOHAbvoJ9BNgUORFwRA.png" /></figure><p>Then send this list to another system using a Web Activity or Stored Procedure.</p><p><strong>🧠 Why Does It Exist?</strong></p><p>In ADF flows, it’s sometimes necessary to store not just the last value, but the <strong>entire history, the entire list</strong>.</p><p>This is where “The Scribe” comes in — it keeps <strong>a log </strong>in the pipeline’s memory.</p><blockquote><em>“I don’t write a single file, I write the entire story.”</em></blockquote><blockquote><em>— The Scribe</em></blockquote><p>This way, you can:</p><blockquote>• Collect which files were processed in each operation,</blockquote><blockquote>• Archive the results returned from the API,</blockquote><blockquote>• Store the pipeline history in debug and logging scenarios.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpRvUxRIQYURgWZ9-7AWpA.png" /></figure><p><strong>🧱 How It Works:</strong></p><p><strong>1. Define Variable:<br></strong>o <strong>Name</strong>: processedFiles<br>o <strong>Type</strong>: Array<br>o <strong>Default Value</strong>: []</p><p><strong>2. Append Variable Activity:<br></strong>o <strong>Variable name</strong>: processedFiles<br>o <strong>Value</strong>: @item().name</p><p><strong>3. Use:<br></strong>o @variables(&#39;processedFiles&#39;) → access in subsequent activities.</p><p><strong>📘 Example:</strong></p><pre>{<br>  &quot;name&quot;: &quot;AppendFileName&quot;,<br>  &quot;type&quot;: &quot;AppendVariable&quot;,<br>  &quot;typeProperties&quot;: {<br>    &quot;variableName&quot;: &quot;processedFiles&quot;,<br>    &quot;value&quot;: &quot;@item().name&quot;<br>  }<br>}</pre><p><strong>🔁 When to Use:</strong></p><p>“The Scribe” is called to write a result to memory at each iteration, especially in <strong>repetitive operations</strong> (like ForEach).</p><p><strong>Usage Scenarios:</strong></p><blockquote>• Collecting processed file names</blockquote><blockquote>• Listing API responses</blockquote><blockquote>• Logging error records</blockquote><blockquote>• Generating a summary report at the end of the pipeline</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lw5kF8iQEvzW0ku1TG-IVA.png" /></figure><p><strong>📘 Example:</strong></p><p>ForEach Activity → Add each file processed to the processedFiles list with Append Variable</p><p>At the end of the pipeline, send a message to Slack saying “Total {length(processedFiles)} files processed.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*cFeRT7u4i2uTucRl_j-_mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*_Xl04SKRK9VA6QdOf2rsuA.png" /></figure><p><strong>🧩 Sample Scenario</strong></p><p><strong>Scenario: </strong>All CSV files in a folder will be processed, each processed file name will be added to the list, and reported via email at the end of the pipeline.</p><p><strong>🪄 ADF Flow:</strong></p><p>1. <strong>Get Metadata</strong> → List files</p><p>2.<strong> ForEach</strong> → For each file:<br>o Copy Activity → Move file to SQL<br>o Append Variable → Add file name to processedFiles list</p><p>3. <strong>Web Activity</strong> → Send report via Slack or Logic App:<br>“✅ Processed files: @join(variables(‘processedFiles’), ‘, ‘)”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9oLI3UsI2UVOXiCQc0h8w.png" /></figure><blockquote><strong>📘 Tip: </strong>Append Variable only works with arrays.<br>Use “Set Variable” to update single-valued variables.</blockquote><p>🧩 Not all data is created equal.</p><p>About to meet the watchdog who eliminates the false and protects the true: <strong>The Guardian </strong>🛡️</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6qHNxZahr6VLEuYmrsdQjA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3bfe16d5eaa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delete Activity — “The Cleaner”]]></title>
            <link>https://mathchi.medium.com/delete-activity-the-cleaner-ac63ed962166?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac63ed962166</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Tue, 17 Mar 2026 08:08:17 GMT</pubDate>
            <atom:updated>2026-03-17T08:08:17.154Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Silent Guardian of the Data Universe</strong></p><p>Chaos in the data world begins with accumulated files, old logs, and forgotten folders.</p><p>But ADF’s unsung hero refuses.</p><p>He arrives silently, does his job, and leaves behind a pristine order.</p><blockquote><em>“I don’t move data, I don’t transform it…</em></blockquote><blockquote><em>But in the end, I maintain order.”— The Cleaner 💣</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7QwaBeECHX2DG1JAY-oTRg.png" /></figure><p><strong>🧩 Who is “The Cleaner”?</strong></p><p><strong>The Delete Activity</strong> is used in Azure Data Factory to systematically delete unnecessary or old data.</p><p>It acts as a <strong>data cleansing agent</strong> — it removes files or rows from the target dataset.</p><p>In ADF pipelines, it typically appears at the end of the job, but its impact is felt throughout.</p><p><strong>⚙️ What Does It Do?</strong></p><p>The Delete Activity’s function is simple yet critical:</p><p><strong>Deleting target files or directories.</strong></p><p>This is typically used to:</p><blockquote>• Remove old, committed files,</blockquote><blockquote>• Clean up temporary staging areas,</blockquote><blockquote>• Reset log or cache folders,</blockquote><blockquote>• Perform cleanups based on rules like “older than 7 days” or “processed files.”</blockquote><p><strong>📘 Example:</strong></p><p>Delete all files older than 7 days from a Blob Storage container.</p><pre>&quot;typeProperties&quot;: {<br>  &quot;dataset&quot;: { &quot;referenceName&quot;: &quot;BlobDataset&quot; },<br>  &quot;recursive&quot;: true,<br>  &quot;enableLogging&quot;: true,<br>  &quot;maxConcurrentConnections&quot;: 5<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Y2XFr6oET7CLSl5YXaY7w.png" /></figure><p><strong>🧠 Why Does It Exist?</strong></p><p>ADF doesn’t just move data; it manages its lifecycle as <strong>a complete story</strong>.</p><p>And every story requires a cleanup at the end.</p><blockquote><em>“There’s no rebirth without destruction.”</em></blockquote><blockquote><em>— The Cleaner 💣</em></blockquote><p><strong>Delete Activity</strong> completes this cycle.</p><p>It balances the data lifecycle, reduces costs, and maintains order.</p><p><strong>⏱️ When to Use It?</strong></p><p>“The Cleaner” is typically called:</p><ul><li>To remove temporary files at the <strong>end of a job</strong></li><li>To reset the target folder <strong>before a new load</strong></li><li>In <strong>log rotation</strong> or <strong>data retention</strong> scenarios</li><li>To clean up <strong>old backups</strong> or <strong>snapshots</strong>.</li></ul><p><strong>🧩 Sample Scenario:</strong></p><blockquote>1. Copy Activity → Move data from Blob</blockquote><blockquote>2. Stored Procedure Activity → Load into SQL</blockquote><blockquote>3. Delete Activity → Delete the file in the Blob</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5wRjrVqP48-qQkPDPzaDPA.png" /></figure><blockquote><strong>📘 Tip:</strong> You can clean it along with its subfolders by setting “recursive = true.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*PfQyvLAGjTFW1t8hV0TPdQ.png" /></figure><p><strong>🧩 Bonus: Smart Cleanup Scenario</strong></p><p><strong>Scenario: </strong>There is a pipeline that runs every night.</p><p>After new data is loaded, files older than 7 days should be deleted.</p><p><strong>🪄 ADF Flow:</strong></p><blockquote>1. <strong>Copy Activity</strong> → Import new CSVs to SQL.</blockquote><blockquote>2. <strong>Stored Procedure Activity</strong> → Mark the data.</blockquote><blockquote>3. <strong>Delete Activity </strong>→ Delete *.csv files in the blob with the condition of LastModified &lt; utcnow() - 7 days.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JWZp_idK_5fBMrUpBh3uYQ.png" /></figure><blockquote><strong>📘 Tip: </strong>To get the log before deletion, add the “Get Metadata + ForEach + Web Activity” combination.</blockquote><p>📜 Not every hero takes notes, but he does. Meet the archivist who collects and carries knowledge into the future: <strong>The Scribe</strong> 📝</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4yfEciwdSSvVuzqMJvejw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac63ed962166" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set Variable Activity — “The Memory Keeper”]]></title>
            <link>https://mathchi.medium.com/set-variable-activity-the-memory-keeper-3e21ad4cbea7?source=rss-2b2acb66658e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e21ad4cbea7</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[azure-data-factory]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Mehmet Akturk]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 08:08:56 GMT</pubDate>
            <atom:updated>2026-03-09T08:08:56.713Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Memory Keeper of Data Flow</strong></p><p>Every hero has a memory.</p><p>But in the ADF universe, there’s a wise man who protects that memory, never forgets it, and reminds it when necessary:</p><p><strong>“The Memory Keeper.”</strong></p><p>Pipelines flow, activities run, data moves — but some information must be temporarily stored.</p><p>That’s when he steps into the picture, silently holding a value in his mind, and revealing it when the time is right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RJ0i5uHIgu5CRcH8b2Eocg.png" /></figure><p><strong>🧠 Who is “The Memory Keeper”?</strong></p><p><strong>The Set Variable Activity</strong> is the hero of Azure Data Factory, <strong>managing pipeline memory</strong>.</p><p>It sets the value of a variable, stores it, and ensures its use in subsequent activities.</p><blockquote><em>“I don’t move data… I remember it.”</em></blockquote><blockquote><em>– The Memory Keeper 🪶</em></blockquote><p>This hero <strong>makes data flow dynamic</strong> within a pipeline.</p><p>Conditions, loops, parameters — they all work thanks to the information it holds.</p><p><strong>⚙️ What Does It Do?</strong></p><p><strong>The Set Variable Activity</strong> is used to assign values ​​to variables defined within the pipeline.</p><p>These values ​​are used in conditional flows, loops, or operations like logging.</p><p><strong>Usage Scenarios:</strong></p><blockquote>• Temporarily storing each file name in ForEach</blockquote><blockquote>• Holding a flag value in conditional flows (e.g., success = true)</blockquote><blockquote>• Storing API or Lookup output in a variable</blockquote><blockquote>• Displaying log or error messages at the end of the pipeline</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V86W58MaX2lV51xTlhwd-Q.png" /></figure><p><strong>📘 Example:</strong></p><blockquote><em>Set Variable: filename = @item().name <br></em><strong>🧠 Tip:</strong> Define variables first, then assign them — In ADF, variable definition is done at the pipeline level.</blockquote><p><strong>🧩 How It Works:</strong></p><p><strong>1. Define Variable:</strong></p><p>Create a variable at the pipeline level:<br>o <strong>Name</strong>: fileName<br>o <strong>Type</strong>: String, Bool and Array<br>o <strong>Default Value</strong>: (optional)</p><p><strong>2. Add a Set Variable Activity:<br></strong>o <strong>Name</strong>: SetFileName <br>o <strong>Variable Name</strong>: fileName<br>o <strong>Value</strong>: @item().name</p><p><strong>3. Use:<br></strong>o Access it in other activities with @variables(&#39;fileName&#39;).</p><p><strong>📘 Example:</strong></p><pre>{<br>  &quot;name&quot;: &quot;SetFileName&quot;,<br>  &quot;type&quot;: &quot;SetVariable&quot;,<br>  &quot;typeProperties&quot;: {<br>    &quot;variableName&quot;: &quot;fileName&quot;,<br>    &quot;value&quot;: &quot;@item().name&quot;<br>  }<br>}</pre><p><strong>🔁 When to Use:</strong></p><p>“The Memory Keeper” is called to <strong>store temporary but important information within a pipeline.</strong></p><p>It is especially ideal for data control in dynamic flows (ForEach, If Condition, Switch).</p><p><strong>Use Cases:</strong></p><blockquote>• Keeping the name or ID of the current element in ForEach</blockquote><blockquote>• Using the value from the API response in the next activity</blockquote><blockquote>• Combining variables to create a log message</blockquote><blockquote>• Carrying state information in a conditional flow</blockquote><blockquote><em>🧠 A small variable can drive a large amount of logic.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9x9HlzQPOWch8lY2O9XfRQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*8JZx-VTwx0Dtjc4ctag2oQ.png" /></figure><p><strong>📘 Sample Scenario</strong></p><p><strong>Scenario: </strong>You want to process the files in the blob one by one and log each file name.</p><p><strong>🧩 Pipeline Flow:</strong></p><p>1. <strong>Get Metadata Activity</strong>: Get the file list.</p><p>2. <strong>ForEach Activity</strong>: @activity(&#39;GetFiles&#39;).output.childItems</p><p>3. Inside:<br>o Set Variable → fileName = @item().name<br>o Copy Activity → move the file.<br>o Web Activity → send a “File &lt;fileName&gt; processed” message to Slack.</p><blockquote><strong>📘 Tip: </strong>When assigning variables in ForEach, prefer sequential execution.</blockquote><blockquote>Variable values ​​can be mixed up in parallel mode.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jvXP3ZOc_iCY2B78RR2t7Q.png" /></figure><p><strong>🧠 Why?</strong></p><p>In ADF, a data flow is a logical story. But that story must be remembered.</p><p>“The Memory Keeper” is the memory of this story.</p><p>It holds the pipeline’s past and directs its future.</p><blockquote><em>“I don’t change the flow…</em></blockquote><blockquote><em>But I remember the reason for the flow.”</em></blockquote><blockquote><em>– The Memory Keeper 🪶</em></blockquote><p>💣 Every system needs cleaning.</p><p><strong>The Cleaner</strong>, the guardian of order that eliminates unnecessary data, is coming! 🧹</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*eqfZC2GhGqKmmosrMelIbQ.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e21ad4cbea7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>