<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nshuti Elisé, MSc on Medium]]></title>
        <description><![CDATA[Stories by Nshuti Elisé, MSc on Medium]]></description>
        <link>https://medium.com/@enshuti?source=rss-4a3fb7db36c2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_gComKj5jeyMPL6f0NL16g.jpeg</url>
            <title>Stories by Nshuti Elisé, MSc on Medium</title>
            <link>https://medium.com/@enshuti?source=rss-4a3fb7db36c2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:23:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@enshuti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cost Analysis and Architectural Decisions in an End-to-End Streaming Data Engineering Project…]]></title>
            <link>https://medium.com/@enshuti/cost-analysis-and-architectural-decisions-in-an-end-to-end-streaming-data-engineering-project-8434576b832c?source=rss-4a3fb7db36c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8434576b832c</guid>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Nshuti Elisé, MSc]]></dc:creator>
            <pubDate>Fri, 04 Jul 2025 19:18:16 GMT</pubDate>
            <atom:updated>2025-07-04T19:19:02.473Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSOQn084JMD2dOfHrvzZvQ.png" /></figure><h3><strong>Cost Analysis and Architectural Decisions in an End-to-End Streaming Data Engineering Project (Azure Databricks vs Azure Functions)</strong></h3><p>For the past two weeks, I’ve been deep-diving into real-time data streaming using Azure Databricks and Azure Functions, guided by<a href="https://www.linkedin.com/in/mrk-talkstech?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAElYxR8BX2hRzc5M76HVIXSxcJWtPDjXbLQ"> Mr. K Talks Tech</a><a href="https://www.youtube.com/watch?v=TIjgNlkvqxI&amp;t=15995s"> ‘s 5-hour YouTube video</a> on Weather Data Real-Time Streaming. What was meant to be a “weekend project” evolved into a deep, two-week exploration and honestly, every minute was worth it.</p><p>Here’s why:</p><p>As data engineers, it’s easy to get caught up in the excitement of solving technical challenges:</p><ul><li>Connecting APIs and message queues,</li><li>Triggering data processing jobs,</li><li>Building ETL pipelines,</li><li>Designing dashboards.</li></ul><p>Whether you’re working in Azure, AWS, GCP, on-prem Spark clusters, Airflow, or even lightweight serverless setups, these tasks are where many of us find excitement and flow.</p><p>But while we dive into these builds, there’s an often-overlooked skill that matters just as much as technical execution: making architecture decisions that balance cost and performance for the business.</p><h3>Why Cost Estimation Matters for Data Engineers</h3><p>Since the deep dive was done using Azure in the video, I’ll use it as the context here. Azure makes it easy to spin up resources; a few clicks, and you’re running Databricks clusters, Event Hubs, and Functions. But each resource comes with a cost, and without clear visibility, these costs can spiral quickly, especially for startups or small businesses trying to extract value from their data without draining their budgets.</p><p><strong>Cost estimation is not just a finance responsibility; it is a data engineer’s responsibility.</strong></p><p><strong>The Experiment: Databricks vs. Azure Functions</strong></p><p>During the project, we tested two ingestion approaches:</p><ul><li>Azure Databricks</li><li>Azure Functions</li></ul><p>Both worked technically, but the real question was:</p><p><strong>“Which is more cost-effective while meeting our performance needs?”</strong></p><h3>Using the Azure Pricing Calculator to Estimate Costs</h3><p>To answer this, we relied on the <a href="https://docs.azure.cn/en-us/cost-management-billing/costs/pricing-calculator"><strong>Azure Pricing Calculator</strong></a>, a powerful tool that helps translate expected usage into estimated costs. Whether you’re a startup or an enterprise, this tool helps you plan and budget your Azure usage effectively by simulating different configurations, regions, and usage patterns.</p><h3>1️. Azure Databricks Cost Estimation</h3><p>When you apply this to your own projects, you can adjust the region to the nearest server for lower latency and potentially lower costs. But for this hands-on learning, I simply followed the video’s setup to focus on pipeline building and cost reasoning in context.</p><p>In the video, we used Australia East in the Azure Pricing Calculator:</p><ul><li>Instance: Standard DS3 v2 (14 GB RAM, 4 cores)</li><li>Tier: Premium (for RBAC &amp; Unity Catalog)</li><li>Usage: API calls every 30 seconds, so the cluster runs 24/7 (730 hours/month).</li></ul><p>The estimated costs were:</p><ul><li>Compute: ~$245/month</li><li>DBU: ~$401.50/month</li><li>Total: ~$546/month</li></ul><p>💡 Insight: Even a small Databricks cluster running continuously can quickly become a significant cost center.</p><h3>2️. Azure Functions Cost Estimation</h3><p>One of the most impressive aspects of Azure Functions is that you get 1 million executions per month completely free on the Consumption Plan.</p><p>For our 30-second API polling setup, we easily stay within this free tier:</p><ul><li>Total Cost: $0</li></ul><p>And even if you exceed 1 million executions, the cost is minimal — around $0.20 for each additional million executions.</p><p>💡 Insight: Azure Functions offer an incredibly cost-effective (often free) way to handle lightweight ingestion in your data pipelines.</p><p>For small businesses or startups, this is especially valuable. You may not need massive data processing, but you still need reliable ingestion for critical data streams like customer interactions, IoT sensor updates, or small-scale real-time APIs. Azure Functions let you build and test these pipelines without worrying about cost from day one, making experimentation and scaling much more accessible.</p><h3>Using Cost Analysis in the Azure Portal</h3><p>During the experiment, I also used Azure’s Cost Analysis tool:</p><ul><li>Event Hub: ~8 NZD (compute charges).</li><li>Databricks: ~1 NZD (only when the cluster was on briefly).</li><li>Azure Functions: $0.</li></ul><p>Our average daily spend was ~60 cents, demonstrating how small teams can run real-time streaming ingestion pipelines affordably.</p><h3>Performance Comparison: Databricks vs. Azure Functions</h3><h3>1. Azure Functions excel when:</h3><ul><li>Workloads are lightweight ingestion or transformation tasks.</li><li>You want serverless, event-driven execution.</li><li>Infrastructure management should be minimal.</li></ul><h3>2. Azure Databricks shines when:</h3><ul><li>Processing massive workloads (terabytes of data).</li><li>Performing complex Spark-based ETL pipelines.</li><li>Needing advanced orchestration and high scalability.</li></ul><h3>Key Takeaways for Data Engineers (Especially Azure Users)</h3><ul><li>Use the Azure Pricing Calculator before starting projects.</li><li>Always match resource choice with workload characteristics.</li><li>Use Azure Functions for lightweight ingestion to save costs.</li><li>Use Azure Databricks for heavy lifting only when needed.</li><li>Cost-aware architecture decisions can save your team thousands while maintaining performance.</li></ul><p>Learn more about the <a href="https://docs.azure.cn/en-us/cost-management-billing/costs/pricing-calculator">Estimate costs with the Azure pricing calculator</a></p><p>If you’re looking to collaborate on data projects or need consultation feel free to connect with me on LinkedIn or check out my website:<a href="https://nshutielise.github.io/"> https://nshutielise.github.io/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8434576b832c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>