<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Srikar Mogaliraju on Medium]]></title>
        <description><![CDATA[Stories by Srikar Mogaliraju on Medium]]></description>
        <link>https://medium.com/@srikar.mogaliraju?source=rss-9cf761c8df86------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DIAML7FuXBU625tFjaCUWw@2x.jpeg</url>
            <title>Stories by Srikar Mogaliraju on Medium</title>
            <link>https://medium.com/@srikar.mogaliraju?source=rss-9cf761c8df86------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:13:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@srikar.mogaliraju/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Learned SQL the Slow Way. That’s Why AI Doesn’t Scare Me. Can Today’s Juniors Say the Same?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/i-learned-sql-the-slow-way-thats-why-ai-doesn-t-scare-me-can-today-s-juniors-say-the-same-62d6433255cf?source=rss-9cf761c8df86------2"><img src="https://cdn-images-1.medium.com/max/2400/1*DzipqFBP2ebSkvPQlSjwXw.png" width="2400"></a></p><p class="medium-feed-snippet">Before YouTube tutorials and ChatGPT, we learned from MSDN pages, hex error codes and long waits on forums. Some thoughts on what that&#x2026;</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/i-learned-sql-the-slow-way-thats-why-ai-doesn-t-scare-me-can-today-s-juniors-say-the-same-62d6433255cf?source=rss-9cf761c8df86------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/i-learned-sql-the-slow-way-thats-why-ai-doesn-t-scare-me-can-today-s-juniors-say-the-same-62d6433255cf?source=rss-9cf761c8df86------2</link>
            <guid isPermaLink="false">https://medium.com/p/62d6433255cf</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Srikar Mogaliraju]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 19:57:34 GMT</pubDate>
            <atom:updated>2026-11-17T19:59:31.464Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[What 12 Years of Data Engineering Taught Me That No Certification Did]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">In 2014 I thought data engineering was about getting the job to turn green.</p><p class="medium-feed-link"><a href="https://medium.com/@srikar.mogaliraju/what-12-years-of-data-engineering-taught-me-that-no-certification-did-5ef6fcc7a7ca?source=rss-9cf761c8df86------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@srikar.mogaliraju/what-12-years-of-data-engineering-taught-me-that-no-certification-did-5ef6fcc7a7ca?source=rss-9cf761c8df86------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ef6fcc7a7ca</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Srikar Mogaliraju]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 16:11:35 GMT</pubDate>
            <atom:updated>2026-11-14T16:11:35.160Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[We Underestimated Microsoft Fabric CU Consumption. Here’s Where It Went]]></title>
            <link>https://medium.com/@srikar.mogaliraju/we-underestimated-microsoft-fabric-cu-consumption-heres-where-it-went-9b75078c467e?source=rss-9cf761c8df86------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b75078c467e</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Srikar Mogaliraju]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 10:50:02 GMT</pubDate>
            <atom:updated>2026-11-10T11:02:43.752Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A lakehouse that looked fine on paper, where the capacity units were actually going and the five questions I ask now before I size anything.</em></p><h3>It started with a WhatsApp message</h3><p>7:40 on a Tuesday morning. The client’s reporting lead pinged me asking why the leadership dashboard was still showing Monday’s numbers. Not an alert, not a ticket. Just a message with a screenshot and a question mark.</p><p>I opened the Fabric capacity metrics app expecting to see a failed pipeline. What I saw instead was an F64 sitting deep in the red. Interactive requests were getting rejected, background jobs were stuck in the queue and the overnight medallion run that should have been done by 5:30 was still writing Silver at a quarter to eight.</p><p>We had sized that capacity three months before. Spreadsheet, Microsoft reference deck, a 30 percent buffer on top. On paper we were using maybe half of it. What we found out that week was that the thing had been throttling on and off for close to three weeks and nobody noticed, partly because the 24 hour smoothing hides short bursts and partly because business users only look at a dashboard properly when it is wrong.</p><p>So this is where the CUs actually went. I’ll say upfront that none of it was clever. All four causes were things we knew about in theory. We just hadn’t put a line for them in the sizing model, and I don’t think most people do.</p><h3>What we thought we were buying</h3><p>Our sizing had three inputs. Daily source volume (about 40 GB across six systems), the number of pipelines we planned to build (22) and a guess at Power BI load based on report count and roughly 180 named users. Run that through Microsoft’s CU guidance, add the buffer, and you land on an F64. Commercial team liked it because it was a round number. Client liked it because it came in under what they’d budgeted.</p><p>The problem was the mental model behind the spreadsheet. We treated the capacity like a VM. So many cores, so many hours, this much headroom. A Fabric capacity doesn’t behave like that. It’s more like a shared credit pool that every workload draws from at very different rates, and some of those workloads draw when nobody asked them to. Background work gets smoothed over 24 hours, interactive work over a few minutes, so you can run well past 100 percent for a good while and see nothing, then get hit later when the smoothed debt catches up.</p><p>There was no row in our sheet for any of that. There was no row for maintenance, no row for what the reporting team would do once they had access and definitely no row for engineers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kWg0calhg0wKi98GoMyi_g.png" /></figure><h3>Where the CUs went</h3><p>We pulled a full week of consumption out of the metrics app, broke it down by item and operation and sat with it for an afternoon. Four things explained most of the gap.</p><p><strong>Nobody had scheduled any Delta maintenance.</strong> Bronze and Silver were being written by small, frequent pipeline runs, every 15 minutes for a couple of the sources. Small writes make small files. We had one Silver table with north of 40,000 parquet files under it. Every downstream notebook was paying for that on every run, and we had no OPTIMIZE or VACUUM anywhere in the schedule. I’m a bit embarrassed by this one because I’ve written the “remember to compact your tables” slide myself more than once. After we added a nightly OPTIMIZE pass the main Silver notebook went from 48 minutes to 19, and the CUs burned by the overnight run dropped by roughly 40 percent. The compaction job costs something too, obviously, but nowhere near what the fragmentation was costing.</p><p><strong>Semantic model refreshes that nobody owned.</strong> Eleven import mode models were set to refresh every 30 minutes between 7 am and 8 pm. Someone in the business had asked for “near real time” during UAT, the reporting team obliged, and nobody ever went back to it. That’s 26 full refreshes a day per model, each one re-reading entire tables from the lakehouse, against data that actually changed twice a day. Import refresh is one of the heaviest background operations on a capacity. We rewired the refreshes to trigger off the pipeline completion event instead of a timer and that bucket dropped by about 60 percent overnight.</p><p><strong>One report.</strong> A single leadership report had a card visual doing a distinct count of customers across the full fact table, no filter, no aggregation. Every time somebody opened it, Fabric ran that from scratch. Around 20 people opened it between 9 and 9:30 every morning, which on its own was enough to tip the capacity into interactive throttling. We built a small aggregation table in Gold and changed about six lines of DAX. Took one afternoon. It had been running like that for five weeks.</p><p><strong>Engineers on the production capacity.</strong> We had one capacity for everything. Exploratory notebooks, testing joins, re-running failed loads, all of it on the same pool serving the business. A notebook somebody left open at 6 pm holds its Spark session, and its CUs, until the session times out. On a normal weekday development work was something like a quarter of total consumption. We moved dev and test onto an F4 on pay-as-you-go, paused outside working hours, and that was honestly the cheapest and most effective change of the lot.</p><p>None of this is hidden knowledge. It’s all in the docs somewhere. What we were missing was a sizing habit that assumes these costs are real, rather than a model that only counted the pipelines we intended to build.</p><h3>What we changed</h3><p>We did not upgrade the capacity. That was the first thing the client asked for and the first thing I pushed back on. An F128 would have doubled the monthly bill and hidden the same four problems behind more headroom, and they would have come back in six months when the next three source systems landed.</p><p>Over about four weeks we did the following, roughly in order of how much effort each took. Split dev onto its own F4 and paused it outside office hours. Added nightly OPTIMIZE and a weekly VACUUM to every lakehouse table over a couple of GB. Rewired the semantic model refreshes to fire on pipeline completion. Rebuilt the one bad report on an aggregation table. And then the thing that mattered most long term, a weekly 15 minute capacity review with the metrics app open, where the top five items by CU each get a named owner and a question.</p><p>After that the capacity settled at around 55 percent average utilisation with peaks under 85, no throttling events in the next ten weeks and the overnight run finishing by 5:15 again. Monthly Fabric spend stayed exactly where it was. If anything we ended up with room to bring in two source systems the client had been holding back on.</p><p>The lesson I took from it isn’t really about Fabric. Any consumption priced platform, Databricks DBUs, Snowflake credits, Fabric CUs, punishes the gap between what you planned to run and what actually runs. The sizing model has to include the housekeeping, the reporting layer and the people, not just the pipelines.</p><h3>What I check on day one now</h3><p>Before I put a number on a Fabric capacity for anyone, these are the questions. If the answer to any of them is “we haven’t thought about it yet”, the number is wrong.</p><ol><li>Is dev on its own capacity, and can somebody pause it?</li><li>Who owns Delta maintenance, and is OPTIMIZE scheduled from sprint one, not sprint eight?</li><li>What’s the refresh cadence on every semantic model, and who actually asked for it?</li><li>Which single report or query is the most expensive thing on the capacity, and has anyone looked?</li><li>Is someone reading the capacity metrics app every week, or only when something breaks?</li></ol><p>Fabric makes it very easy to build and just as easy to overspend. The capacity you buy is a budget, not a machine. If you treat it that way from day one, the F-number you pick ends up mattering a lot less than you’d think.</p><p><em>If you’ve hit a CU culprit that isn’t on this list I’d like to hear about it in the responses. I’m collecting them for a follow-up on Direct Lake versus Import, which is where the next round of surprises lives.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b75078c467e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>