<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roland Wenzlofsky on Medium]]></title>
        <description><![CDATA[Stories by Roland Wenzlofsky on Medium]]></description>
        <link>https://medium.com/@r.wenzlofsky?source=rss-d2fd690c84------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gPRKdO_1ymihwP9m5Y3oqg.jpeg</url>
            <title>Stories by Roland Wenzlofsky on Medium</title>
            <link>https://medium.com/@r.wenzlofsky?source=rss-d2fd690c84------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:07:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@r.wenzlofsky/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Pay-as-You-Go to Pricing Lock-In]]></title>
            <link>https://medium.com/@r.wenzlofsky/from-pay-as-you-go-to-pricing-lock-in-e308ff9fc169?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/e308ff9fc169</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 13:27:53 GMT</pubDate>
            <atom:updated>2026-08-17T13:27:53.192Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WxfWPlmz0dFVT4FSZzx3jg.png" /></figure><h3>Rethinking Cloud Pricing in the Age of AI</h3><h4>Part 2: Governance, Predictability and Pricing Lock-In</h4><h3>Predictability Depends on Governance</h3><p>One obvious counterargument to the discussion in the previous section is that modern cloud platforms already provide sophisticated governance mechanisms. Budgets, resource monitors, alerts, quotas, detailed usage reporting, and automated notifications enable organizations to maintain a high degree of financial control despite highly dynamic consumption models. At first glance, this appears to contradict the argument that increasingly granular billing makes operating costs more difficult to predict.</p><p>The observation is entirely correct.</p><p>Modern cloud platforms are capable of providing remarkably predictable operating costs. The important distinction is not whether predictability can be achieved, but rather <strong>how</strong> it is achieved.</p><p>Traditional enterprise environments derived much of their financial stability from relatively fixed infrastructure. Once a platform had been sized appropriately, day-to-day workload fluctuations generally had only a limited influence on monthly operating expenditure. Hardware eventually required expansion, storage increased, and licensing evolved over time, but the relationship between daily workload behavior and monthly infrastructure cost remained comparatively weak.</p><p>Consumption-based environments operate according to a different principle.</p><p>Infrastructure costs increasingly follow workload behavior. Predictability therefore depends less on installed capacity and more on the organization’s ability to understand, monitor, and govern resource consumption. Budgets must be defined, alerts maintained, quotas configured, chargeback mechanisms implemented, and workload behavior continuously analyzed. Predictable operating costs therefore become the result of operational governance rather than a by-product of infrastructure ownership.</p><p>This represents a subtle but important shift in the economics of operating analytical platforms.</p><p>Predictability has not disappeared.</p><p>Its source has changed.</p><h3>The Emergence of FinOps</h3><p>This shift helps explain why an entirely new operational discipline has emerged during the cloud era.</p><p>FinOps is often described as a financial management framework for cloud computing. While technically correct, this description understates its significance. FinOps exists because consumption-based infrastructure introduced operational activities that simply did not exist to the same extent in traditional environments.</p><p>Historically, database professionals concentrated on workload management, statistics collection, indexing strategies, SQL optimization, partitioning, and storage management. Their primary objective was to maximize system performance while maintaining operational stability.</p><p>Modern cloud platforms extend these responsibilities considerably.</p><p>Organizations now optimize warehouse sizing, auto-suspend thresholds, reserved capacity, storage lifecycle policies, scheduling decisions, workload placement, and resource utilization with the same discipline previously applied to SQL tuning. Performance optimization has gradually been complemented by financial optimization.</p><p>This evolution should not be interpreted as a disadvantage of cloud computing. Quite the opposite. Effective governance frequently produces substantial cost savings and enables organizations to align infrastructure expenditure much more closely with actual business activity.</p><p>Nevertheless, governance is not free.</p><p>Like every operational activity, it consumes time, expertise, tooling, and organizational effort. The cloud did not eliminate operational responsibility. It redistributed it.</p><h3>Precision Comes with Complexity</h3><p>One of the greatest advantages of increasingly granular pricing models is the visibility they provide into resource consumption. Organizations can identify expensive applications, departments, dashboards, users, and, in some environments, even individual queries. Workload costs that previously disappeared inside a shared infrastructure budget can now be attributed with remarkable precision.</p><p>This visibility enables better decisions.</p><p>At the same time, however, it also changes the questions organizations begin asking.</p><p>Instead of simply asking whether a workload performs well, organizations increasingly ask whether it performs economically.</p><p>Which dashboard generated yesterday’s increase in compute consumption?</p><p>Which AI agent executed thousands of additional queries overnight?</p><p>Which automated workload justifies its operating expense?</p><p>Which department should pay for shared analytical services?</p><p>Performance engineering therefore begins to intersect with financial engineering.</p><p>Database professionals continue optimizing SQL performance, but they increasingly optimize workload economics as well. Understanding the cost implications of architectural decisions gradually becomes as important as understanding their performance characteristics.</p><p>The objective has not changed.</p><p>Organizations still seek efficient analytical platforms.</p><p>The definition of efficiency, however, has expanded beyond technical performance alone.</p><h3>Every Organization Has a Different Price for Predictability</h3><p>Discussions about cloud economics often assume that minimizing infrastructure expenditure is the primary objective.</p><p>Business decisions are rarely that simple.</p><p>Every organization assigns a different value to financial predictability.</p><p>A rapidly growing technology company with highly dynamic workloads may willingly accept fluctuating monthly operating costs if consumption-based pricing minimizes overall expenditure. A multinational financial institution operating within strict budgeting processes may reach a completely different conclusion. For such organizations, predictable infrastructure costs often simplify procurement, financial forecasting, executive reporting, and long-term investment planning. Paying a modest premium for financial stability may therefore represent a perfectly rational business decision.</p><p>This observation helps explain why the cloud market has not converged toward a single commercial model.</p><p>Consumption-based pricing and reserved capacity continue to coexist because they optimize different business objectives.</p><p>Customers are not merely selecting infrastructure.</p><p>They are selecting the economic model that best reflects how their organization operates.</p><h3>Pricing Models Continue to Evolve</h3><p>The evolution of pricing models is by no means unique to cloud computing.</p><p>Many readers will remember the early years of mobile telecommunications, when providers introduced increasingly sophisticated tariffs designed to align prices more closely with network utilization. Charges varied according to the time of day, weekend usage, destination networks, or promotional campaigns. From a theoretical perspective, these models represented an efficient method of allocating network costs according to actual consumption.</p><p>Over time, however, customer priorities changed.</p><p>Many providers gradually transitioned toward flat-rate subscriptions, not because they necessarily represented the most efficient pricing model from the operator’s perspective, but because customers increasingly valued simplicity and predictable monthly bills over highly optimized tariff structures.</p><p>Whether cloud computing will eventually follow a similar evolution remains uncertain.</p><p>The comparison serves a different purpose.</p><p>It reminds us that pricing models should never be regarded as permanent. They evolve together with technology, competition, customer expectations, and changing patterns of resource consumption.</p><h3>Pricing Lock-In</h3><p>This observation leads to another aspect of cloud computing that receives surprisingly little attention.</p><p>Discussions about vendor lock-in almost exclusively focus on technology. Organizations evaluate proprietary SQL dialects, storage formats, APIs, migration complexity, or the effort required to move applications between platforms. These considerations remain important and should continue to influence platform selection.</p><p>Cloud computing, however, introduces another form of dependency.</p><p>Organizations adopting a cloud platform are not merely adopting its technology.</p><p>They are also adopting its pricing philosophy.</p><p>This distinction may appear subtle, yet its long-term implications can be significant.</p><p>If a provider changes its pricing model, the economics of an application may change considerably even though the application itself remains technically identical. Unlike technical lock-in, pricing lock-in does not require software to be rewritten before its effects become visible. The workload continues to execute exactly as before. Only its financial characteristics change.</p><p>The recent evolution of the AI industry provides an interesting example.</p><p>The first generation of generative AI products achieved widespread adoption through relatively simple subscription models. As enterprise adoption accelerated and AI workloads expanded dramatically, providers increasingly introduced or expanded token-based pricing for APIs, premium models, and autonomous agent frameworks. Organizations therefore discovered that AI costs increasingly resembled utility bills rather than traditional software licenses.</p><p>Nothing fundamental about the underlying applications necessarily changed.</p><p>The pricing philosophy did.</p><p>This observation does not imply that token-based pricing is inappropriate. Consumption-based pricing frequently represents a fair and economically efficient mechanism for allocating infrastructure costs.</p><p>It does, however, demonstrate that organizations evaluating strategic cloud platforms should consider not only technical architecture but also long-term pricing strategy. A platform may be technically straightforward to migrate away from while remaining deeply embedded in the economics of the business through its pricing model.</p><p>Technical lock-in and pricing lock-in therefore deserve to be evaluated with equal attention.</p><p>The final part of this article examines how artificial intelligence may once again change the economics of cloud computing and why future pricing models may eventually optimize something very different from infrastructure utilization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e308ff9fc169" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Pay-as-You-Go to Pricing Lock-In]]></title>
            <link>https://medium.com/@r.wenzlofsky/from-pay-as-you-go-to-pricing-lock-in-fa34ef7e6981?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa34ef7e6981</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 13:20:21 GMT</pubDate>
            <atom:updated>2026-08-17T13:20:21.499Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIQniWV3OOwPOStfM0UhCA.png" /></figure><h3>Rethinking Cloud Pricing in the Age of AI</h3><h4>Part 1: The Evolution of Cloud Pricing</h4><h3>Introduction</h3><p>Vendor lock-in has been discussed almost since the first enterprise databases appeared. Traditionally, the discussion has focused on technical aspects such as proprietary SQL dialects, storage formats, APIs, migration complexity, or the effort required to move applications from one platform to another. These topics remain highly relevant, particularly as cloud vendors continue expanding their ecosystems with proprietary services.</p><p>There is, however, another form of dependency that receives considerably less attention: <strong>pricing lock-in</strong>.</p><p>Unlike technical lock-in, pricing lock-in does not arise because an application becomes difficult to migrate. Instead, it emerges when the economics of a business become closely coupled to a provider’s pricing philosophy. An application may remain technically unchanged for years, yet its operating costs can change significantly simply because the provider modifies the way resources are billed. In other words, the application continues to execute the same workload, but its financial characteristics change without a single line of code being modified.</p><p>This distinction becomes increasingly important as cloud platforms evolve. Modern analytical databases are no longer competing solely through performance, scalability, or functionality. They increasingly compete through different economic models, each designed to optimize different customer priorities. Understanding these models therefore becomes just as important as understanding the underlying technology.</p><p>The objective of this article is not to argue that one pricing model is superior to another. Every model offers advantages and disadvantages depending on workload characteristics and business objectives. Instead, the article examines how cloud pricing has evolved, why multiple pricing philosophies continue to coexist, and why pricing itself may become another important dimension of vendor lock-in as artificial intelligence changes the way organizations consume compute resources.</p><h3>Pricing Models Are Business Models</h3><p>Cloud pricing is often discussed as though it were a technical characteristic of a platform. In reality, technology merely determines what can be measured, while the pricing model reflects a commercial decision regarding which of those measurable resources become relevant for billing.</p><p>Modern cloud platforms are capable of measuring warehouse runtime, bytes processed, storage consumption, query execution, network traffic, or even the resources required by individual AI workloads. Choosing which of these metrics ultimately becomes the commercial basis for charging customers is therefore primarily a business decision enabled by technology rather than an engineering requirement.</p><p>This distinction is important because pricing models optimize different business objectives.</p><p>A consumption-based model aligns revenue closely with customer activity. As customers execute more workloads and derive greater value from the platform, provider revenue increases accordingly. From the vendor’s perspective, this creates an attractive business model because commercial success naturally follows customer success.</p><p>Customers, however, do not always optimize for the same objective.</p><p>Some organizations prioritize elasticity and maximum infrastructure utilization. Others place greater emphasis on stable operating costs that simplify budgeting, procurement, and financial forecasting. Large enterprises often value predictability because infrastructure costs become part of long-term financial planning rather than simply operational expenditure.</p><p>Consequently, pricing models should not be viewed as competing technical implementations. They represent different commercial strategies that address different customer requirements.</p><h3>The Evolution of Compute Billing</h3><p>The evolution of cloud pricing can largely be described as a gradual movement toward increasingly granular measurement of resource consumption.</p><p>A simplified progression looks as follows:</p><ul><li>Dedicated infrastructure</li><li>Dedicated virtual warehouses</li><li>Warehouse runtime billing</li><li>Per-second warehouse billing</li><li>Workload-aware billing</li><li>Query-level billing</li><li>Potential future billing per AI task or autonomous agent</li></ul><p>The common objective throughout this evolution has been to align infrastructure costs more closely with actual resource consumption. The assumption is straightforward: if resources can be measured more precisely, they can also be charged more precisely.</p><p>Recent innovations illustrate this trend particularly well.</p><p>Snowflake’s Adaptive Warehouses, for example, automatically adjust compute allocation according to workload characteristics, reducing the amount of manual warehouse sizing traditionally performed by administrators. Whether or not individual customers choose this model, it clearly demonstrates a broader industry direction in which infrastructure management increasingly shifts from administrators toward software.</p><p>Similar developments can be observed across the industry. Platforms increasingly automate scaling decisions, workload placement, and resource allocation while exposing increasingly detailed consumption metrics to customers. From an engineering perspective, this represents remarkable progress. From a commercial perspective, however, it also introduces new pricing possibilities because consumption can now be measured with much greater precision than was previously possible.</p><h3>The Market Is Moving in Two Directions</h3><p>At first glance, these developments might suggest that the cloud industry is steadily converging toward increasingly granular consumption-based pricing.</p><p>Interestingly, the market itself tells a different story.</p><p>While providers continue investing heavily in sophisticated consumption models, many simultaneously offer pricing models designed specifically for customers who prefer predictable operating costs.</p><p>Google BigQuery, for example, supports both on-demand pricing and reserved capacity. Amazon Redshift offers provisioned clusters alongside serverless deployments. Microsoft Fabric is largely built around dedicated capacity. Snowflake continues to support traditional warehouse-based pricing in addition to newer workload-aware approaches.</p><p>This coexistence is unlikely to be accidental.</p><p>If one pricing philosophy were universally superior, maintaining multiple commercial models would provide little benefit. Instead, cloud providers increasingly acknowledge that customers optimize for different business objectives. Some organizations value maximum elasticity and are willing to accept fluctuating monthly costs. Others deliberately choose predictable spending because financial stability is itself considered a competitive advantage.</p><p>An interesting observation follows from this.</p><p>Rather than converging toward a single pricing philosophy, the cloud industry appears to be expanding customer choice. Providers are no longer asking which pricing model is objectively best. Instead, they increasingly allow customers to select the model that best matches their workload characteristics and business priorities.</p><p>This observation also weakens a common misconception frequently encountered in cloud discussions. The evolution of cloud platforms is not simply a journey from dedicated infrastructure toward ever finer consumption pricing. Instead, it is becoming a portfolio of commercial models designed for different operational requirements.</p><h3>Billing Granularity and Cost Predictability</h3><p>One of the defining characteristics of traditional enterprise infrastructure was the relative stability of its operating costs. This should not be interpreted as suggesting that costs never changed. Hardware eventually required expansion, storage increased over time, and software licensing evolved as environments grew. Nevertheless, daily workload fluctuations generally had comparatively little influence on monthly infrastructure expenditure.</p><p>Consumption-based pricing fundamentally changes this relationship because operating costs become increasingly dependent upon workload behavior rather than installed capacity.</p><p>Monthly expenditure may now be influenced by query complexity, execution frequency, dashboard refresh intervals, concurrent workloads, ETL activity, AI-generated SQL, autonomous agents, or automatic resource allocation decisions made by the platform itself. Capacity planning therefore gradually evolves into workload planning, while financial forecasting increasingly depends upon understanding how applications and users consume resources.</p><p>This observation is sometimes interpreted as implying that consumption-based environments are inherently less predictable. That conclusion would be inaccurate.</p><p>Modern cloud platforms provide extensive governance capabilities, including budgets, quotas, alerts, resource monitors, and sophisticated reporting mechanisms. With appropriate operational controls, organizations can achieve highly predictable operating costs even within dynamic consumption-based environments.</p><p>The important distinction is therefore not whether predictable costs are achievable.</p><p>It is <strong>how</strong> they are achieved.</p><p>Traditional environments derived much of their financial stability from relatively fixed infrastructure. Modern cloud environments increasingly derive predictability from governance, monitoring, and continuous operational management.</p><p>That difference becomes important because governance itself introduces another dimension of operational effort — one that has given rise to an entirely new discipline that will be examined in the next section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa34ef7e6981" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Wrong Response to a Shifting Market]]></title>
            <link>https://medium.com/@r.wenzlofsky/the-wrong-response-to-a-shifting-market-e0554abd0437?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0554abd0437</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[job-hunting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[jobs]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 22:11:52 GMT</pubDate>
            <atom:updated>2026-04-20T22:11:52.852Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UDwZ4ik9M31cY8R4yj-A5g.png" /></figure><p><em>Three default moves senior technical practitioners make when rates begin to compress, and why each makes the underlying position worse.</em></p><p>A familiar pattern is visible among senior technical practitioners. The market for their services has begun to shift. Rates are softer than they were three years ago. Engagements are shorter. Junior teams, augmented by AI tooling, are producing work at speeds that would have been implausible in 2022. The senior practitioner responds. Usually in one of three ways.</p><p>The practitioner often feels the shift before naming it. The rate resistance at the last contract negotiation. The client who asked what AI could do for their project. The dinner conversation where a younger colleague described delivering in two weeks what used to take two months. The referral that did not come this quarter. These details accumulate into a background disquiet that sits behind the calendar and the client calls, and that is usually not spoken of with anyone — not with colleagues, who are competitors, and not with clients, to whom the disquiet cannot be shown.</p><p><strong>The first response is to work harder</strong>. More hours, more projects, a tighter schedule, perhaps a second engagement running in parallel with the first. <strong>The second is to learn the newest platform</strong> — to certify in whatever the current fashion is, whether Databricks, Snowflake, or one of the AI-adjacent specialties — on the assumption that additional technical breadth will restore the lost pricing power. <strong>The third is to wait</strong>: to assume that the current market conditions are a temporary distortion and that rates will recover once the AI hype cycle settles and enterprises remember the value of experience.</p><p><strong>All three responses are wrong.</strong> In most cases, they make the underlying position worse.</p><p>What they have in common is more important than what distinguishes them. Each of the three assumes that the game being played has not changed, only that the difficulty has increased. Work harder at the same game. Certify for the same game. Wait out a bad patch of the same game. The evidence is against this. The game has changed. The pricing structure of senior technical implementation work is being reset, not temporarily suppressed. Competing harder, credentialing more, or waiting longer within the old structure produces worse outcomes than repositioning outside of it.</p><h3>Working Harder</h3><p>The instinct to work harder is the most natural response. It is also the most costly. It is natural because every senior practitioner has built their career on exactly this response. Harder work solved earlier problems — the tight deadline, the difficult client, the underperforming colleague, the technology transition. The assumption that it will solve this one as well is defensible on past evidence. It is also wrong.</p><p>The exhaustion has a different character at fifty-five than it had at forty. It is physical, not emotional. A Saturday does not restore it. The body registers cumulative load differently than it did twenty years earlier, and the practitioner notices this, usually in private, and usually without mentioning it. To mention it would be to admit that the terms of trade have shifted. Most practitioners are not yet ready to admit that, even to themselves.</p><p>The problem with working harder in the current market is not that it fails to produce results. It produces results. Each additional engagement billed, each weekend worked, each parallel project delivered translates into revenue. <strong>The problem is that each unit of effort produces less revenue than it did three years ago,</strong> and the trajectory is downward rather than flat. Working harder at a rate that is being compressed means earning more hours of labor but less per hour. The additional hours do not compensate for the lower rate.</p><p>There is a second problem, which matters more than the revenue question. Working harder consumes the exact resource that would make repositioning possible. Repositioning — the development of a different service, a different pricing structure, a different market position — requires time to think, to write, to build relationships, to design offerings that do not yet exist. A practitioner working sixty-hour weeks at the existing practice has no such time. The harder they work at the current model, the less capacity they have to construct the next one.</p><p>Consider a pattern. A senior freelance consultant in European banking, with twenty years of experience as an Informatica specialist, charged his peak rate in 2022 for roughly 150 billable days a year. By 2025, under rate pressure and with shorter engagements, he accepts roughly 20 percent less per day but bills 40 percent more days across parallel projects. His gross revenue is roughly flat. His exhaustion is higher. His rate trajectory is still downward. Over three years, he has produced no publications, no specialized offerings, no partnerships — no assets that would allow him to reposition when the current rate compresses below the level at which additional days can compensate.</p><p>The cost compounds. A senior practitioner who spends three years working harder at an eroding rate has, at the end of those three years, lower rates, depleted energy, no new assets, and no repositioning in progress. A practitioner who spent the same three years working somewhat less but building a differentiated position has, at the end, a practice that commands what the old one did or more. The first path looks prudent year by year. The second looks prudent only when the full arc is visible. Most practitioners do not see the arc until the third or fourth year. By then, the gap between the two paths has become difficult to close, and the recognition itself is painful in a specific way — the pain of having worked very hard on something that was worsening the situation.</p><h3>Learning the Newest Platform</h3><p>The second response is more sophisticated but fails in a different way. Certifying on the current platform of record — Databricks today, whatever the market favors eighteen months from now — feels like the exact kind of professional development that the situation demands. The reasoning is straightforward. Rates are compressing in the old specialty. The new specialty is where the budget is moving. Credentialing in the new specialty should therefore restore the lost pricing.</p><p>This reasoning is wrong in three ways.</p><p>The first is that certification lags market demand by 12 to 18 months, meaning that by the time a senior practitioner completes the certificate on the current platform of interest, junior practitioners have already accumulated meaningful project experience in it and are being priced accordingly. The certificate does not close that gap. It credentials a position that younger competitors already occupy with both certification and experience. The senior practitioner arrives at the new platform as a late entrant, without the cost advantage that typically allows late entrants to compete.</p><p>The second is that the premium historically paid for deep expertise is not being paid, in the current market, for deep expertise in the newest platform. It is being paid for deep expertise in legacy systems that regulated industries are still running and cannot easily replace. A practitioner with twenty years of enterprise data warehousing experience is paid a premium for that experience because banks and insurers cannot easily source it. A practitioner who has recently certified on Databricks is paid roughly what any Databricks-certified consultant is paid, with general seniority adjusted modestly upward. The premium attaches to scarce expertise, not to fashionable expertise.</p><p>The third is that the treadmill does not stop. A practitioner who certifies on the current platform this year will need to certify on the next one in two years, and the one after that in four. Each certification cycle consumes months of effort that do not produce revenue, and the underlying pricing structure of whatever the current platform is continues to compress as AI tooling reaches it. The treadmill is not a path to a better position. It is a mechanism for preserving the current position at increasing cost, and for most practitioners, it ends in exhaustion rather than in repositioning.</p><p>There is a further cost, less visible in the accounting. The senior practitioner who certifies on a new platform is, within that platform&#39;s social structure, a junior. They sit in courses next to practitioners half their age. They answer certification exam questions about concepts they have encountered for 6 months, while colleagues around them have worked with those same concepts for 3 years. The asymmetry is not cruel — it is simply the fact that the platform is new and they are new to it. But the experience does something to a practitioner who has spent two decades being the person consulted rather than the person learning. No amount of framing it as humility removes what the body registers. Something has been relinquished that was not relinquished voluntarily.</p><p>Consider the practitioner who acts on this reasoning. A senior Oracle DBA with eighteen years in Austrian insurance, observing that Oracle engagements are shortening, completes a Databricks certification in 2024 at a cost of six months of part-time study. He expects rates comparable to his legacy Oracle work. His first Databricks engagement in 2025 pays roughly 35 percent less than his previous Oracle rate, which is the going rate for newly certified Databricks consultants regardless of seniority. <strong>His 18 years of Oracle experience are seen by the market as irrelevant to the new role</strong>. His prior specialty — still in demand at banks and insurers running legacy systems — has been de-emphasized during the transition. Reclaiming it now would require rebuilding relationships he allowed to lapse.</p><p>None of this argues against learning new platforms. It argues against treating certification as the repositioning itself. Platform competence is necessary to remain credible in the market. It is not sufficient to command premium pricing, and treating it as sufficient is the error.</p><h3>Waiting for the Market to Correct</h3><p>The third response is the quietest and, in some ways, the most damaging. It consists of continuing the existing practice substantially unchanged, reducing rates modestly when necessary to retain engagements, and assuming that the current market conditions are a temporary distortion produced by AI hype that will normalize within a reasonable period. The premise is that experience will reassert its value once the current enthusiasm for AI-augmented junior teams meets the reality of what those teams can actually deliver in regulated enterprise environments.</p><p>The premise contains some truth, which is why it is seductive. AI tooling in 2026 is producing work of uneven quality at enterprise scale, and a correction will occur as organizations discover the limits of the current approach. The expected correction, however, is not a return to the previous market structure. It is a reshaping of the market into a structure in which AI-augmented teams remain central and senior practitioners are absorbed into different roles within that structure — architecture, governance, advisory, exception handling — rather than restored to the implementation work they previously occupied.</p><p>Waiting, therefore, is not a neutral posture. It is a bet that the market will return to its previous state. That bet is unlikely to pay out, and while it is not paying out, the practitioner who has made it is burning through reputational capital and cash reserves in a practice that is slowly contracting. The patient strategy becomes, over time, a liquidation strategy.</p><p>The waiting produces a specific kind of loneliness. Colleagues who repositioned are too busy with their new practices to describe how they did it. Colleagues who did not will not say aloud that they did not. Spouses may sense that something is wrong, but usually do not know the specifics of day rates or engagement structures well enough to help. The practitioner carries the situation alone, often for years, while the external evidence accumulates quietly — the referral that did not come, the renewal that was not offered, the invitation to the advisory board that went to someone younger. None of these events, taken individually, is a crisis. Taken together, they describe a trajectory. The practitioner recognizes the trajectory before anyone around them does, and usually before they are ready to act on the recognition.</p><p>Consider the practitioner who takes this path. A senior data warehouse architect with 22 years of experience across European banking and telecom, respected in his sector, chose in 2022 to maintain his practice unchanged on the assumption that rates would recover. Between 2022 and 2025, his day rate declined by roughly 30 percent through a series of individually reasonable concessions to retain clients. Engagements grew shorter. Referrals slowed as his peers repositioned into adjacent areas where he did not appear. By 2025, his reputation was intact, but his practice had contracted significantly. He had produced no new assets — no book, no partnerships, no differentiated offerings — that would allow him to move now from a weaker starting position than he held three years earlier. The three years of waiting did not preserve his position. They eroded it slowly enough that no single year’s erosion felt serious, until the cumulative effect became visible.</p><p>The asymmetry is clear. Waiting has a low ceiling and a substantial floor risk. Repositioning has a higher ceiling and a comparable floor risk. The expected value of waiting is almost certainly lower than that of repositioning, and the distribution is worse on the downside as well.</p><h3>The Common Error</h3><p>Each of the three responses shares a hidden premise. <em>The premise is that the market is the same market it was three years ago, only more difficult, and that the correct response is to compete harder, credential more, or wait patiently within the existing structure. The premise is false.</em></p><p>The market has not become more difficult. It has become different. Implementation work at the senior level — the work of building, tuning, and delivering data platforms or equivalent enterprise technology — is being systematically repriced downward because AI tooling is compressing the time required to produce it. This is not a temporary distortion. It is a structural shift in the underlying economics, and it is likely to continue and intensify over the next several years rather than reverse.</p><p><strong>What is not being repriced downward, or at least not to the same degree, is a different kind of work</strong>. Architectural judgment about which systems to build and why. Advisory work on regulatory obligations, risk frameworks, and the interaction between business strategy and technical constraints. Design work at the boundary between business domains and technical implementation, where the translation requires deep experience in both. Trust-based relationships with senior stakeholders who rely on the practitioner for judgment rather than for hours. These categories of work are harder to compress with AI tooling because they depend on context, relationships, and tacit knowledge that no tool can produce.</p><p>The senior practitioner whose revenue is primarily from implementation is competing in the compressed market. The senior practitioner whose revenue is primarily advisory, architectural, or judgment-based is competing in a stable or growing market. The correct response to the current shift is not to work harder at implementation, not to certify in new implementation platforms, and not to wait for implementation rates to recover. It is to shift the practice&#39;s center of gravity away from implementation and toward the categories of work that are not being compressed.</p><p>This is a slower response than any of the three wrong responses. It requires changing what the practice sells, not just how much is sold or at what price. Most practitioners resist this because the existing offering is familiar and the new one is not, because the transition itself creates revenue uncertainty, and because the work of repositioning is qualitatively different from the work they have been doing for 20 years.</p><p>The resistance has another source, which is worth naming even if dwelling on it would not help. A practice built over twenty or twenty-five years is not only a source of income. It is a structure of identity. The specific technologies, client relationships, reputation, and way of introducing oneself at industry events — these are not interchangeable parts that can be swapped out when market conditions change. They are what the practitioner became over the course of their adult working life. Asking a senior practitioner to reposition is not asking them to change jobs. It is asking them to become a different professional self while continuing to support whatever dependents, obligations, and plans were built on the existing self. The difficulty of this is not an excuse. But pretending it is not difficult is the reason so much advice on this topic fails to reach the people it claims to be for.</p><h3>The Direction of a Correct Response</h3><p>The full shape of a correct response is beyond the scope of a single essay and will be developed at length in subsequent work. The direction, however, can be stated in general terms.</p><p>A correct response begins not with activity but with diagnosis. Before new services are built, new pricing structures are designed, or new market positions are pursued, the practitioner should conduct an honest assessment of what the existing practice actually sells, to whom, and on what basis. <em>The assessment is uncomfortable because it often reveals that the practice is selling implementation hours to buyers who do not distinguish between senior and junior implementation, which is the position from which rates compress fastest.</em></p><p>The uncomfortable part is not the logical recognition. It is what happens when a practitioner, alone with a spreadsheet or a list of past engagements, reaches the conclusion that the answer to “why would a client still pay premium rates for this” is thinner than they had been telling themselves. There is usually a moment in the diagnosis when what had been suspected becomes undeniable. The moment is unpleasant. It is also the moment from which every subsequent good decision follows.</p><p>The diagnosis is followed by a decision about where in the value chain the practice should operate. The options are not infinite. Most senior technical practitioners, for structural reasons, will find that the correct position is somewhere in the territory bounded by architecture, advisory, specialized design, and trust-based long-term advisory relationships with enterprise clients. The position depends on the practitioner’s credentials, relationships, and areas of deepest expertise. It is not a matter of choosing freely from a menu but of identifying which of the limited number of viable positions the practitioner is best credentialed to occupy.</p><p>Once the target position is identified, the repositioning consists of three parallel efforts. The first is the construction of offerings that serve that position, which usually means retainer-based advisory, fixed-fee architecture engagements, or specialized consulting packages rather than hourly implementation. The second is the development of assets that establish credibility in the new position — publications, partnerships, case studies, and credentials that signal to buyers that the practitioner operates at the advisory rather than implementation level. The third is the gradual migration of existing clients toward engagements that fit the new position, and the selective acquisition of new clients who arrive already seeking the advisory work rather than the implementation work.</p><p>Three worked examples illustrate what this looks like in practice.</p><p>The first is a senior IBM DataStage consultant in European banking who, in 2024, stopped accepting any implementation engagements. She restructured her offering around fixed-fee architecture reviews for bank-wide data platform migrations, priced at a premium that reflects the architectural value rather than the implementation time. She now runs four of these per year, supplemented by two advisory retainers. Her gross revenue is roughly 15 percent higher than in 2022. Her effort is lower. She no longer competes on day rates against AI-augmented junior teams, because her offering is not priced in days. The move required her to turn down implementation engagements for nine months while the new offering established itself. The period felt costly at the time. In retrospect, it was the narrow window in which the repositioning was even possible.</p><p>The second is a senior SAP consultant at an Austrian insurance company who repositioned as a regulatory advisory specialist for finance compliance. Rather than billing implementation hours, he now runs multi-year advisory retainers with a small number of clients. His implementation work has declined to near zero, and he considers that an improvement rather than a loss. The advisory retainer format makes him nearly immune to AI-driven price compression because the value he provides is regulatory interpretation and risk judgment — work that depends on his twenty years of observed regulatory behavior across European financial institutions and that no tool can produce. His gross income is roughly 20 percent higher than at his peak billable-hour practice in 2021, from approximately 40 percent fewer working hours.</p><p>The third is a senior data warehouse architect in DACH telecom who invested eighteen months in writing a book on enterprise migration strategy. The book did not earn meaningful royalties and was never intended to. It anchored his credibility as an authority on migration architecture. That credibility allowed him to reposition from hourly implementation to fixed-fee engagements at the strategy level, where his twenty-five years of experience finally commanded a premium that reflects their actual value rather than a premium compressed by commoditization in implementation work. His total billed days per year declined by 40 percent between 2022 and 2025. His gross income rose by 30 percent over the same period. The time freed by fewer billable days is being invested in a second book and a conference speaking program that he expects will drive the next stage of premium pricing.</p><p>The three examples illustrate different forms of the same move: changing the pricing structure from days to a fixed fee, changing the service from implementation to advisory, and using content assets to anchor premium pricing for work that is substantively similar to what was sold before. None of the three practitioners learned a new platform as the primary repositioning move. None worked harder than they had been working. None waited for the market to recover. Each constructed a position that the compressed implementation market does not reach.</p><p>None of this is fast. The repositioning timeline for most practitioners is 2 to 3 years, not 2 to 3 months. The practitioner who begins the work in year one is substantially repositioned by year three. The practitioner who begins in year three is starting the repositioning work from a weaker financial and reputational position than they would have had in year one, and the process is correspondingly harder.</p><h3>What to Do This Quarter</h3><p>The most useful question a senior technical practitioner can ask this quarter is not “what new skill should I learn,” or “which platform should I certify on,” or “when will the market recover.” It is a different question. What position should this practice occupy, and what would have to be true for it to occupy that position in three years?</p><p>Answering that question well is the beginning of the correct response. The answer, when it is honest, will usually reveal that the practice is not currently positioned where it needs to be in three years, and that deliberate reconstruction is required. The reconstruction is the work. Working harder at the existing position, certifying on a new platform, or waiting for a correction are substitutes for the work. They feel productive because they are familiar and because they produce visible activity. They are not productive. The work — the diagnosis, the repositioning, the gradual reconstruction of what the practice sells and to whom — is what produces durable outcomes in a market that will not return to what it was.</p><p>The senior technical practitioner who begins this work in 2026 will be substantially repositioned by 2028. The practitioner who defaults to one of the three wrong responses will, in 2028, be in a materially worse position than today. <strong>The market will not return to its previous structure</strong>. The practitioners who act on that premise, now, will find that senior expertise still commands premium rates — in a market they have deliberately constructed rather than one they have waited to be restored.</p><p>One more observation. The practitioners who are currently navigating this, often alone and usually without speaking to anyone about it, are not few. The specific experience of being a senior technical expert between forty-five and sixty in the current market — watching a practice built across two decades or more shift under conditions that were not foreseen and cannot be easily reversed — is not an individual failing. It is a demographic condition. It is happening to a large population at once, across several countries and specialties, at the same time. Recognizing that it is shared is not a solution. But the isolation in which the situation is usually faced makes it harder to see clearly than it needs to be, and makes the first necessary step — the honest diagnosis — harder to begin. The first step is the diagnosis. The second, nearly as useful, is the recognition that the diagnosis does not belong solely to the practitioner.</p><p><em>The examples in this essay are composites drawn from patterns observed across European enterprise consulting. They illustrate repositioning patterns but do not describe specific individuals.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0554abd0437" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Does Not Fail Because of the Model. It Fails Because of the Data.]]></title>
            <link>https://medium.com/@r.wenzlofsky/rag-does-not-fail-because-of-the-model-it-fails-because-of-the-data-2e1c5cd0cc34?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e1c5cd0cc34</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Sun, 12 Apr 2026 13:22:50 GMT</pubDate>
            <atom:updated>2026-04-12T13:22:50.713Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*05thOO_c-7M3kate_QkWcA.png" /></figure><p>Every enterprise AI strategy deck I have seen in the past years contains the same promise: “We will build a RAG-based knowledge assistant that lets employees query our internal documents in natural language.” The board nods. The budget gets approved. Six months later, the assistant gives wrong answers, misses critical information, and nobody trusts it.</p><p>The post-mortem always blames the model. “We need a better LLM.” “The AI is hallucinating.” “Maybe we should switch to a different vendor.”</p><p>The model is rarely the problem. The data pipeline is.</p><p>After more than 20 years of building enterprise data warehouses, I have watched this pattern repeat with every new technology wave. The tooling changes. The failure mode does not. Organizations invest in the shiny endpoint — the BI tool, the dashboard, the AI chatbot — and underinvest in the unglamorous work that makes it reliable: data ingestion, transformation, quality assurance, and governance. RAG is no exception.</p><h3>What RAG Actually Is</h3><p>The concept is simple. A large language model does not know anything about your company. When you ask it about your internal policies, your contracts, or your compliance rules, it either admits ignorance or invents a plausible-sounding answer. RAG fixes this by searching your actual documents before the model generates its response. The relevant passages get injected into the prompt, and the model answers based on your content rather than its training data.</p><p>The architecture has three layers: a data pipeline that ingests and prepares your documents, a retrieval system that finds the most relevant passages for a given question, and a language model that synthesizes a coherent answer. Everyone focuses on the third layer. The first layer is where projects succeed or fail.</p><h3>The Data Pipeline Decides Everything</h3><p>On Databricks, a RAG data pipeline follows the same medallion architecture you would use for any analytical workload. Raw documents - PDFs, Word files - land in Unity Catalog Volumes. A Spark Declarative Pipeline or Auto Loader ingests them incrementally. The native ai_parse_document function extracts structured text, handling OCR, multi-column layouts, tables, and even chart descriptions. The parsed output lands in Delta tables, governed by Unity Catalog.</p><p>So far, this is just data engineering. And the quality of this engineering determines the quality of every answer the system will ever produce.</p><p>Consider what happens when the parsing is poor. A compliance document with a two-column layout gets merged into a single stream of text, interleaving unrelated paragraphs. A table spanning two pages loses its header row. A chart showing risk exposure gets ignored entirely because the parser did not extract a text description. The downstream model receives corrupted input and produces corrupted outputs. No amount of prompt engineering or model upgrades will fix data that was corrupted at ingestion.</p><p>This is not a new problem. It is the same problem we have always had. In every data warehouse I have built, the most expensive defects originated at the source interface — not at the reporting layer.</p><h3>Chunking Is a Data Modeling Decision</h3><p>After parsing, the text must be split into smaller segments for retrieval. The industry calls this chunking. I call it a granularity decision because that is exactly what it is.</p><p>Every data warehouse professional has faced the question: at what level of detail should we store the data? Too coarse and you lose precision. Too fine and you lose context. The answer always depends on the business requirements and the access patterns. Chunking is the same tradeoff.</p><p>Split a document into chunks that are too large, and the retrieval returns vaguely related passages that dilute the answer. Split it into chunks that are too small, and individual fragments lack the context needed to be useful. Split it at arbitrary boundaries, every 500 characters, regardless of where sentences end, and you destroy the logical structure of the content.</p><p>The right approach is to split at meaningful boundaries: paragraphs, sections, or topic changes. Databricks supports this through libraries like LangChain and custom Python UDFs that respect the document structure produced by ai_parse_document. Techniques like chunk overlap, which carry forward a portion of the previous chunk, prevent information from being lost at the split point. More advanced methods use embedding models to detect where the topic changes and place boundaries accordingly.</p><p>These are data modeling decisions. They require understanding the content, the use case, and the access patterns. They require the same judgment we apply when designing fact tables, choosing partition keys, or defining aggregation levels. An AI engineer who has never built a data pipeline will default to a naive fixed-size split and wonder why retrieval quality is poor.</p><h3>Governance Is Not Optional</h3><p>In a data warehouse, access control is a first-class concern. Not every user should see every table. Not every report should include every data point. Regulatory requirements dictate who can access what, and the consequences of getting it wrong are severe.</p><p>RAG systems have exactly the same requirement, and most implementations ignore it entirely. If your knowledge base contains board-level strategy documents alongside general employee handbooks, every query hits the entire corpus. An intern asking about the vacation policy might receive chunks from a confidential M&amp;A briefing — not because the model is malicious, but because the retrieval had no way to enforce access boundaries.</p><p>On Databricks, Unity Catalog provides the governance layer. The same permission model that controls access to tables and models also applies to the Volumes that store raw documents and the Delta tables that store processed chunks. This is not a feature that can be configured later. It is a design requirement from day one.</p><h3>Metadata Makes or Breaks Retrieval</h3><p>A retrieval system without metadata is like a data warehouse without indexes. It works, technically, but it performs poorly at scale and produces unreliable results.</p><p>Every chunk in your knowledge base should carry metadata: which document it came from, which section, when it was last updated, which department owns it, and what classification level it holds. Without this metadata, the retrieval system relies entirely on semantic similarity — a blunt instrument that cannot distinguish between a current compliance policy and an outdated draft, or between an approved procedure and a rejected proposal.</p><p>Databricks recently introduced what they call an Instructed Retriever, which uses metadata to filter and constrain retrieval before similarity search even begins. Instead of finding the ten most similar chunks and hoping they are relevant, it enforces rules like “only retrieve from compliance documents published after January 2025.” This is predicate pushdown for unstructured data — a concept that will feel immediately familiar to anyone who has optimized partition elimination.</p><p>Metadata extraction, enrichment, and maintenance are pure data engineering work. It is also the work that most RAG tutorials skip entirely.</p><h3>Monitoring Is Where Discipline Shows</h3><p>Most RAG implementations end at deployment. The chatbot goes live, someone writes a message announcing it, and the team moves on to the next project. Nobody monitors whether the retrieval quality degrades as new documents are added. Nobody checks whether updated policies have been re-indexed. Nobody measures whether the answers are actually correct.</p><p>In the data warehouse world, this would be unthinkable. We build reconciliation checks, data quality dashboards, and alerting for pipeline failures. We track data lineage so that when a report looks wrong, we can trace the problem back to its source.</p><p>Databricks provides Lakehouse Monitoring for RAG applications, which can scan outputs for hallucinations, toxic content, and quality degradation. But the monitoring architecture, what to measure, what thresholds to set, and how to handle drift, requires the same operational discipline we have been practicing for decades in data warehousing. The tooling is new. The mindset is not.</p><h3>The Real Skill Gap</h3><p>The AI community has no shortage of talent for model selection, prompt engineering, and fine-tuning. What it lacks — acutely — is talent for the foundational data work that determines whether an AI application delivers business value or becomes an expensive embarrassment.</p><p>Consider the skills a production RAG system requires: reliable data ingestion, incremental loading, and error handling. Document parsing that preserves structure and context. Chunking strategies that balance precision and recall. Metadata extraction and enrichment. Access control and data governance. Pipeline monitoring and data quality assurance. Version management for evolving documents.</p><p>This is a data engineering skill set. It is also, not coincidentally, the skill set of anyone who has spent years building and maintaining enterprise data warehouses.</p><p>The model is a commodity. You can swap it with a single configuration change. The data pipeline is the product. It is what determines whether your RAG system gives a compliance officer a trustworthy answer grounded in current policy — or a confidently wrong answer based on a corrupted chunk from an outdated draft.</p><p>If you have spent your career ensuring that the right data reaches the right people in the right form, you already possess the most critical skill for building AI systems that actually work. The technology has changed. The discipline has not.</p><p><em>Roland Wenzlofsky is the founder of DWHPro, a vendor-neutral data warehousing consultancy with over 20 years of enterprise experience across Teradata, Snowflake, and Databricks. He is a Teradata Certified Master and the author of “Teradata Query Performance Tuning.”</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e1c5cd0cc34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineering Is Not Dying. But Management Is Doing Its Best to Kill It.]]></title>
            <link>https://medium.com/@r.wenzlofsky/data-engineering-is-not-dying-but-management-is-doing-its-best-to-kill-it-a315445fe37d?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/a315445fe37d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Sun, 05 Apr 2026 14:40:42 GMT</pubDate>
            <atom:updated>2026-04-05T15:02:45.018Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How corporate cost-cutting is creating the exact problem AI was supposed to solve.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hG0r9VKTZyUVJqq4oBJfSg.png" /></figure><p>Let me describe a scene playing out in companies across Europe and North America right now.</p><p>A management team sits in a meeting room. The CFO presents a slide. It shows the cost of the onshore data engineering team — experienced people who understand the business, know the systems, and can both write code and design architecture. Next slide: a proposal to replace them with an offshore team at one-third the cost. Per head. On paper. Everyone nods. The decision is made before the coffee gets cold.</p><p>Six months later, the same management team sits in the same room, wondering why their Databricks costs tripled, why nobody understands how the pipelines work, and why the new offshore team keeps asking questions that the people who were fired could have answered in thirty seconds.</p><p>I wish this were exaggerated. It’s not. I see it happening everywhere. And the timing makes it particularly absurd, because these companies are doubling down on an offshoring model at the exact moment AI is about to make that model obsolete.</p><p>Let me explain why.</p><h3>The model that worked — past tense</h3><p>For twenty years, the economics of data engineering were simple. You had expensive work (writing ETL pipelines, SQL transformations, Informatica workflows, PySpark jobs) and you had expensive people in expensive countries doing it. A senior data engineer in Germany or the US costs €/$150,000–200,000 per year. The same skill set in India costs $30,000–60,000. The work was structured enough to offshore: an architect writes specifications, and an offshore team implements them.</p><p>It worked. Not perfectly — time zones, communication overhead, context loss, the never-ending cycle of “that’s not what I meant” in the specification review — but well enough. Companies saved money. Consultancies like TCS, Infosys, and Wipro built empires on this arbitrage. Millions of engineers in Bangalore, Hyderabad, and Pune built careers translating specifications into code.</p><p>Here’s the thing, though. This model worked because <em>human labor</em> was the only way to write code. The offshore engineer was cheaper than the onshore engineer, but both were dramatically more expensive than the next option.</p><h3>Enter AI, stage left</h3><p>An AI coding assistant costs roughly $20–200 per month. Per developer. Not per hour. Per month.</p><p>It generates a working PySpark in seconds. It converts Teradata SQL to Spark SQL while you wait. It builds Airflow DAGs from natural language descriptions. It writes DBT models. It documents your pipeline. It suggests optimizations. It doesn’t need onboarding, doesn’t have visa complications, doesn’t take holidays, and doesn’t resign for a 30% raise at a competitor after eighteen months.</p><p>Is it perfect? No. Does it replace <em>all</em> human judgment? Absolutely not. We’ll get to what it can’t do.</p><p>But here’s the math that should be keeping every outsourcing executive awake at night:</p><p>The work that was offshored — translating specifications into code — is <em>exactly</em> the work AI automates best. Structured. Pattern-based. Well-documented. Clear inputs, clear outputs. This is not a coincidence. This is the same reason it was offshorable in the first place. If a task is structured enough to be put in a specification document and handed to someone who doesn’t know your business, it’s structured enough for AI to handle.</p><p>The offshore model competed on cost against onshore labor. AI competes on cost against <em>all</em> labor. And AI wins.</p><h3>Now watch what management does next</h3><p>So here we are. AI is automating the code layer. The smart move would be to recognize this, retain the people who understand architecture and business context, and let AI absorb the implementation work. Smaller teams, more senior, more capable.</p><p>That’s not what’s happening.</p><p>What’s happening instead — and I genuinely cannot believe I have to write this — is that management teams are <em>laying off their experienced generalists and replacing them with more offshore coders</em>. They’re cutting the architecture layer and doubling down on the code layer. The exact layer AI is about to eat.</p><p>Let me walk you through the decision process, because it’s a masterclass in optimizing for the wrong decade.</p><p>The management team sees economic pressure. They look at headcount. They see a senior data engineer in Europe who costs €170,000. She’s been with the company for ten years. She knows every pipeline, every business rule, every workaround, every reason why that one table has a weird column name from a merger in 2019. She can write code <em>and</em> sit down with the business to explain why the proposed architecture won’t work.</p><p>Management sees: €170,000.</p><p>They hire three offshore engineers at €40,000 each. Total: €120,000. Savings: €50,000. The CFO puts this on a slide labeled “efficiency.”</p><p>Here’s what actually happens.</p><p><strong>Months 1–6:</strong> The offshore team takes over. Things more or less work. The documentation left by the fired generalist is enough to keep the lights on. Management feels vindicated. “See? Same output, lower cost.”</p><p><strong>Months 7–12:</strong> The questions start. The offshore team encounters a pipeline they don’t understand. They can’t find documentation for the business logic. They make assumptions. Some assumptions are wrong. Data quality issues start appearing in reports. The business team notices. The offshore team asks for more specifications. Nobody onshore has the context to write them, because that person was fired.</p><p><strong>Months 12–18:</strong> AI coding tools have matured further. The offshore team’s implementation work — the very work they were hired for — is increasingly automatable. Management starts wondering why they’re paying €120,000 for three people to do work that an AI tool could do for €600 per year. Meanwhile, architectural problems are piling up. Databricks costs are rising. Nobody understands which pipelines can be consolidated. Nobody knows why the cluster auto-scaling is behaving strangely. There’s no one left who understands the system at an architectural level.</p><p><strong>Months 18–24:</strong> Management realizes they need a senior architect. They go to market. The same profile they fired 18 months ago now costs €200,000 because everyone else fired their generalists too, and the good ones are locked into contracts at companies that were smarter. The new hire needs six months to rebuild the institutional context that was walked out the door. Total cost of the “efficiency” decision: more than if they’d simply kept the original person and given them AI tools.</p><p>This is not a hypothetical scenario. I’m watching it happen at multiple companies simultaneously. It’s the corporate equivalent of selling your roof because the sun is shining, then acting surprised when it rains.</p><h3>What management gets wrong about AI and offshoring</h3><p>The root error is treating AI as just another reason to cut headcount, rather than recognizing it as a force that changes <em>which</em> headcount matters.</p><p>Management hears “AI can write code” and translates it to “we need fewer people.” That’s half right. You do need fewer people <em>writing code</em>. But you need <em>more</em> people — or at least the <em>same</em> people — who understand architecture, evaluate AI output, and make design decisions.</p><p>The generalist they fired wasn’t expensive because she wrote code. She was expensive because she understood the system well enough to make decisions that saved the company millions in computing costs, data quality incidents, and architectural dead ends. The code was 30% of her value. The architecture was 70%.</p><p>By firing her, management kept the 30% (offshore coders) and discarded the 70% (architectural judgment). Then AI automated the 30%. Now they have neither.</p><p>There’s also a deeper misunderstanding about what offshoring and AI optimize for. Offshoring optimizes for <em>labor cost</em> — same work, cheaper humans. AI optimizes for <em>labor elimination</em> — same work, no humans. These are not the same strategy. You cannot combine them by saying “let’s offshore AND use AI.” If AI does the work, there’s nothing left to offshore.</p><p>Companies that offshore their implementation teams and then adopt AI tools end up paying for two solutions to the same problem. The offshore team writes code. The AI tool also writes code. Someone onshore has to review both. The “efficiency” is now three layers of cost, where one would do.</p><h3>The split that actually matters</h3><p>Let’s set aside management decisions for a moment and look at what’s structurally happening to the profession.</p><p>Data engineering is splitting into two layers. This isn’t a forecast. It’s already happening.</p><p><strong>The code layer:</strong> Writing transformations. Building pipelines. Debugging SQL. Wiring up orchestration. Translating requirements into working code. This layer was commoditized first by offshoring, now by AI. Its economic value is approaching zero — not because the work doesn’t matter, but because the cost of performing it is collapsing.</p><p><strong>The architecture layer:</strong> Deciding how data flows. Choosing platforms. Designing for cost, performance, and reliability. Understanding trade-offs between materialization and virtualization, between streaming and batch, between lakehouse and warehouse. Knowing <em>why</em> a system was designed a certain way, not just <em>how</em> it was built. This layer is becoming more valuable because AI can’t do it, offshoring can’t commoditize it, and every organization desperately needs it.</p><p>For 20 years, these layers were bundled into a single job description. The same person who designed the architecture also wrote the code. That made sense when code was expensive to produce. Now that code is cheap (via AI), the bundle breaks. And the two layers have very different economic futures.</p><h3>What AI can actually do — honestly</h3><p>Let’s be precise, because the hype isn’t helping anyone.</p><p><strong>Code generation works.</strong> AI assistants generate working PySpark, SQL, dbt models, and Airflow DAGs from natural language descriptions. Not always perfect. Often needs adjustment. But correct often enough that the human role shifts from writing to reviewing.</p><p><strong>Data profiling is automated.</strong> AI examines a dataset and describes its structure, types, null patterns, and statistical distributions in seconds. Work that used to take an afternoon.</p><p><strong>Documentation is generated.</strong> AI reads a pipeline and produces documentation. It also reviews existing documentation and flags outdated sections.</p><p><strong>Platform translation is emerging.</strong> AI converts Teradata SQL to PySpark, translates Informatica mappings into Airflow DAGs, and migrates stored procedures between SQL dialects. Not perfectly. But well enough that a migration project that used to need 50 offshore engineers for 6 months might now need 10 engineers for 3 months. The outsourcing industry should be terrified by this.</p><p><strong>Testing and monitoring are improving.</strong> AI generates data quality checks, suggests edge cases, and spots anomalies in pipeline behavior.</p><p>All of this is real. All of this is in production today. None of it is hype.</p><h3>What AI cannot do — also honestly?</h3><p><strong>AI cannot decide whether you need Spark or Snowflake.</strong> It can list the differences. The actual decision depends on your workload patterns, team skills, vendor relationships, budget, compliance requirements, and strategic direction. These are judgment calls that require organizational context.</p><p><strong>AI cannot design your data architecture.</strong> Batch or streaming? Materialize or virtualize? Lakehouse or warehouse? These are trade-offs that depend on cost, latency, team capability, and business requirements. AI proposes options. Humans weigh them.</p><p><strong>AI cannot understand your business context.</strong> It can calculate revenue per region. It can’t tell you the region definitions changed because of a merger, the finance team counts deferred revenue differently, and the CEO only cares about year-over-year using the old methodology.</p><p><strong>AI cannot evaluate its own output.</strong> This is the critical one. AI generates a syntactically correct Spark job that triggers a massive, unnecessary shuffle. The code compiles. The tests pass. The architecture is terrible. Spotting this requires understanding how distributed systems actually work.</p><p>Which brings us back to the management problem. The person who spots the architectural flaw in AI-generated code is the generalist they just fired.</p><h3>The cost equation nobody wants to hear</h3><p>Let’s compare three models with real numbers.</p><p><strong>The old onshore model:</strong> Six senior engineers × €170,000 = €1,020,000/year. Expensive. But they know the systems, the business, and the architecture.</p><p><strong>The offshoring model (what management is choosing now):</strong> Two onshore leads × €170,000 + eight offshore engineers × €40,000 = €660,000/year. Looks cheaper. But add communication overhead, specification writing, review cycles, quality issues, and knowledge loss. Real cost: probably €800,000+ when you include rework and incidents. And the offshore team is doing work that AI will automate within two years.</p><p><strong>The AI-augmented model (what smart companies are choosing):</strong> Three senior engineers × €180,000 + AI tools × €2,000/year = €542,000/year. Fewer people, more senior, all architecturally capable, all using AI for the code layer. No specification documents, no handoff delays, no time zone management. Higher output than either of the previous models.</p><p>The AI-augmented model is cheaper than offshoring <em>and</em> produces better results. Let that sink in. The “cost-saving” decision to offshore is actually more expensive than the alternative.</p><p>But it gets worse. The offshoring model requires ongoing management overhead (coordination, specification, review). The AI-augmented model doesn’t. The offshoring model loses architectural knowledge when onshore people leave. The AI-augmented model retains it because the architects are doing the work directly. The offshoring model will face another cost restructuring when AI fully automates the offshore team’s work. The AI-augmented model is already adapted.</p><p>Management teams choosing offshoring right now are selecting the most expensive option and calling it efficiency.</p><h3>What this means for different people</h3><h3>If you’re an engineer in India or Eastern Europe</h3><p>The implementation role is shrinking. This is not your fault, and it’s not about your talent — it’s about economics. The strategic response is to move toward architectural understanding, domain expertise, and AI-adjacent skills. An engineer in Bangalore who deeply understands Spark’s execution model and can design data architectures is just as valuable as one in Berlin. The value has shifted from volume (many people writing code) to depth (fewer people making architectural decisions). The depth is geography-independent.</p><h3>If you’re an engineer in the US or Western Europe</h3><p>You’re not safe just because you’re onshore. If your work is primarily implementation, you’re as replaceable as anyone. Your protection isn’t your passport — it’s your ability to work at the intersection of technology, architecture, and business context. Invest in understanding <em>how platforms work</em>, not just <em>how to write code for them</em>.</p><h3>If you’re in management</h3><p>Stop. Think about what you’re actually cutting.</p><p>When you lay off a generalist who understands both code and architecture, you’re not removing a cost line. You’re removing a capability that took years to develop and cannot be replaced by hiring three people who lack it. AI will handle the code. You need the architecture. If you fire the architect and keep the coders, you’ve chosen to keep the part that’s being automated and discard the part that isn’t.</p><p>Retain your generalists. Give them AI tools. Let the AI absorb the implementation work. Reduce your offshore team gradually as AI takes over that workload. Your total cost will be lower, your quality will be higher, and your architectural knowledge stays in-house. <strong>Or ignore this, offshore everything, and budget for the emergency architect hiring in 18 months. Your choice.</strong></p><h3>If you’re in the outsourcing industry</h3><p>The model of deploying large teams of implementation engineers at a cost advantage is structurally challenged. You cannot compete on cost with software that works for €50/month. The firms that survive will pivot from selling implementation labor to selling architectural expertise and AI integration services. This is a higher-value but lower-volume business. Fewer people, different skills, different margins. The industry will be smaller but more specialized.</p><h3>Where does this all land</h3><p>The data engineering profession isn’t dying. It’s being compressed. Fewer people, doing higher-value work, augmented by AI that handles the mechanical parts.</p><p>The tragedy is that, in many organizations, management is accelerating the wrong transition. Instead of embracing the compression — smaller teams, more architectural, AI-augmented — they’re clinging to the last decade’s playbook. Cut onshore costs. Offshore, the implementation. Celebrate the savings. Ignore the fact that the implementation layer they’ve offshored is the same layer AI is about to eliminate.</p><p>In two years, these companies will be hiring expensive architects to fix problems that didn’t need to exist. The fired generalists will be happily employed elsewhere. The offshore teams will struggle against AI tools that can do their work in seconds. And management will be in the same meeting room, wondering what went wrong.</p><p>I could tell them. But they probably won’t ask — because asking costs €170,000 a year and they found someone cheaper.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a315445fe37d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Most Dangerous System in Your Bank Runs on Excel]]></title>
            <link>https://medium.com/@r.wenzlofsky/the-most-dangerous-system-in-your-bank-runs-on-excel-bb0b084973cc?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb0b084973cc</guid>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Sun, 22 Mar 2026 19:36:27 GMT</pubDate>
            <atom:updated>2026-03-22T19:44:13.053Z</atom:updated>
            <content:encoded><![CDATA[<p><em>“The spreadsheet is the most dangerous piece of software ever created.”</em> — Daniel Lemire</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmonCDt8jIj7exvIctLRsw.png" /><figcaption>Where is Gerhard?</figcaption></figure><p>There is a system in your bank that nobody calls a system. It has no documentation. It has no change management process. It has no backup strategy, no disaster recovery plan, and no access control beyond the fact that one person knows the password to the folder on the shared drive.</p><p>It processes regulatory data. It generates board reports. It calculates risk figures that are submitted to the financial supervisory authority.</p><p>It is an Excel file with macros. And the person who built it is either about to retire or has already left.</p><p>If you work in data warehousing for European banks — and I have done so for over 20 years — you know exactly what I am describing. You have seen it in every institution. Not as an exception. As a rule.</p><h3>How It Happens</h3><p>Nobody sets out to build critical infrastructure in Excel. It starts innocently. A business analyst needs a report that IT cannot deliver for another six months. The analyst builds it in Excel. It works. Management is satisfied. The report becomes a recurring deliverable.</p><p>Over the next two years, the spreadsheet acquires macros. It pulls data from three different sources via copy-paste. Someone adds a VLOOKUP that references another workbook on a different share. A pivot table is added for the monthly board presentation. A colleague adds conditional formatting so the risk figures turn red when they exceed a threshold.</p><p>By year three, it is no longer a spreadsheet. It is an application. It has business logic, data integration, transformation rules, and output formatting. It has everything a proper ETL pipeline has — except version control, testing, documentation, audit trails, and the ability to function when its creator is unavailable.</p><p>In the regulated world, this is not merely inconvenient. It is a compliance risk.</p><h3>The Audit Problem</h3><p>Regulators do not care whether a number was produced by a data warehouse or a spreadsheet. They care whether the process that produced it is documented, reproducible, and auditable.</p><p>A properly designed data pipeline satisfies this requirement. The process is defined in code. The code is version-controlled. The execution is logged. The output is traceable.</p><p>An Excel macro satisfies none of these requirements. The logic is embedded in cells and VBA modules that are opaque to anyone who did not build them. There is no execution log. There is no version history beyond “Final_v3_FINAL_edit_NEW.xlsx” saved on a network drive. When the auditor asks, “How was this number calculated?” the answer is frequently, “We need to ask Gerhard, but he is on holiday.”</p><p>I have personally witnessed audit findings in European banks where the root cause was an undocumented Excel process that produced numbers for regulatory reporting. The remediation was not a technical fix. It was a project.</p><h3>Why IT Cannot Solve It Alone</h3><p>The standard response from IT departments is: “Put it in the data warehouse.” This is correct in principle. But it ignores reality.</p><p>The business analyst built the Excel solution because IT could not deliver on time. If IT now says, “Give us 12 months, and we will rebuild it properly,” the business hears the same message that caused the problem in the first place. The analyst will nod, agree, and continue using Excel.</p><p>The gap is not technical. It is organisational. Business teams need autonomy over their data processes. IT needs governance and control. These two requirements are in tension, and Excel has been the unofficial resolution for decades — a resolution that works until it does not.</p><p>What is needed is a middle ground: tools that give business teams the autonomy to manage their own data workflows and reporting, while enforcing the governance, auditability, and structure that regulators and IT require. This is not a new data warehouse. It is the layer between the data warehouse and the business user — the layer where Excel currently sits, unsupervised.</p><h3>The Cost of Doing Nothing</h3><p>Banks that tolerate Excel-based critical processes are accumulating risk in three dimensions:</p><p><strong>Operational risk.</strong> A key person leaves, becomes ill, or simply makes a copy-paste error. The process breaks. The board report is delayed or incorrect. This happens more often than any CIO will publicly admit.</p><p><strong>Regulatory risk.</strong> An auditor identifies an undocumented, non-reproducible process in a compliance-relevant data chain. The finding triggers a remediation project that costs more than the proper solution would have cost in the first place.</p><p><strong>Efficiency loss.</strong> Senior analysts — people who should be analysing data — spend hours each month manually assembling, formatting, and distributing reports. This is not value creation. It is data logistics performed by the department’s most expensive people.</p><h3>A Practical Alternative</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*_XefcR-JkO8wghzIeNshzQ.png" /></figure><p>I have recently come across a company that addresses exactly this problem. <a href="https://www.etixpert.com/">ETIXPERT</a> is a specialist for data-driven process automation in regulated industries — banking, insurance, and industrial organisations — with over 25 years of experience.</p><p>They offer two platforms that directly replace the Excel-based workflows I have described:</p><p><strong>Zagreus</strong> is a server-side workflow and reporting engine. It automates the generation and distribution of reports (PDF, Excel) in a fully audit-proof manner. It eliminates macros, manual distribution lists, and the kind of shadow IT that accumulates when business teams solve problems on their own. The entire execution is server-side — no client-side macros, no local dependencies. It integrates with SharePoint, Jira, Power BI, and other enterprise tools.</p><p><strong>niota</strong> is a low-code data platform for business teams. It allows departments to manage structured data and workflows with built-in governance — role-based access, multi-tenancy, and full traceability. It gives the business the autonomy they need without sacrificing the control IT requires. It is designed specifically for regulated environments and dynamic processes where data quality and auditability are non-negotiable.</p><p>I want to be transparent: I mention them here because their solutions directly address a problem I have observed in nearly every banking and insurance project I have worked on. The recommendation is based on relevance to the topic, not a commercial obligation.</p><h3>The Question to Ask</h3><p>If you work in a European bank — or any regulated institution — ask yourself this: how many critical processes in your organisation currently depend on an Excel file that one person maintains?</p><p>If the answer makes you uncomfortable, that discomfort is the correct response.</p><p>The data warehouse handles the heavy lifting. But the last mile — where data becomes reports, decisions, and regulatory submissions — is where Excel still reigns. And it is where the risk sits.</p><p>Fixing the data warehouse is important. Fixing the last mile is urgent.</p><p><em>Roland Wenzlofsky is the founder of DWHPro, a Vienna-based data warehouse engineering consultancy and official partner of PwC Austria. He is a Teradata Certified Master with over 20 years of experience in enterprise data warehousing across the European banking, insurance, and telecom sectors. He is the author of “Teradata Query Performance Tuning” and writes about the reality of data engineering at </em><a href="https://www.dwhpro.com/"><em>dwhpro.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb0b084973cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Agents Are Quietly Rewriting the Cloud Database Cost Model]]></title>
            <link>https://medium.com/@r.wenzlofsky/ai-agents-are-quietly-rewriting-the-cloud-database-cost-model-88693017a5b0?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/88693017a5b0</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 17:19:33 GMT</pubDate>
            <atom:updated>2026-03-16T17:19:33.502Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GWevb0HrA8swaIgj9MzvBg.png" /></figure><p>I have spent over twenty years building and optimising enterprise data warehouses, and in that time, I have watched the industry cycle through several dominant narratives. The current one is elasticity. Cloud databases are sold on a simple promise: compute can suspend when idle, scale back up when needed, and spare you the cost of owning fixed capacity around the clock. Warehouses auto-suspend and auto-resume. Some platforms offer both on-demand and capacity-based models. Others allow capacity to be paused to save costs.</p><p>That pricing story made sense when workloads were mostly human-shaped: analysts active during the day, ETL windows at night, quieter weekends, and plenty of idle time in between. In that world, idle-elimination features could produce meaningful savings.</p><p>But that narrative assumes the system is idle often enough for elasticity to matter. And AI agents are pushing data platforms in exactly the opposite direction.</p><h3>The old assumption: bursty workloads</h3><p>Most cloud migration business cases quietly rely on one economic assumption: workloads are bursty. Some jobs spike hard, but they do not run continuously. This is exactly where pay-per-use models shine.</p><p>What is rarely mentioned in those business cases is that even cloud vendors position capacity commitments as beneficial for steady-state workloads, which implies the standard elastic model is not always the cheapest choice once utilisation becomes stable.</p><p>That nuance matters. The phrase “you only pay when you use it” describes one end of the pricing spectrum, not the entire cloud pricing landscape. Major platforms already support reservations, commitments, and capacity-oriented purchasing models alongside pure usage pricing.</p><p>In my experience, the elasticity argument is strongest in pre-production environments, development clusters, and seasonal workloads. It is weakest in mature production environments where the system is busy most of the time anyway. This is precisely the scenario that AI agents are creating.</p><h3>What changes when agents arrive</h3><p>Once autonomous agents begin running continuously against a data platform, idle windows shrink.</p><p>That can mean agents monitoring data quality, checking anomalies, generating SQL through semantic layers, querying structured and unstructured sources, refreshing feature stores, running background investigations, or servicing natural-language requests from business users across time zones. Cloud-native AI agents are already designed to plan tasks, use tools, and analyse both structured and unstructured data sources — and they do so without waiting for someone to open a dashboard.</p><p>This does not mean every deployment becomes saturated around the clock overnight. But it does mean the probability of persistent background activity rises sharply. A warehouse that used to sleep between user sessions now wakes up more often, suspends less often, and may spend much more time at sustained utilisation. And warehouses continue consuming credits while running, regardless of whether a human initiated the query.</p><p>At that point, the economic advantage of pure pay-per-use pricing becomes weaker. If utilisation is consistently high and predictable, reservations, commitments, or owned capacity can become more attractive. Cloud platforms already recommend capacity commitments for steady-state workloads, and some now offer capacity reservations for 1-year terms at significant discounts compared to pay-as-you-go rates.</p><p>This is not a universal rule. Sometimes, cloud still wins because of hardware efficiency, operations, concurrency, or contract terms. But the direction is clear: the more continuously the system runs, the less value you get from “scale to zero.”</p><h3>The second problem: cost opacity</h3><p>Agents do not just reduce idle time. They also change cost visibility.</p><p>With human-driven workloads, organisations usually have a rough mental model of spend: how many analysts they have, when the dashboards run, when ETL starts, and what the month-end surge looks like. Agents introduce a more opaque pattern. They decide what steps to take, which tools to call, how often to retry, and how broadly to explore — without asking for permission first.</p><p>The fact that cloud vendors are now building dedicated governance layers specifically for AI agents and LLM endpoints is itself an acknowledgment that these workloads need specialised control. If agents behaved like ordinary reporting jobs, there would be no need for separate monitoring.</p><p>To be fair, runaway cost is not unique to agents. Poorly governed BI, bad ETL design, and uncontrolled ad hoc workloads have been generating ugly cloud bills for years. I have seen this in almost every large environment I have worked in. But agents amplify the problem because they operate more continuously, make more decisions without human review, and can execute at a cadence that exceeds normal human query patterns. That is why vendors are now adding AI-specific monitoring and controls rather than treating these workloads like ordinary compute.</p><h3>The evidence is already showing up in budgets</h3><p>The spend side is no longer hypothetical. Survey data from mid-to-large software firms shows average monthly AI budgets rising roughly 36 percent year over year, reaching approximately $85,000 per month, while only about half of organisations say they can confidently evaluate their AI return on investment.</p><p>Those numbers are not specific to database agents alone. They cover broader AI spending. But they matter because they show the same pattern: AI adoption is accelerating faster than cost clarity. The industry is getting better at deploying AI than at predicting its production costs.</p><p>Anyone who has managed a large data warehouse knows this feeling. It is the same dynamic that played out when self-service BI tools arrived, and suddenly every analyst could launch expensive queries without going through a DBA. The difference with agents is that they do not go home at 18:00.</p><h3>Why does this strengthen the on-prem argument</h3><p>In an agent-heavy environment, fixed-capacity systems start to look different.</p><p>On-premise platforms do not magically become cheaper in every case. They still carry software, support, power, cooling, and administrative costs. But they do offer something; cloud consumption models often struggle to provide boundedness under continuous load. Once capacity is bought and installed, the short-run infrastructure cost of an additional query is often low until the system reaches its throughput limits.</p><p>The question then becomes less “what will this query cost?” and more “can the platform absorb this workload mix without hurting service levels?” That is a shift that favours platforms with mature workload governance.</p><p>I know this territory well. Workload management frameworks on mature on-premise platforms support workload throttles, system throttles, concurrency controls, and delay or reject behaviour when a system gets busy. Throttles reduce resource contention by delaying or rejecting work once thresholds are exceeded. These capabilities have been refined over decades and are well documented. In an agent-saturated world, they do not become less relevant — they become more relevant. When additional work is cheap to launch but expensive to let run unchecked, governance becomes part of the value proposition.</p><h3>Cloud vendors are already adapting</h3><p>Cloud vendors can see this shift, too, and they are responding.</p><p>Major platforms now support resource budgets for AI agents, including automated actions when spending crosses defined limits. Dedicated AI gateways are being positioned as the control plane for governing and monitoring LLM endpoints and coding agents. Some platforms document both token-based billing for data agent usage and capacity-consumption reporting for autonomous agent workloads.</p><p>These are meaningful additions. But they also reveal something important: once you need specialised budgets, monitoring, rate limits, capacity controls, and agent-specific billing views, you are already moving away from the clean simplicity of the original elasticity narrative. The cloud model is not disappearing. But it is becoming more governed, more contractual, and more capacity-like, which is precisely what it was supposed to replace.</p><h3>The real reframing</h3><p>This is the part most migration business cases still avoid.</p><p>If AI agents materially increase the frequency of workloads and reduce idle time, the central economic question changes. The comparison is no longer just “elastic cloud versus fixed on-prem.” It becomes something more nuanced: cost predictability, throughput governance, and bounded spending on one side — operational flexibility and outsourced infrastructure on the other.</p><p>That is a very different discussion from the one most organisations had when they signed their cloud migration contracts.</p><p>It is not a death sentence for cloud databases. It is a warning that the old sales story was built around a workload shape that AI agents may steadily erode. The more your platform behaves like an always-on system, the less persuasive “pay only for what you use” becomes — because you are using it all the time.</p><p><em>Roland Wenzlofsky is the founder of </em><a href="https://www.dwhpro.com/"><em>DWHPro</em></a><em>, a Vienna-based enterprise data warehousing consultancy. He is a Teradata Certified Master with over twenty years of experience across European banking, insurance, and telecom, and the author of “Teradata Query Performance Tuning.” DWHPro provides vendor-neutral advice across Teradata, Snowflake, and Databricks.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88693017a5b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Next Data Warehouse Migration Will Fail for the Same Reason the Last One Did]]></title>
            <link>https://medium.com/@r.wenzlofsky/your-next-data-warehouse-migration-will-fail-for-the-same-reason-the-last-one-did-6fc138b05f06?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fc138b05f06</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[cloud-migration]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[teradata]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Sat, 14 Mar 2026 23:50:08 GMT</pubDate>
            <atom:updated>2026-03-14T23:50:08.263Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ygr4LiEiJRJ-7Ol2xhLCfQ.png" /><figcaption>Good job guys , now the last mile…</figcaption></figure><p>There is a question that is almost never asked in the data platform debate, and the reason it is not asked is that nobody benefits from the answer.</p><p>The question is this: if every modern platform can load, transform, and make clean data available to users, why does it matter which one we use?</p><p>The user opening a dashboard does not care whether the data sits in Teradata block storage or in Parquet files on Amazon S3. The user does not care whether the slowly changing dimension logic was implemented in BTEQ or in PySpark. The user does not care how file compaction works, how deltas are detected, or whether the transaction log is a permanent journal or a JSON file in the _delta_log directory.</p><p>The user cares about one thing: correct numbers, delivered on time, every morning.</p><p>Every platform on the market — Teradata, Databricks, Snowflake, BigQuery, Microsoft Fabric — can deliver this. The fundamental purpose of a data warehouse has not changed since the concept was formalised in the early 1990s: ingest raw data, apply business rules, produce clean tables for analytical consumption. The patterns have evolved. Batch ETL gave way to ELT. Real-time streaming entered the picture. Schema-on-read and data mesh became topics of discussion at conferences. But the job the business is paying for remains identical.</p><p>And yet, companies are spending millions to replace one platform with another. The consultancies are busy. The vendors are delighted. The conference circuit is thriving. But nobody seems to pause and ask: Are we solving the right problem?</p><p>In my experience — and I write this after more than 20 years of building and tuning data warehouses, most of that time on Teradata, with increasing work on Snowflake and now Databricks — the answer, in the majority of cases, is no.</p><h3>The Problem That Survives Every Migration</h3><p>Here is the observation that no vendor will put on a slide, no consultancy will lead with in a proposal, and no conference keynote will dedicate more than thirty seconds to:</p><p><strong>The biggest problem in most data warehouse environments is not the platform. It is data quality.</strong></p><p>It has inconsistent business rules. It is unclear who owns the data. It is governance that exists in a PowerPoint deck but not in practice. It is 2000 source-to-target mappings that have accumulated over 15 years, each with slightly different interpretations of what constitutes an “active customer.” It is the transformation logic that nobody fully understands because the person who wrote it left the company in 2017.</p><p>I have seen companies migrate from Teradata to Snowflake and discover, during reconciliation testing, that the numbers on the new platform do not match those on the old platform. Not because the migration was flawed. Because nobody had ever checked whether the old numbers were correct in the first place. They were not. They had not been correct for years. Nobody had noticed because nobody had ever validated from scratch.</p><p>This is not an isolated case. In my experience, it is the norm.</p><p>A platform migration does not fix data quality. It does not reconcile business rules. It does not create ownership where none existed. What a platform migration does — and this is its one genuine contribution to data quality — is make these problems visible. The reconciliation effort forces teams to look at the transformation logic they have not examined in a decade.</p><p>That is valuable. But it is not a reason to migrate. It is a reason to audit your data architecture, regardless of which platform it runs on.</p><h3>Governance Is Not a Checkbox</h3><p>When I speak of governance, I do not mean the compliance section in a project charter. I mean the ability to answer, at any moment, questions such as:</p><p>Who owns the definition of “revenue” in this organisation — and is that definition applied consistently across every pipeline, every report, and every data product that references it?</p><p>When a source system changes a field format at 2 AM on a Saturday, who is responsible for detecting the change, assessing its impact, and propagating the correction through every downstream dependency?</p><p>Can the number in the board report be traced, step by step, back through every transformation to the source record — and can someone explain why it is what it is?</p><p><strong>If these questions cannot be answered on the current platform, they will not be answerable on the new one either</strong>. The platform does not create governance. People create governance. The platform is, at best, a tool that supports them in doing so. At worst, it is a distraction from the fact that nobody is doing the work.</p><h3>Architecture First. Tools Second.</h3><p>The order matters. Before any platform evaluation begins, the data architecture must be assessed. Not the technology architecture — the data architecture. The business rules, data flows, quality controls, and lineage.</p><p>This means auditing the business rules. Are they documented? Are they consistent across pipelines? Or do three different ETL jobs contain three different definitions of the same metric, each producing a slightly different number?</p><p>This means documenting the data lineage. Can any report number be traced back to its source? If this cannot be done on Teradata, it will not be possible on Databricks either — unless it is deliberately designed into the migration, which adds months of work that most migration timelines do not account for.</p><p>This means validating the numbers. Running the same business question against the data warehouse and against a manual extract from the source system. If the results do not match, the organisation has a data quality problem that no platform change will resolve.</p><p>These problems are platform-independent. They must be addressed first. If the architecture is sound, any modern platform will serve the organisation well. If it is not, no platform will compensate for it.</p><h3>When the Platform Genuinely Matters</h3><p><strong>None of the above should be read as an argument that all platforms are interchangeable. They are not. There are legitimate reasons to change platforms. But these reasons are fewer than the industry would have us believe, and they deserve honest examination.</strong></p><h3>The Talent Pool</h3><p>This is the most consequential factor and the one discussed least honestly.</p><p>If an organisation employs 15 experienced Teradata developers and zero Spark engineers, migrating to Databricks does not merely change the technology. It invalidates the skills of the entire data team. The pipeline still performs the same function. But nobody knows how to diagnose it when it fails at three in the morning.</p><p>The reverse is equally true. The ratio of available Teradata talent to Spark and Python talent has shifted dramatically over the past decade. Universities teach Python and Spark. Graduates have never encountered a Primary Index. Teradata still maintains university partnerships and has recently transitioned its traditional certification programme to a free accreditation model — but the pipeline of incoming Teradata talent continues to thin with each graduating class.</p><p><strong>This is not a commentary on the quality of the technology.</strong> It is a demographic reality that forces technology decisions regardless of whether the business case supports them.</p><p>In my experience, this single factor drives more platform migrations than any technical requirement.</p><h3>The Ecosystem Beyond ETL</h3><p>The core data pipeline — ingest, transform, serve — is identical across platforms. But modern analytical work extends beyond the pipeline.</p><p>Data science, machine learning, real-time streaming, and the processing of semi-structured and unstructured data are workloads that a traditional data warehouse was not originally designed for. If an organisation requires nothing more than clean data for BI reports and regulatory submissions, Teradata delivers this superbly. It has done so for decades. The SQL optimiser is mature, the workload management is battle-tested, and the platform’s reliability in regulated European environments is difficult to match.</p><p>But the moment the business requires model training on warehouse data, the situation changes. Teradata Vantage offers in-database analytics — the SCRIPT table operator for running Python and R in the database, and QueryGrid for federating queries to external systems. These capabilities exist, and they function. But in practice, they are limited in scope and operationally cumbersome compared to a platform like Databricks, where data science is a first-class citizen: notebooks, MLflow, GPU clusters, and the training data coexist in a single environment.</p><p>Whether this matters depends entirely on whether the organisation genuinely performs data science, or merely discusses it in board presentations. <strong>In my observation, the number of companies that truly need a unified analytics and machine learning platform is considerably smaller than the number of companies purchasing one.</strong> But for those who do, the productivity difference is substantial.</p><h3>The Real Cost of Migration</h3><p>Any engineer who has been through a large-scale platform migration understands that the licensing fee is the smallest line item. The real cost is the 12 to 18 months of parallel operation, during which the organisation pays for and maintains two platforms simultaneously. It is the thousands of hours of regression testing, comparing output row by row, table by table. It is the business disruption when reports are temporarily unavailable or inconsistent during the transition period.</p><p>And it is the human cost. Senior engineers who have spent 15 years mastering a platform are informed that their expertise is now obsolete. Some will retrain. Some will not. Some will leave during the migration, taking with them institutional knowledge that was never documented and cannot be replaced. I have observed this more than once. The impact on the project is severe and is almost never accounted for in the migration business case.</p><p>Anyone who claims a major platform migration is a six-month project is either selling something (snake-oil?) or has never completed one.</p><h3>Vendor Lock-In — In Both Directions</h3><p>Every platform creates lock-in. Teradata creates it through proprietary SQL extensions and design decisions embedded in thousands of table definitions. Databricks creates it through Spark APIs, notebook ecosystems, and Unity Catalog. Snowflake creates it through its proprietary storage architecture — it is not possible to point an external compute engine at Snowflake’s micro-partitions without first unloading the data — and through the growing ecosystem of Snowflake-native services such as Snowpark, Streamlit, and Cortex that bind workflows to the platform.</p><p>The industry narrative suggests that open file formats such as Parquet and Delta Lake make the Lakehouse architecture easier to leave. In theory, this is correct. <strong>In practice, the cost of rewriting 5,000 notebooks is not materially different from rewriting 5,000 BTEQ scripts</strong>. Lock-in is not principally about file formats. It is about the intellectual property embedded in transformation logic, the operational procedures a team has built over the years, and the institutional knowledge that exists only in engineers&#39; minds.</p><p>But the argument must be made in the other direction as well: remaining on a legacy platform is its own form of lock-in. When the vendor’s roadmap diverges from the organisation’s needs, when support contracts become extortionate, when every new capability requires an expensive upgrade cycle, when integration with modern tools grows increasingly painful — staying can be as costly as leaving. The question is not “migrate or stay.” The question is: which form of lock-in would serve the organisation better over the next five years?</p><h3>The Honest Assessment</h3><p>I run a consultancy that helps organisations with data platform architecture and migrations. That is precisely why I can say, with some authority, when a migration is not needed.</p><p>The organisations that genuinely benefit from a platform change are those where the current platform cannot support real, funded, and urgent new business requirements — not aspirational ones. Where the talent situation has deteriorated to the point where it constitutes a staffing risk — not a hypothetical concern, but an actual inability to hire or retain. Where the commercial terms have become genuinely untenable. Where the vendor’s roadmap has diverged so far from the organisation’s direction that every project becomes a fight against the platform rather than a use of it.</p><p><em>Everything else — and I include a considerable portion of the migration projects currently underway across European and US enterprises — is vendor-driven anxiety, analyst-driven hype, and consultancies generating revenue from complexity.</em></p><p>I say this as someone who runs a consultancy. The best advice I have given to clients, on more than one occasion, is: do not migrate. Fix your architecture instead.</p><h3>The One Thing That Matters</h3><p>The data pipeline does not care what engine runs it. Teradata, Databricks, Snowflake — they are all capable tools in the hands of competent engineers.</p><p>The difference between a successful data warehouse and a failing one has never been the platform. It is the quality of the data. The clarity of the business rules. The discipline of governance. The competence of the people.</p><p>Get those right, and the platform becomes a detail. Get those wrong, and no migration on earth will save you.</p><p>I would be interested to hear from those who have been through a migration: did it solve the problems you expected it to solve? Or did the same problems simply reappear on the new platform?</p><p><em>Roland Wenzlofsky is the founder of DWHPro, a Vienna-based data warehouse engineering consultancy. He is a Teradata Certified Master with over 20 years of experience in enterprise data warehousing across the European banking, insurance, and telecom sectors. He is the author of “Teradata Query Performance Tuning” and writes about the reality of data engineering at www.dwhpro.com.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fc138b05f06" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Comparing Teradata, Snowflake, and Databricks. Start Asking These Questions Instead.]]></title>
            <link>https://medium.com/@r.wenzlofsky/stop-comparing-teradata-snowflake-and-databricks-start-asking-these-questions-instead-f5cd917d1cd6?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5cd917d1cd6</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[teradata]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Thu, 12 Mar 2026 19:37:56 GMT</pubDate>
            <atom:updated>2026-03-12T19:37:56.930Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bc9Q4DDK8gHhyFgM0phG-A.png" /></figure><p>A vendor-neutral look at the architectural trade-offs that actually matter</p><p>I have spent over 20 years building and optimising enterprise data warehouses, mostly on Teradata, and increasingly on Snowflake and Databricks. What I have observed repeatedly is that platform selection decisions are driven by slide decks, analyst quadrants, and licensing pressure — not by the architectural questions that determine whether the platform will actually work for a given organisation.</p><p>This article is not a feature comparison. It is a list of questions I would ask if I were the person who has to sign the purchase order — and live with the consequences for the next five to ten years.</p><h3>The storage-compute separation: do you actually benefit?</h3><p>Both Snowflake and Databricks are built on a fundamental architectural principle that Teradata does not share: the separation of storage and compute. Data lives as files on cloud object storage (S3, ADLS, GCS), and compute resources spin up independently, read what they need, and shut down. You pay for each independently, and each scales independently.</p><p>This sounds compelling in every vendor presentation. The question I would ask first is brutally simple: <strong>what does my workload profile actually look like?</strong></p><p>If the organisation runs a steady, predictable analytical workload eighteen hours a day — which is typical in banking, insurance, and telecom — the storage-compute separation buys less than the slide deck suggests. You are paying for compute that is already running. Elasticity does have value for intra-day peaks — scaling from four nodes to sixteen during the month-end close, then back to four — but even that value depends on how pronounced and frequent those peaks actually are. Most enterprises I have worked with dramatically overestimate their peak-to-trough ratio.</p><p>The follow-up question is equally important: <strong>what is my actual data gravity?</strong> If 80% of queries hit the same 20% of tables — which is common in a properly designed data warehouse — then the separated storage model forces repeated movement of that hot data across the network into a local cache. In Teradata, data resides permanently on the AMP — there is no network hop between the storage and compute tiers, though redistribution between AMPs during query execution incurs a separate cost that exists in all distributed systems. The question is not whether caching solves the storage-compute gap, but whether the overhead of constantly re-warming caches is worth the flexibility gained.</p><h3>The cold cache problem: a cost nobody puts on the slide</h3><p>Both Snowflake and Databricks rely on local SSD caching to bridge the latency gap between their compute nodes and the remote object storage where data permanently resides. The mechanism is conceptually similar: when a query fetches data from remote storage, that data is cached locally on NVMe SSDs so subsequent queries avoid the network round-trip.</p><p>There is a critical difference between the two implementations. Snowflake caches data in its original columnar micro-partition format. Databricks Photon transcodes the data into a different columnar format optimised for SIMD vectorised processing. The Databricks cache is not just the same data stored closer to the CPU — it is the data reorganised for how the Photon engine consumes it.</p><p>In both cases, the cache is volatile. Suspend a Snowflake virtual warehouse, and the cache is gone. Terminate a Databricks cluster, and the cache is gone. This creates a question that belongs in every TCO analysis but rarely appears there: <strong>what is the cost of a cold start, measured in business terms?</strong></p><p>If the morning reporting window opens at 06:00 and the warehouse was suspended overnight, how long will it take for dashboards to respond at an acceptable speed? What does it cost to keep the warehouse running all night just to keep the cache warm? For Databricks specifically, how much of the Photon transcoding overhead is paid on every cold start? In fairness, modern partition pruning on both platforms means the cold-cache penalty applies only to the working set, not the entire dataset — but even a pruned working set of several hundred gigabytes still takes meaningful time to re-cache over a network.</p><p>To be fair, Databricks offers mitigations: cluster pools pre-allocate instances so the virtual machines are already running when a job starts, and serverless compute shifts lifecycle management to Databricks itself. These reduce the <em>time</em> to cold start, but they do not eliminate the underlying constraint — the cache is still cold, and the data still needs to be fetched and transcoded before queries run at full speed.</p><p>These are real operational costs. They never appear in the vendor TCO calculation. To be precise, Teradata also has a caching hierarchy — the FSG cache bridges the gap between RAM and local disk, just as the cloud platforms bridge the gap between SSD and object storage. The difference is the size of the latency gap being bridged. RAM to local disk is microseconds. SSD to object storage across a network is milliseconds. That is orders of magnitude, and it is why a cold start on Teradata is a non-event while a cold start on a cloud platform is an operational concern.</p><h3>Photon and the hybrid execution model: how much of your workload actually benefits?</h3><p>Databricks built the Photon engine to address a fundamental limitation: Apache Spark runs on the JVM, which introduces garbage collection pauses, object creation overhead, and an inability to exploit modern CPU features such as SIMD vectorisation. Photon is written from scratch in C++ and processes data in columnar batches rather than row by row.</p><p>The performance improvement is real. Databricks reports two to eight times speedup on analytical workloads, and the architectural reasoning is sound — native code with explicit memory management will always outperform a garbage-collected runtime for this class of work. This is, incidentally, the same conclusion that traditional data warehouse engines like Teradata arrived at years ago: native code, direct hardware access, no garbage collector in the execution path. The difference is that Photon has to solve a harder deployment problem — running on heterogeneous commodity cloud hardware with elastic scaling and open data formats, rather than on a fixed, purpose-built appliance.</p><p>However, there is an architectural constraint that deserves close scrutiny. Photon does not replace Spark — it runs alongside it. The query optimiser routes each physical operator to either Photon or the standard Spark engine, depending on whether Photon supports it. If a query calls a Python or Scala UDF, uses the RDD API, or hits an unsupported operation, execution falls back to the JVM for the remainder of the workload.</p><p>The question I would ask is: <strong>what percentage of my current workload uses functionality that Photon does not support?</strong> If the ETL pipelines are full of custom transformations — and in my experience, most enterprise pipelines are — then the Photon performance numbers are irrelevant for those workloads. You are buying a sports car that can only be driven on certain roads.</p><p>It is worth noting that Databricks has improved this situation. Python UDFs now use Apache Arrow serialisation for the handoff between Photon and the JVM, which significantly reduces the serialisation penalty compared to earlier versions. The gap is narrowing. But the core architectural point remains: native SQL on Photon will always outperform UDF-heavy workloads, and organisations should audit their pipeline complexity before assuming Photon benchmarks apply to their reality.</p><p>The follow-up: <strong>What is the roadmap for closing the Photon coverage gap?</strong> If Databricks plans to support UDFs natively within two years, that changes the calculus. If not, the organisation must plan for a permanent two-tier performance reality within the same platform.</p><h3>Result caching: an underappreciated differentiator</h3><p>Snowflake offers a result cache at the cloud services layer that stores the output of every query for twenty-four hours. If the same query is resubmitted and the underlying data has not changed, the result is returned instantly — no warehouse required, no compute cost incurred.</p><p>In a BI-heavy environment where the same dashboards get refreshed by dozens of users throughout the day, this is architecturally significant. The query runs once, and everyone else gets the answer for free.</p><p>Databricks does not have an equivalent mechanism at the platform level. It offers Delta Cache for data caching and DataFrame persistence in memory, but neither provides the zero-compute-cost instant return of an identical query that Snowflake’s result cache delivers. Photon accelerates the execution of each query, but each query still executes.</p><p>Teradata handles this differently again. There is no built-in result cache in the Snowflake sense, but the combination of FSG cache (which keeps recently accessed data blocks in RAM on each AMP) and the lack of network latency to storage means that repeated queries against the same data are already fast by design.</p><p>The question for any BI-heavy organisation evaluating all three platforms: <strong>how many of my daily queries are effectively duplicates?</strong> If the answer is “a significant percentage,” Snowflake has a structural advantage that raw engine speed cannot compensate for.</p><h3>Teradata in this comparison: why are you actually leaving?</h3><p>This is the question I would honestly ask myself before spending a single euro on migration planning.</p><p>If Teradata meets the organisation&#39;s needs and performance is acceptable, then the architectural trade-offs of both cloud platforms — cold caches, network latency to storage, hybrid execution engines, volatile compute — are problems the organisation is <em>choosing</em> to adopt. The Teradata architecture does not have these problems because it never separated storage from compute in the first place.</p><p>The compelling reasons to leave are real, but they are specific: the licensing model is unsustainable, the organisation needs elasticity it genuinely cannot get, or it needs capabilities that Teradata does not offer natively. And those capabilities are not trivial — native JSON and semi-structured data querying without ETL, zero-copy data sharing with external partners, Python and ML model execution inside the compute engine, streaming ingestion as a first-class citizen, and an open storage format that prevents permanent vendor lock-in at the data layer. These are genuine architectural advantages of the cloud platforms, not marketing fluff, and for some organisations, they are the deciding factor.</p><p>To be fair to Teradata as well, this is not the same platform it was ten years ago. VantageCloud offers cloud infrastructure deployment, and ClearScape Analytics adds native ML and advanced analytics capabilities. Teradata has moved toward cloud-native features. But the core performance differentiator — data locality on the AMP, no network hop to storage — remains the architectural property that sets it apart, and that property is inherent to the coupled design, not something that can be bolted on to a decoupled system.</p><p>If the answer is primarily “licensing cost,” then the honest follow-up is whether the total cost of ownership after migration — including the migration itself, the re-engineering of ETL, the re-training of teams, and the operational costs of managing cache behaviour, cluster sizing, and performance tuning in a fundamentally different architecture — is actually lower.</p><h3>The meta-question</h3><p>If I were sitting across the table from any of these three vendors, the final question I would ask is this:</p><p><strong>Show me a reference customer in my industry, at my scale, who migrated from Teradata and achieved the promised TCO reduction after twenty-four months in production — not a proof of concept, not a sandbox, but a full production migration including the cost of the migration itself.</strong></p><p>In my experience, that reference is extremely difficult to produce. Not because the platforms are bad — all three are technically excellent in their respective design spaces — but because most large-scale migrations are still in progress, or have quietly exceeded their budgets.</p><p>The right platform is the one whose architectural trade-offs align with your actual workload, usage patterns, and your organisational capability to operate it. That answer is different for every organisation, and no vendor slide deck will give it to you.</p><p><em>Roland Wenzlofsky is the founder of DWHPro, a Vienna-based enterprise data warehousing consultancy. He is a Teradata Certified Master with over twenty years of experience across European banking, insurance, and telecom, and the author of “Teradata Query Performance Tuning.” DWHPro provides vendor-neutral advice across Teradata, Snowflake, and Databricks.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5cd917d1cd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microsoft Fabric vs. Databricks: What Enterprise Teams Actually Need to Know]]></title>
            <link>https://medium.com/@r.wenzlofsky/microsoft-fabric-vs-databricks-what-enterprise-teams-actually-need-to-know-f701239323e1?source=rss-d2fd690c84------2</link>
            <guid isPermaLink="false">https://medium.com/p/f701239323e1</guid>
            <dc:creator><![CDATA[Roland Wenzlofsky]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 16:38:55 GMT</pubDate>
            <atom:updated>2026-03-08T16:38:55.877Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HUVyHhaTCsJ-JbaXkNRgfA.png" /></figure><p>When Microsoft launched Fabric, it promised to unify analytics under one roof. Databricks, meanwhile, has been building its lakehouse platform for years. Both aim to be your central data platform — but they make fundamentally different bets on how much complexity to show you.</p><p>Having worked across Teradata, Snowflake, and Databricks environments in European banking and insurance for over 20 years, I’ve seen this pattern before. The question is never “which platform is better?” It’s “Which trade-offs fit your organization?”</p><h3>The Architecture Split: Storage vs. Catalog</h3><p>The difference starts at the foundation.</p><p><strong>OneLake</strong>, Fabric’s storage layer, acts as a single unified data lake per organization — or per <em>tenant</em> (Mandant), as it’s called in the Microsoft world. Think of it as OneDrive for analytical data. Everything lives under one logical namespace. When you need to access data sitting elsewhere — in Azure Data Lake Storage, Amazon S3, or another lakehouse — you create a <strong>shortcut</strong>. It’s essentially a symbolic link that makes remote data appear local. No ETL, no copying.</p><p><strong>Databricks</strong> takes a different approach. Your data lives wherever you put it — S3, ADLS, GCS — and <strong>Unity Catalog</strong> provides the governance layer on top. You register external tables that point to data in cloud storage. The table definition lives in the catalog; the data lifecycle is managed outside of Databricks.</p><p>The practical distinction: OneLake shortcuts are a <strong>storage-level</strong> abstraction (remote data appears as local files), while Databricks external tables are a <strong>catalog-level</strong> abstraction (remote data is registered as queryable objects).</p><p>Both avoid data duplication. Both leave your source data untouched if you delete the reference — drop an external table in Databricks or remove a shortcut in Fabric, and only the metadata disappears. The underlying files stay exactly where they were.</p><h3>The Visibility Question: Explicit vs. Hidden Complexity</h3><p>This is where the philosophical divide gets interesting.</p><p>Databricks makes Unity Catalog a first-class citizen. You work with a three-level namespace — catalog.schema.table — and you&#39;re expected to understand it. You create catalogs, assign them to workspaces, define schemas, set permissions at each level, and consciously decide whether a table is managed or external. It&#39;s powerful, but it demands competence.</p><p>Fabric abstracts most of this away. Under the hood, there absolutely is a metastore — built on Delta Lake with a metadata layer. But you interact with <em>lakehouses</em> and <em>workspaces</em>, not catalogs. Create a lakehouse, drop files in, and Fabric auto-discovers tables, registers them, generates a SQL endpoint, and makes everything visible in Power BI. You rarely think about catalog plumbing.</p><p>This reflects their target audiences. Databricks assumes data engineers and platform teams who want fine-grained control. Fabric targets a broader group — including analysts and BI developers — who just want things to work.</p><h3>Easier to Start vs. Easier to Master</h3><p>Here’s where I push back on the narrative that Fabric is simply “easier.”</p><p>Hiding complexity and eliminating complexity are two very different things. The complexity is still there. It’s just behind a curtain. And that creates real problems in enterprise environments.</p><p><strong>Diagnostics.</strong> When something breaks in Databricks, you can dig into the Spark UI, check catalog permissions, and trace execution plans. In Fabric, you’re more dependent on whatever Microsoft chooses to surface.</p><p><strong>Tuning.</strong> Fabric makes opinionated defaults for you — compute sizing, caching, file layout, optimization. When those defaults don’t fit your workload, you have fewer levers to pull. Databricks gives you more rope. You can hang yourself with it, but you can also build exactly what you need.</p><p><strong>Cost transparency.</strong> This is the one industry that regulated industries should care about most. Fabric’s capacity-based pricing bundles everything together. Databricks itemizes compute, storage, and features more explicitly.</p><p>This is the same pattern we’ve seen in the Snowflake world. No indexes, no tuning knobs, auto-scaling warehouses — great until you get the bill and can’t explain why a query consumed 200 credits.</p><h3>What This Means for Regulated Industries</h3><p>In European banking, insurance, and telecom, the ability to audit and govern your platform usually outweighs ease of onboarding. Auditors want to see clearly defined governance hierarchies, explicit access controls, and traceable data lineage.</p><p>Databricks with Unity Catalog gives you this directly — the three-level namespace maps cleanly to organizational governance structures. Fabric can achieve similar outcomes, but you’ll lean more heavily on workspace-level security and Microsoft Purview integration to satisfy compliance requirements. It works, but it’s a different path.</p><h3>The Bottom Line</h3><p>Fabric is easier to <strong>start</strong> with. Databricks is easier to <strong>master</strong>.</p><p>Which one matters more depends on your team’s maturity, your governance requirements, and how much you need to understand what’s happening under the hood. For a self-service analytics team building dashboards, Fabric’s integrated experience is compelling. For a data engineering team running complex pipelines across regulated environments, Databricks’ explicit control pays dividends.</p><p>The honest answer — as always in enterprise data — is that the best platform is the one your organization can actually operate and govern.</p><p><em>Roland Wenzlofsky is the founder of DWHPro, a Vienna-based data warehouse engineering consultancy. A Teradata Certified Master with 20+ years of experience in enterprise data warehousing across European banking, insurance, and telecom, he serves as a vendor-neutral advisor across Teradata, Snowflake, and Databricks. He is the author of “Teradata Query Performance Tuning”</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f701239323e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>