Move the workloads. Modernise what you can. Keep the business running. Exit the data centre. Demonstrate progress.

Cloud migration can create a paradox when we become better at migrating legacy architecture than modernising it. Technical debt is easier to carry into the future when organisations migrate more quickly. Instead of being redesigned, legacy systems are moved. New infrastructure preserves outdated operational models. Complexity survives the move.

There is nothing inherently wrong with moving an existing application into the cloud.

In many cases, it is exactly the right decision. However, the problem comes when “temporary” compromises quietly become the new architecture.

A legacy database is migrated rather than redesigned. An old integration survives behind a new API. A batch process becomes a cloud-native scheduled job. Familiar reporting structures are recreated on a modern platform.

And because the new environment is faster, more scalable, and easier to provision, the underlying debt becomes harder to see, not less significant.

Why this matters now

Cloud migration has matured. The question is no longer simply, “Should we move?” It is increasingly, “What should we change while we move?”

I saw this tension in a recent project where business applications were migrated from on-premises infrastructure to AWS.

The priority was to migrate successfully while minimising disruption. So, while the technology underneath changed, the business processes around it remained largely untouched. Staff continued using the same customer onboarding and approval workflows they had used before.

On the surface, that made sense. Changing the applications and the way people worked at the same time could introduce unnecessary risk. Keeping familiar processes helped the business maintain continuity while the migration took place.

But one problem remained: customer onboarding was still slower than it needed to be.

The migration hadn’t removed the underlying bottlenecks in the workflow. The same steps still had to be completed, the same approvals still had to happen, and customers still experience delays while their applications moved through the process.

The applications were now running in AWS, but the business outcome hadn’t fundamentally changed.

If customer onboarding is still too slow after migration, the next question should be why. Which steps genuinely add value? Where do approvals get stuck? What could be automated, simplified or removed altogether?

Cloud migration creates an opportunity to ask those questions. It doesn’t answer them for us.

Three questions worth asking

1. Are we migrating workloads or capabilities?

A workload is easy to count. A capability is harder to measure.

A programme can proudly report hundreds of applications migrated while the business experiences very little meaningful change.

Before moving anything, ask what business capability it supports and whether that capability should work differently in the future.

2. Which debt are we deliberately carrying?

Technical debt is not automatically bad. In many cases, taking on debt is a rational trade-off that allows the business to move faster.

The problem is not the debt. The problem is pretending it does not exist.

If an application is being migrated unchanged because speed matters more than redesign, document the decision. Assign ownership. Set a review date.

3. Are we measuring migration speed or transformation?

Speed is attractive because it produces a number. Transformation is harder because it produces outcomes.

A successful migration should eventually tell a story beyond workloads moved and servers decommissioned.

Can teams release changes faster? Can data be shared and used more effectively?

Can the organisation adapt products and processes without rebuilding technology every time? Has operational complexity genuinely decreased?

Those are the measures that determine whether migration created transformation or simply relocated existing systems.

Slowing down before migrating is sometimes the most beneficial thing a cloud program can do. Not everything needs to be modernised. However, not all things are worthy of preservation.

Cloud platforms provide extraordinary technical flexibility. They make it easier to scale, deploy, automate, and innovate. What they do not provide is architectural discipline.

That still comes from people who are willing to challenge assumptions, question long-standing dependencies, and ask whether yesterday’s design decisions still serve today’s objectives.

Takeaway

The next time someone celebrates how quickly an organisation has migrated to the cloud, the most important question may not be:

“How fast did you move?” It may be: “How much of the old thinking came with you?”

The best cloud migrations do not simply relocate technical debt. They expose it. They create the opportunity to decide what is worth keeping, what should be modernised, and what should finally be retired.

Because transformation is not measured by how much infrastructure reaches the cloud. It is measured by how much unnecessary complexity is left behind.

The cloud migration paradox: faster migration, slower transformation was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

Congratulations. You survived your pilot.

The demo landed well. The numbers looked promising. Stakeholders nodded approvingly. Leadership agreed there was something worth pursuing and authorised the next phase.

And then… not much happened.

Six months later, the pilot is still being referenced in presentations. It occasionally resurfaces during innovation discussions. Maybe it has even expanded to a second use case. But it hasn’t become a core business capability. It hasn’t changed how the organisation operates at scale.

If that sounds familiar, you’re not failing at Agentic AI. According to BCG, 74% of organisations have yet to develop the capabilities needed to move beyond proofs of concept and generate tangible value from AI. The bottleneck is rarely the technology itself. More often, it’s the absence of the operating model, governance, ownership and change required to scale. [1]

You’re failing at something much older, and much harder: organisational change.

Passing the pilot was never the finish line — It was the beginning of the journey

One of the biggest mistakes organisations make is treating a successful pilot as proof that they are ready for scale and immediately moving on to the next agent they want to build.

In reality, three very different milestones exist. A Proof of Concept demonstrates that the technology works. A Proof of Value demonstrates that the technology creates measurable business outcomes, whether that’s reducing effort, improving service, accelerating decisions, or lowering risk. Scaling into production proves something entirely different: it proves an organisation can operate, govern, maintain, and continuously improve AI solutions as part of business as usual.

Most organisations successfully complete the first stage. Many complete the second. Only a small number successfully make the transition to the third. And when they fail, it is rarely because the technology stopped working.

The excitement is understandable. The first pilot delivers results, stakeholders see the potential, and attention quickly turns to identifying the next opportunity. But scaling Agentic AI isn’t about building more agents. It’s about building the capability to run, govern, adopt, and continuously improve them across the organisation. It’s about making success repeatable by creating a framework that works for your organisation.

The real challenges start after the demo

Across the Agentic AI programmes I have reviewed, the technology has almost never been the primary constraint. The agents performed as expected, the models generated useful outputs, and the automation delivered measurable improvements. What was missing was everything around the technology.

In many cases, no capabilities existed to support scaling. There was no operating model, no governance structure, no intake process, no standards, and no clear ownership for what happened after deployment. Just as importantly, there was little focus on organisational readiness. Teams were expected to adopt new ways of working without understanding how the agent would affect their roles, responsibilities, or performance measures.

The technology was ready. The organisation wasn’t.

No one owned the agent once the project team disbanded. No one had a process to decide which use case got built next, or why. There was no forum to catch a bad decision before it reached a customer, an employee, or a regulator. The teams whose work the agent touched were never brought into the design and simply inherited a fait accompli. Unsurprisingly, some resisted it. In more than one case, the people closest to the process were still being measured on the manual work the agent was meant to remove.

None of that shows up in a pilot’s success metrics. All of it determines whether the pilot survives contact with the rest of the organisation.

Why successful pilots stall

During the pilot phase, a dedicated project team provides structure, sponsorship, decision-making, and momentum. Problems get solved quickly because everyone involved is focused on a single objective: proving the concept. Once the pilot concludes, that temporary structure disappears and a new set of questions begins to emerge.

Who owns the agent? Who is responsible for monitoring performance? Who approves changes? What happens when regulations change? How are new use cases prioritised? How do you ensure different teams are building agents in a consistent and controlled way?

Another challenge often emerges at this point: adoption.

Building an agent and getting people to use it are two very different things. Successful organisations actively manage adoption from day one. They engage business users in the design process. They communicate the purpose of the solution early and transparently. They align incentives and measures with the future-state process rather than the current one.

Most importantly, they help people understand how AI will augment their work rather than simply replace activities.

When adoption is treated as an afterthought, resistance fills the gap. Even the best-designed agent will struggle to deliver value if the people expected to use it do not trust it, understand it, or see a place for themselves in the future state.

Executive sponsorship and clear, consistent communication across the business are critical to overcoming that resistance. People need to understand not only what is changing, but why it is changing, how it affects them, and what role they play in the future state. When leaders actively champion the change and business areas are engaged throughout the journey, adoption increases, trust grows, and the value promised in the pilot is far more likely to be realised in practice.

Without clear answers, pilots enter a state of limbo. They are technically successful but organisationally unsupported. The result is a growing portfolio of disconnected AI experiments that demonstrate potential but fail to generate enterprise-wide impact.

What “ready to scale” actually looks like

Recently, I helped design and establish an Agentic AI Centre of Excellence for a professional services organisation that had already demonstrated promising results through several AI initiatives.

The organisation did not need another pilot. It needed a framework for scaling what was already working. The objective was straightforward: to move from isolated experimentation to a repeatable, governed, and value-driven capability.

That required several foundational components.

First, ownership and accountability needed to be clearly defined. Every agent required a named business owner accountable for outcomes, risks, adoption, and ongoing performance.

Second, a structured intake and prioritisation process was introduced. New use cases would be assessed against strategic alignment, business value, feasibility, and risk rather than competing for attention based on visibility or stakeholder influence.

Third, a Design Authority was established to review solutions before they entered production. This created an opportunity to address governance, architecture, compliance, and operational considerations early rather than reacting after deployment.

Finally, a value realisation framework was implemented to ensure success could be measured consistently. “The agent works” ceased to be the benchmark. The focus shifted to measurable business outcomes and sustainable value creation.

Alongside the creation of these capabilities, two Agentic AI Proof of Value initiatives were delivered in Finance and People & Culture. The use cases successfully demonstrated how Agentic AI could streamline activities, reduce manual effort, and improve the employee experience.

Those outcomes were important, but the more important achievement was creating the conditions for success to be repeated. The agents mattered, but the operating model around them is what made them scalable.

The questions every leadership team should be asking

Before launching the next wave of Agentic AI initiatives, organisations should pause and assess whether they are truly ready to scale.

A few questions can quickly reveal the answer.

• Who owns each agent once the implementation team has moved on?
• How are new use cases identified, assessed, and prioritised?
• What governance exists before solutions reach customers, employees, regulators, or critical business processes?
• How are risks monitored and managed throughout the lifecycle of an agent?
• How is value measured after deployment, and who is accountable for delivering it?
• What investment has been made in change management, communications, training, and adoption?

And perhaps the most revealing question of all:

If your current pilot expanded to ten times its size tomorrow, would your existing operating model still work?

If the honest answer is no, then the organisation is not yet scaling Agentic AI. It is simply extending a pilot.

The speed of scaling is the speed of change

One lesson has become increasingly clear across every AI transformation programme I’ve been involved in.The ability of a Centre of Excellence to scale an AI programme is directly proportional to the ability of the organisation to change.

Not the quality of the models, not the sophistication of the agents and not the size of the technology investment.

Change readiness, leadership alignment, user adoption, trust, and the willingness to redesign the way work gets done are ultimately the factors that determine how quickly value can be realised at scale.

The exam isn’t over

A successful pilot proves an idea has merit. It does not prove an organisation is prepared to operationalise that idea across teams, functions, and business units.That requires a different kind of readiness.It requires ownership, governance, capability building, and, most importantly, people adopting new ways of working.

The most common obstacles to scaling Agentic AI are rarely technical. They are organisational. They are rooted in change management, adoption, incentives, trust, organisational politics, competing business priorities, internal pressures, conflicting goals, and resistance to change.

The organisations generating the greatest value from AI are not necessarily building better agents than everyone else. They are building stronger mechanisms to help their people adopt and scale those agents.

And there is another reality leaders should recognise.AI is moving exponentially. Humans do not.

We are naturally wired to think in linear terms. We expect progress to happen gradually and predictably. Agentic AI does not behave that way. Capabilities that seem experimental today can become mainstream far faster than organisations are prepared for.

Ironically, AI’s biggest scaling challenge is not the technology itself.: It’s us.

We are still the ones making investment decisions, approving budgets, defining policies, redesigning processes, and deciding whether new ways of working are adopted or rejected. We are still the ones acquiring solutions, signing contracts, and operationalising change.

For better and for worse, humans remain the bottleneck. But that is also good news. If people sit at the centre of the problem, they must also sit at the centre of the solution.

The organisations that succeed over the next few years will not be those that build the most agents. They will be the ones that help their people adapt, trust, and evolve alongside them, turning isolated AI successes into sustainable business capabilities.

Because surviving the pilot was never the destination. It was simply the beginning of the journey.

Enrique Pantoja

Agentic AI and Automation Technology Director

You survived your Agentic AI pilot. Now what? was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

We can. And in fact, we already do. But sovereignty is not a checkbox. It is a whole spectrum. Different data sets require different levels of sovereignty, and every organisation needs to decide what level of sovereignty is required for each workload — and the trade-offs it is willing to accept. A Power BI dashboard with public data is very different from a platform that processes classified case files or sensitive citizen data, for example.

Evaluating sovereignty

The European Commission’s Cloud Sovereignty Framework defines sovereignty objectives and Sovereignty Effectiveness Assurance Levels (SEALs) to make cloud sovereignty more measurable in procurement and risk assessment. Instead of relying on broad claims such as ‘European cloud’ or ‘sovereign cloud,’ organisations can assess concrete dimensions such as strategic alignment, legal and jurisdictional exposure, data and AI control, operational independence, supply-chain transparency, technology openness, security and compliance, and environmental dependency. This is especially relevant for public-sector organisations and regulated industries. Sovereignty requirements are becoming part of tenders and board-level risk discussions. The question is no longer only whether a platform is secure or scalable, but also whether the organisation can explain and defend its sovereignty decisions. This is a vital consideration for clients, shareholders, employees and regulators.

Realistic sovereignty

Full European digital sovereignty (SEAL 4) across all dimensions is difficult to achieve today. Not because the idea is wrong, but because modern technology and hardware are deeply interconnected in a global supply chain and depend on resources not found in the EU and chips not manufactured in the EU.

This does not mean that sovereignty is impossible, but it does mean that organisations need to be precise. Which risk are we trying to reduce, for which data, under which regulation, and at what cost? For most workloads, a major public cloud provider remains the best choice because it offers mature services, strong security capabilities, advanced AI features, and a fast developer experience. For other workloads, especially those involving classified, sensitive, or strategically important data, a more sovereign architecture may be required.

Three practical routes for data & AI platforms

At Valcon, we work from data and AI strategy to end-to-end implementation. We help clients assess sovereignty risks and translate them into practical architecture choices, after which we implement, together with the client, a data & AI platform in their environment of choice.

For data and AI platforms, we typically see three realistic routes:

1. Public cloud for speed and innovation: platforms such as Databricks provide a strong developer experience, mature data and AI capabilities and continuous innovation. This can be the right choice when the data classification and risk profile allow it.
2. A disconnected data platform for highly restricted data: some organisations work with classified or extremely sensitive data that cannot be stored or processed in a public cloud. In those cases, a disconnected data platform that can run air-gapped (with locked-down access to the internet or any other digital environment) within the perimeter of the client’s organisation can be the right solution. This increases infrastructure and operational complexity, but it also enables the organisation to process data safely within its own network boundaries.
3. A sovereign data and AI platform for controlled flexibility: when a US public cloud is not acceptable, but a fully disconnected platform is too heavy, a sovereign data and AI platform can provide a middle ground. It is based on open-source technology, is designed to be portable, and can run in a private cloud, on European cloud providers such as STACKIT, Scaleway or OVHcloud, or on dedicated infrastructure.

A client may decide to keep general analytics workloads on a hyperscaler, for example. While moving sensitive AI inference workloads to a controlled European environment. That is often more realistic than trying to make the entire IT landscape sovereign overnight.

Sovereign data & AI platform architecture

A sovereign data and AI platform should avoid unnecessary lock-in while still offering a modern engineering experience. Valcon builds on proven open-source components by Databricks such as Apache Spark, Delta Lake, Unity Catalog and MLflow, and combines them with technologies such as vLLM and open-source AI inference models. The platform can run on Kubernetes, uses S3-compatible object storage and supports vector storage for AI workloads. This architecture supports both an exit strategy from public cloud, a hybrid scenario, and a standalone setup.

One size doesn’t fit all

Valcon already has considerable experience of creating data sovereignty within the financial, public and defence sectors. This is where Valcon can help: not by pushing one default platform, but by helping clients decide which workloads need which level of control and then building the platform to match. That can be in a client’s own environment, on a European cloud provider, or on a public cloud platform where that is the right fit.

Sovereignty is not about choosing one technology forever. It is about understanding your risks, making conscious trade-offs, and building a platform strategy that gives you the right level of control.

Come and speak to Valcon about data sovereignty

If you are having similar discussions, feel free to reach out to speak to us. As a European company, we feel the responsibility to help European clients with their sovereignty challenges and solve them. Please contact: tim.ellens@valcon.com and sander.de.jongh@valcon.com

Sovereign data & AI platforms — a path forward for European organisations was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

Artificial intelligence is often positioned as a silver bullet: faster decisions, smarter experiences, step‑change productivity. The reality, particularly in large enterprise environments with complex legacy data, is far more nuanced.

Recently, our team worked with a global organisation facing a deceptively simple challenge: how to help frontline specialists find the right technical information quickly, reliably and safely.

What followed was a multi‑month journey through performance bottlenecks, architectural trade‑offs, and some hard‑won lessons about where AI genuinely adds value — and where it should be kept firmly on a leash. And valuable knowledge on how to scale enterprise AI through performance tuning.

The client challenge: knowledge at scale, under pressure

The client operates in a highly regulated, safety‑critical environment. Their frontline workforce relies on vast volumes of technical documentation: repair manuals, diagnostic guides, service bulletins and procedural updates accumulated over decades.

Two pressures were colliding. The first was information overload — for even relatively standard tasks, documentation could span thousands of pages. Finding the right section often required deep expertise and time-consuming navigation. The second issue was shrinking experience levels — as experienced specialists retired, newer staff had to handle increasing technical complexity with less tacit knowledge to fall back on.

Traditional keyword search was no longer sufficient. The documents were structured for display, not meaning. Semantic relationships were weak, terminology inconsistent and the same concept could appear under different names depending on context.

The ambition was clear: provide a conversational, context‑aware interface that allows specialists to ask questions in natural language and receive actionable, trustworthy guidance — grounded entirely in authorised source material.

The solution: retrieval‑augmented AI, built for the real world

The solution combined modern AI techniques with deliberately conservative engineering choices.

At its core was a retrieval‑augmented generation (RAG) architecture. Rather than asking a large language model (LLM) to ‘know’ the answers, the system retrieves relevant source documents first and then asks the model to synthesise an answer strictly from those sources.

This distinction mattered. In safety‑critical environments, hallucinations are unacceptable. Every generated answer needed to be traceable back to official documentation, with clear signalling of what was retrieved versus what was generated.

The front end was designed around the realities of the user context: minimal distraction, emphasis on safety information, clear step-by-step flows, and support for hands‑free interaction such as voice input and read‑back.

Behind the scenes, however, the real challenge wasn’t functionality — it was performance.

Performance is the product

Early prototypes worked functionally but failed a basic user test: initial response times were taking 50–60 seconds and in a workshop or operational setting, that’s an eternity. Response times would need to be reduced to single seconds.

It quickly became clear that AI performance was being impeded by the cumulative cost of many small decisions, such as:

• Keeping it ‘simple’ with a single LLM call for entire answer generation
• Ensuring the full breadth of relevant content was featured in a ‘rich’ answer
• Traditional separation of functions, i.e. generate answer first and then translate.

Individually, each of these steps appeared reasonable. But together, they pushed response times far beyond acceptable non‑functional requirements.

The team ultimately reduced end‑to‑end response times to around 12–15 seconds through leveraging detailed content and domain knowledge for aggressive parallelisation — the process of breaking one big job down into smaller independent units that can be run at the same time. They did this for content processing, careful model selection and ruthless placement of function to move AI features that added latency into code wherever possible.

One example stood out. Even asking a model to maintain consecutive numbering of references significantly increased the processing time. Seemingly trivial output constraints can impose surprising cognitive and computational overhead. In this case, it was almost trivial to handle this in code, but doing so started moving us away the sort of ‘simple’ implementations most initial software builds strive for.

The myth of ‘just upgrade the model’

A common executive instinct is to assume newer models will automatically fix performance or quality problems. In this case, the experience was the opposite.

Upgrading between models — even within the same family — produced materially different outputs. Prompts that previously yielded concise, technician‑friendly answers suddenly exploded into verbose explanations. In some cases, older models outperformed more advanced ones simply because they adhered more literally to instructions and generated less text with less reasoning. Unless it is constrained, more reasoning means more text which means bigger answers and slower performance.

The implication is important: model upgrades are infrastructure upgrades. They require prompt re‑engineering, regression testing, and often architectural changes. Treating them as infrastructure swaps is a costly mistake.

Where AI helps — and where it hurts

Perhaps the most important lesson was philosophical rather than technical.

AI was invaluable where the problem space was fuzzy: extracting ‘interesting values’ from messy documents, understanding how humans phrase questions, or summarising complex procedures for different experience levels. This is where strong prompt engineering and a domain knowledge layer come in — to place controls around interpretation and to guide output towards intended outcomes.

It performed poorly where deterministic behaviour was required. Tasks such as classification, filtering, exclusion rules and prioritisation often worked better — and faster — when implemented in conventional code.

Similarly, when asked to do multiple activities, for example when generating an answer, often the AI would ‘helpfully’ infer rules about one activity from another leading to unexpected results based on a minor change in the source material put before it and unpredictability when amending the prompt for one activity would alter the output for another.

Over time, a clear design principle emerged:

If a rule can be expressed as a simple conditional statement, write it in code. Use AI only where genuine interpretation is required on an activity-by-activity basis.

This hybrid approach reduced latency, improved consistency and made the system easier to reason about and maintain.

Economics, scale, and the illusion of speed

At enterprise scale — tens of thousands of users — even small inefficiencies become expensive. Token usage, caching strategies, provider‑specific behaviour and request orchestration all have real cost implications.

The team found that prompt caching and reuse of static context dramatically reduced both cost and latency, but only with a proper understanding of the caching capabilities offered by different model providers. Benefits can still be derived from all-or nothing caching rather than more dynamic caching, but more attention to this is required up front.

Another important insight was that perceived speed often matters as much as actual speed. Carefully designed status updates and progress indicators transformed user tolerance, even when the underlying processing time remained unchanged.

Streaming, in other words, is not magic — it is theatre. But sometimes, that theatre is necessary.

What executives should take away

This project reinforced several truths about enterprise AI:

• AI is not slow because it is immature; it is slow because it is doing a lot of different activities humans normally do implicitly
• Architectural discipline matters more than model choice
• Performance tuning is not an optimisation phase — it is core product design
• Determinism and transparency are as important as intelligence in regulated environments; strong prompt engineering and the application of domain knowledge into prompt intent are key

Above all, success came not from chasing ‘more AI’, but from being deliberate about where intelligence was applied and where it was deliberately constrained. For organisations embarking on similar journeys, that may be the most valuable lesson of all.

Performance tuning AI — lessons from the front line was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

For the past year, we have been building a data product for a large retail client that has to comply with the EU Deforestation Regulation (EUDR). The short version of the regulation is this: for products containing in-scope commodities that cross the EU border, the company must file a Due Diligence Statement with the EU’s central system, including geolocation data for the plots where the raw material was produced.

For an operational compliance pipeline, failure does not look like dashboard numbers that are incorrect. The consequences are financial and reputational. The design we created revolved around the core principle of trust and transparency. The goal was not to get 100% accuracy; it was to build a system that could explain why a certain piece of data was not processed. The result is a pipeline that can answer the question of why without much effort.

This post describes how that works in practice. The stack is Databricks (Unity Catalog + Delta Lake) with PySpark pipelines, but the pattern transfers to any Spark platform.

Each contract is a YAML file in the repository, loosely following the Open Data Contract Standard (https://bitol-io.github.io/open-data-contract-standard/). It includes contract metadata, a description block, an ownership section, and one or more schemas. We keep one contract per layer or pipeline: a silver contract, a gold contract, and a handful of pipeline-specific ones, such as item scope, tariff codes, and supplier flows.

A trimmed example:

domain: supply-chain
dataProduct: eudr-compliance
kind: DataContract
version: 1.0.0
status: active
description:
purpose: Supply chain data models for EUDR compliance analysis
limitations: Limited to supply chain operations for specific business units
usage: Track and analyse supply chain operations for regulatory compliance
notes: Every table has a staging counterpart with the same model.
Data lands in staging first, then is merged into the gold table.
team:
  - username: data.engineer.one@client.example 
    role: Owner
    dateIn: 2024–04–04
  - username: data.engineer.two@client.example
    role: Backup
    dateIn: 2025–04–04
  - username: product.leader@client.example
    role: Data Product Leader
    dateIn: 2025–04–04
schema:
  - name: commodities
    physicalName: commodities
    physicalType: table
    description: Master record of regulated commodities linking products to
    their tariff classifications and validity periods
  properties:
    - { name: id, physicalType: string, required: true,
    description: Unique identifier, source: generated at run time }
    - { name: item_no, physicalType: string, required: true,
    description: Item number, source: upstream item master }
    - { name: tariff_no, physicalType: string, required: false,
    description: Tariff number, source: derived in tariff mapping step }
    - { name: valid_from, physicalType: date, required: true,
    description: Start date of validity, source: upstream item master }
    - { name: valid_to, physicalType: date, required: true,
    description: End date of validity, source: upstream item master }
    - { name: insert_date, physicalType: timestamp, required: false,
    description: Record insert timestamp, source: pipeline at run time }

There are three key values in our process:

● Ownership is in the contract. Every contract names an owner, a backup, and a data product leader, with the date they took on the role. When something breaks at 4 pm on a Friday, nobody has to search through an outdated wiki. The line of responsibility is clear and easy to understand.

● Every column documents its source. The source annotation records where a field comes from: an upstream system or a derivation step. It is not machine-parsed lineage; it is human-readable lineage that lives in the one file people actually open.

● Conventions are written down where they bind. The notes field documents the staging-then-merge pattern that every table in the file follows. The contract is the starting point when beginning to work with this data.

The core concept

A contract that is only documentation starts rotting the day it is created. The way we keep ours honest is to make everything downstream dependent on it. The same YAML file feeds four consumers:

                    ┌─────────────────────────────┐
                    │    data contract (YAML)     │
                    │    one per layer / feed     │
                    └──────────────┬──────────────┘
                                   │
       ┌─────────────────┬─────────┴─────────┬─────────────────┐
       ▼                 ▼                   ▼                 ▼
runtime schemas    DDL deployment      catalog docs      test fixtures
(PySpark Struct-   (create & evolve    (table + column   (unit tests build
 Types for frames   tables to match     comments synced   their frames from
 and API payloads)  the contract)       on every deploy)  the same schemas)

If the contract changes, the tables change, the runtime schemas change, the catalogue documentation changes, and the tests verify it all from one pull request.

1. Runtime schemas for the pipelines

A small custom class parses the YAML and converts each schema into a PySpark StructType. Pipelines never hand-write schemas:

loader = DataContractLoader("../model/data_contracts", logger)
commodities_schema = loader.get_schema("commodities", "gold_data_contract.yml")
df = spark.createDataFrame(rows, schema=commodities_schema)

The conversion is simple:

def _convert_to_struct_type(self, schema_def):
    fields = []
    for prop in schema_def.get("properties", []):
        fields.append(StructField(
            prop["name"],
            self._get_field_type(prop),
            nullable=not prop.get("required", False),
            metadata={k: v for k, v in prop.items()
                      if k in ("businessName", "description", "primaryKey")},
        ))
    return StructType(fields)

2. Deploying tables from a contract

The second consumer is a deployment step that treats the contracts the way Databricks treats an asset bundle file: as the desired state. On every environment deployment, a deploy_tables() routine:

1. Loads all contract files and merges schemas that target the same physical table.

2. Runs CREATE TABLE IF NOT EXISTS for anything new.

3. Diffs the live table against the contract with DESCRIBE TABLE.

4. Adds missing columns with ALTER TABLE … ADD COLUMNS.

5. Migrates type changes (more on that below).

6. Drops columns that are no longer in the contract, and can optionally drop whole tables that no contract claims.

Because it is a diff-and-converge loop rather than a one-shot DDL script, it is idempotent: running it against an up-to-date environment is a no-op, and dev, test, and production cannot drift apart because none of them is ever touched by hand.

The one difficult part is type changes. Delta Lake will not alter a column’s type in place for most conversions, so the deployer does a four-step process through a staging column:

spark.sql(f"ALTER TABLE {table} ADD COLUMN {col}_staging {new_type}")
spark.sql(f"UPDATE {table} SET {col}_staging = CAST({col} AS {new_type})")
spark.sql(f"ALTER TABLE {table} DROP COLUMN {col}")
spark.sql(f"ALTER TABLE {table} RENAME COLUMN {col}_staging TO {col}")

It is written this way because of Databricks’ limitations around altering columns in place. We decided early on that it was worth implementing because it gave us a true source of truth for every table. If we left this use case out, we could experience a correctness leak, and developers could directly alter the table state.

3. Documentation that lives in the catalogue

Analysts do not read YAML in a repository; they browse the data catalogue. So the deployment step also syncs every table and column description from the contract into Unity Catalog:

COMMENT ON TABLE catalog.schema.commodities IS 'Master record of regulated commodities…';
ALTER TABLE catalog.schema.commodities ALTER COLUMN tariff_no COMMENT 'Tariff number';

The sync is diff-based and only writes when the contract and the catalogue disagree. Descriptions are written once, in the contract, and show up in Unity Catalog, including in the catalogue’s search. When a description improves in a pull request, the catalogue improves on the next deploy.

4. Tests that break when the contract does

Our unit tests build their fixture DataFrames from the same loader:

contract_loader = DataContractLoader("../model/data_contracts", logger)
schemas = contract_loader.load_contract_schemas("gold_data_contract.yml")

If someone changes a contract in a way the pipeline code cannot handle, the tests constructing fixtures from that contract fail in CI. There is no second version of the schema inside the test suite, which is where schema drift goes to hide.

Why not just use DQX?

Alongside the deployment of tables based on the contracts, we had a few use cases where DQX did not fit, as of August 2026.

Several of our physical tables are fed by more than one flow, each producing a slightly different shape. Rather than copy and paste near-identical schemas, contracts support an extends property, which is resolved by the loader. This is a custom property and is not supported by DQX’s current implementation. I believe schema inheritance is important for consistency and accuracy:

- name: measures_base_schema
  physicalName: measures
  physicalType: table
  properties:
    - { name: commodity_id, physicalType: string, required: true }
    - { name: item_no,      physicalType: string, required: true }

- name: measures_qty_schema
  physicalName: measures
  customProperties:
    - property: extends
      value: measures_base_schema
  properties:
    - { name: sum_plan_qty, physicalType: string, required: false }

- name: measures_vol_schema
  physicalName: measures
  customProperties:
    - property: extends
      value: measures_qty_schema
  properties:
    - { name: dds_net_weight, physicalType: "decimal(22,2)", required: false }
    - { name: dds_net_volume, physicalType: "decimal(22,2)", required: false }

Chains are allowed (base extended by qty extended by vol), a child can override a parent’s property by redeclaring the same name, and new properties are appended. Note the separation between the logical schema name and the physicalName: three logical schemas here describe one physical table.

The flip side of that separation is that when several contract files each declare schemas for the same physicalName, in our case an EU-supplier flow and a non-EU-supplier flow that both write producers and measures, the deployer merges their properties into a superset before converging the table. Each flow’s pipeline builds frames against its own narrower schema; the physical table accommodates the union. Neither table contract needs to know about the other’s columns.

Next steps

Features that we maintain manually, such as contracts as state for tables and schema inheritance, are things I hope Databricks introduces as part of DQX. Schema inheritance is vital for the correctness of schemas and coherence during deployments. State-based tables are long overdue, not just from Databricks but as a standard practice in the industry. With current technology, there is no excuse for having so much hidden with regard to data models and data consistency.

We are also working on a new version of the custom utilities. Although we wish we could wait for new tools to come to us, at a certain point we have to start building them ourselves.

1. We have introduced contract versions and plan to enforce contract versions as breaking changes through our CI process.

2. Combining the DQX tools with our enhanced data contract utilities through a wrapper.

3. Centralising the data contract repository.

Conclusion

If you are introducing data contracts on a similar platform, this is what I would advise you to do:

1. Make the contract executable on day one. The difference between a contract and a document is that something breaks when the contract is wrong. Deriving runtime schemas and DDL from it is what turns documentation into an interface.

2. You do not need a platform purchase to start. Our entire implementation is YAML files, two Python classes of a few hundred lines, and a deployment step. Git and pull-request review are the governance mechanism. That said, I am a fan of Unity Catalog, and this would enrich whatever UC implementation you already have.

3. Put ownership and lineage in the file people open. A team block and per-column source annotations cost minutes to write and answer the two questions every incident starts with: who owns this and where does this field come from.

4. Let the physical and logical models diverge deliberately. Inheritance and merge-by-physical-name kept our contracts DRY while letting one table serve multiple flows.

5. Sequence the rollout by risk. Structure first (schemas, tables, docs), then contract linting, then quality rules and breaking-change checks. Structure is where the regulatory risk lived for us; your order may differ.

Most of the value showed up in human review and during the process, not at runtime. A schema change on this product is a one-file diff that a data engineer, a tester, and a product leader can all read, and everything downstream of that diff is a machine’s job.

One thing I have learned from the advent of AI is that organisations want accountability and ownership. Despite all of the code flying around, there will always be a documented state of your data if you enforce it; do not make it optional. Companies and people want somebody to speak to. Do not neglect the ownership parts of the document just because they do not break your pipeline.

My LinkedIn DMs are open if you’re interested in chatting about this more. Code and data standards are things I am genuinely interested in discussing.

Enforcing Data Contracts on Databricks was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

Generative AI is transforming software development. Tools such as Cursor, Lovable, Bolt, and Replit can generate applications from simple prompts, while low-code platforms such as Appian, Mendix, Pega, Power Platform, and UiPath are embedding AI to automate application delivery. As a result, organisations are increasingly choosing between three approaches: Vibe coding, AI-augmented development and AppGen (AI-driven low-code development). AI-augmented development sits between these extremes, combining professional software engineering practices with AI-powered coding assistants such as GitHub Copilot and Claude Code to accelerate development while maintaining architectural control, code quality, and governance. Vibe coding, AppGen and AI-augmented AI are used to accelerate analysis, design, coding, testing, and delivery.

Although all delivery methods use AI to accelerate development, they are designed for different goals. Vibe coding prioritises rapid creation, while AppGen focuses on delivering enterprise-ready applications that are secure, maintainable, and governed.

Dark factory AI development is a fully autonomous software process where AI agents write, test, review, and deploy code based on text specifications without human intervention on individual changes. This is a part which is not covered in this article and mainly used for development of non-mission critical applications, mainly due to the EU AI Act.

What is Vibe Coding?

Vibe coding is a development style where users describe what they want in natural language and AI generates the code. The emphasis is on speed, experimentation, and creativity. A developer or even a business user can build a working application in minutes.

The benefits are clear:

• Extremely fast prototyping
• Low barrier to entry
• Rapid experimentation
• Ideal for proofs of concept
• Reduced need to write code manually

However, as applications become larger and more business critical, challenges emerge. AI-generated code can be difficult to review, maintain, test, and secure. Organisations often struggle with governance, architectural consistency, compliance requirements, and long-term ownership. The developer may not be able to fully explain the AI-generated code or predict how it will behave in new scenarios, creating a dependency on AI not only for code generation but also for understanding, modifying, and maintaining the application.

What is AI-Augmented Development?

AI-Augmented Development is a professional software engineering approach in which AI assists developers by generating code, tests, documentation, architectural suggestions, and implementation accelerators. Unlike vibe coding the code is accessible for the developer when it is ready to be deployed. AI is used to increase productivity, while developers remain responsible for solution design, code reviews, testing, security, compliance, governance, and operational quality. This approach applies established software engineering best practices throughout the development lifecycle, making it suitable for enterprise-grade production systems where reliability, maintainability and scalability are applied.

What is AppGen?

To explain AppGen, it is important to first understand low-code, the development platform that provides the foundation for AI-assisted application development. Low-code is a software development approach that uses visual modelling, prebuilt components, and configuration to create applications within a structured development environment.

AppGen is an AI-driven application development approach that combines generative AI with a low-code platform to automatically create and refine enterprise applications from business requirements and natural language prompts. The key difference is that applications remain governed by the platform. Low-code platforms provide built-in capabilities such as:

• Security and access management
• Reusable components
• Governance and audit trails
• CI/CD and deployment management
• Monitoring and operational controls
• Integration with enterprise systems

This allows teams to achieve AI-driven productivity while maintaining enterprise standards.

The key difference

In short, Vibe coding delivers maximum speed but lacks the governance, security, and maintainability required for enterprise environments. AI-augmented development accelerates professional software engineering, but testing, review, and validation remain largely manual and time-consuming. AppGen (Low-Code) combines AI-driven productivity with built-in governance, enabling organisations to scale applications safely and efficiently.

Why Enterprises Need More Than Generate Code

Many organisations have successfully adopted AI assistants and coding tools. Yet relatively few have embedded AI into their operational core. The challenge is not creating software, it is integrating AI into business processes, systems, governance frameworks, and compliance requirements.

Several common challenges emerge:

• Context Gap: AI can generate functionality, but without access to connected business data and process context, its business value remains limited.

• Execution Gap: AI can provide recommendations, but applications are needed to orchestrate processes, systems, and people to turn recommendations into actions.

• Governance Gap: Organisations require built-in auditability, compliance, security, and policy controls to deploy AI safely and at scale.

• Fragmentation Gap: Independent adoption of AI tools can create disconnected solutions that are difficult to govern, support, and scale.

Total Cost of Ownership: The Hidden Cost Behind Speed

While all three approaches accelerate software delivery, their long-term Total Cost of Ownership (TCO) differs significantly. Initial development costs are only a small part of the equation. Maintenance, testing, security, governance, compliance, infrastructure, and future enhancements often represent the majority of the costs over an application’s lifetime.

Vibe Coding (Lovable, Bolt, Replit) typically has the lowest cost to create a prototype, but often the highest long-term TCO. As applications grow, organisations face increasing costs related to code review, testing, security remediation, architectural refactoring, documentation, and ongoing maintenance. The absence of built-in governance can also create technical debt and shadow IT, making ownership expensive over time.

AI-Augmented Development (Claude Code, GitHub Copilot, Codex) reduces developer effort and improves productivity, but organisations remain responsible for architecture, testing, quality assurance, security, deployment pipelines, monitoring, and maintenance. Development costs are lower than traditional coding, but long-term ownership still depends heavily on engineering discipline and skilled development teams. As a result, AI-Augmented Development typically delivers a lower TCO than traditional software development, but a higher TCO than AppGen platforms, as organisations must still manage much of the operational, architectural, and governance burden themselves.

AppGen (Low-Code) generally introduces platform and licensing costs upfront, but often delivers the lowest TCO for enterprise applications. Security, governance, deployment management, reusable components, monitoring, and compliance capabilities are provided by the platform, reducing maintenance effort and operational overhead. As a result, organisations can scale applications with fewer resources and more predictable long-term costs.

The Future Is Both; AppGen and AI-Augmented Development

The future is unlikely to be a choice between AI-Augmented Development and AppGen. Instead, organisations will use both.

• Vibe coding for exploration and rapid innovation — for non-enterprise environments.
• AI-Augmented Development for expert developers building complex custom solutions.
• AppGen (low-code) for enterprise execution, governance, and scalability.

As AI continues to democratise software creation, competitive advantage will not come from generating applications faster. It will come from turning those applications into reliable, governed, and maintainable business solutions.

For enterprises, the challenge is no longer building applications with AI. The real challenge is governing, operating, and scaling them across the organisation.

Vibe coding, AppGen, and AI augmented delivery: speed vs. scale in the age of AI was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

The first Databricks notebook is exciting.

The fiftieth notebook is a warning sign.

Every organisation starts its Databricks journey with a vision to modernise and unlock value from its data. The first few notebooks solve an important problem. The solution works, so the pattern gets repeated. Another team adopts it. New use cases arrive, and more notebooks follow.

Soon a data platform is born.

Initially, this approach works remarkably well. New datasets are onboarded quickly, teams move independently and business value is delivered fast.

However, as adoption grows, success begins to create its own challenges. More notebooks are created, more patterns emerge, and more custom logic is introduced. Gradually, a large share of engineering effort shifts from building new capabilities to maintaining existing ones.

This was a reality we faced with a client I recently worked with. The business wanted to move fast across strategic initiatives, but the platform was struggling to keep pace. Every new capability introduced additional complexity, and every new dataset seemed to require more effort than the last.

Eventually, the same question kept resurfacing:

Are we still delivering business value, or are we spending most of our time maintaining capabilities that Databricks can now provide natively?

The existing platform and its challenges

At the time, the platform consisted of nearly fifty pipeline notebooks spread across multiple business units. The implementation was built around custom PySpark transformations and reusable notebook patterns.

As business adoption increased, there was a growing need to introduce platform-wide capabilities such as:

• Change data capture (CDC)
• Schema enforcement
• Slowly Changing Dimension (SCD) Type 2 handling
• Governance tagging
• Row-level security
• Operational monitoring

Individually, none of these requirements were particularly difficult to implement. The challenge lay in the architecture itself.

Every new enhancement required repeated implementation effort across multiple pipelines. Features such as governance, CDC handling and schema validation could not simply be introduced once and reused consistently.

As a result, platform maturity slowed, operational complexity increased, and maintaining consistency across datasets became increasingly challenging.

Perhaps most importantly, architectural standards started relying more on tribal knowledge than on the platform itself.

That became the catalyst for a broader platform redesign.

The upgrade

As Valcon, our goal was to enable developers to spend most of their productive time developing business transformations, while platform capabilities such as governance, schema management, orchestration and deployment should be handled consistently and automatically.

To achieve this, we separated the implementation into two parts:

The framework (the “how”)

A reusable, installable Python package containing:

• Table abstractions
• Schema management
• CDC validation
• Governance enforcement
• Shared utilities

The Pipeline Layer (the “what”)

A Declarative Pipelines implementation with:

• Domain-specific business logic
• Table definitions
• Pipeline registration

Lakeflow Declarative Pipelines (LDP): Letting the platform manage the DAG

The most significant change was the introduction of LDP. Previously, execution order was explicitly managed through workflow design and dependency configuration.

With domain-specific LDP, we turned existing notebooks into series of dependable units, each with its custom transformations while the platform constructed and managed the execution graph automatically.

This significantly reduced orchestration overhead as platform concerns such as managing dependencies, checkpointing, retries and lineage were handled natively.

LDP helped turn architectural principles into platform capabilities, enabling teams to focus on business transformations rather than pipeline management.

Auto CDC: removed an entire category of Custom Code

Our historical SCD1 and SCD2 processing required significant custom merge logic. Every implementation introduced opportunities for subtle differences in behavior.

By adopting Auto CDC flows, historical tracking became a platform concern rather than an application concern.

It removed existing custom merge implementations, brought a consistent CDC pattern across the platform, improved maintainability, and reduced the engineering effort needed to evolve alongside platform requirements.

Citing our business roadmap, this was one of the strongest arguments for adopting modern Databricks capabilities and supercharging the development efforts.

Unity Catalog: governance moved closer to the data itself

During this migration, Unity Catalog became a key architectural building block.

Instead of managing governance as a separate operational activity, governance metadata became part of the schema definition itself.

Classifications, policies and row-level security were embedded directly and applied automatically to tables after pipeline execution.

With this, the platform automatically knew which columns contained sensitive information, which governance tags should be applied, and which policies should be validated.

By moving governance closer to the data itself, Unity Catalog helped transform governance from an operational burden into a scalable platform capability, while preparing the platform for future AI and analytics initiatives.

Databricks asset bundles: enabled platform consistency

Another major step was adopting Databricks Asset Bundles for deployment. Earlier, the deployments relied heavily on custom deployment logic and environment-specific configuration.

With the inclusion of Asset Bundles, deployment became declarative and repeatable. Development, test, pre-production and production environments all followed the same deployment model, with environment-specific configuration injected through bundle definitions.

Asset Bundles helped eliminate one of the most persistent challenges in data platforms:

It works in development, but not in production.

What could be improved further?

While data landscapes are constantly evolving, there is rarely an ultimate state of perfection. Several opportunities for further maturity remain:

1. Framework versioning
   Introduce support for running different framework versions across environments to reduce upgrade risk and improve release flexibility.
2. More metadata-driven patterns
   Move more platform capabilities from code into configuration to simplify onboarding and improve scalability across domains.
3. Continuously challenge custom implementations
   Regularly review custom components against the Databricks roadmap, as capabilities that once required custom development may now be available natively.

Key takeaways

1. Many data platforms gradually accumulate custom solutions for problems that modern platforms already solve.
2. Lakeflow Declarative Pipelines, Auto CDC, Unity Catalog and Asset Bundles helped us simplify the platform while improving consistency, governance and maintainability.
3. We realized that the most valuable code is often the code you no longer need to maintain.

As developers, our natural instinct is to see a problem and start building a solution. More often than not, that is exactly the right thing to do. However, as architects, it is equally important to occasionally step back and challenge our assumptions.

Looking back, the challenge was never implementing CDC, governance or deployment patterns. We knew how to build those capabilities.

The challenge was recognising when we should stop building them ourselves.

We stopped building data pipelines. Databricks started doing it for us. was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

Databricks is a rapidly evolving Data & AI platform, and platform engineers need a manageable way to maintain it. This is where Terraform comes into the picture. In this blog, we present our vision for organising Azure and Databricks resources in a way that can help you build and manage a Lakehouse platform more effectively.

If you need assistance, Valcon, a Databricks Gold Partner, can support you. This blog can be seen as a follow-up to the previous article, Managing Databricks with Terraform: A modular approach. Similar to that blog, this one focuses on Azure Databricks, although most of the concepts described here are also relevant for other cloud providers. We recommend reading the previous article first, as this blog builds upon it and references it throughout. References to the previous article are marked with the abbreviation MDT.

Our main guidelines when building these Terraform modules were:

• We should not mix different Terraform providers or provider configurations within a single module. This helps preserve clear authentication and security boundaries.
• All modules are designed with different operational roles in mind. Different teams often handle Azure administration and various aspects of Databricks administration, so modules are aligned with ownership boundaries.
• Different resources have different lifecycles, so the modules should respect that. Resources with similar rates of change should be grouped together.
• The modules should be simple to use. Usually, not all team members have Terraform experience, so we recommend using YAML and/or JSON files to define configuration wherever possible.
• Security is not an afterthought but is part of the platform by design. This includes network isolation, Unity Catalog permissions, workspace and account-level governance settings, and least-privilege access controls.

The operations required to build and manage the platform can be separated into four categories, each requiring different privileges:

• Azure subscription operations (Required: Contributor and RBAC Administrator roles, or Owner, at the required scope)
• Databricks account-level operations (Required: Databricks Account Admin role)
• Databricks metastore operations (Required: Databricks Metastore Admin role or appropriate Unity Catalog privileges)
• Databricks workspace operations (Required: Databricks Workspace Admin role)

The remainder of this blog is organised around Terraform modules. For each module, we will describe its purpose, key assumptions, the resources that belong to it, and the most important design considerations. The modules represent logical ownership, lifecycle, and security boundaries rather than mandatory Terraform state boundaries. An overview of all modules is shown in Figure 1.

One of the key challenges that influenced this module structure was deploying Databricks in environments where private CI/CD runners could not reach the Databricks account endpoint. Solving this challenge without introducing excessive pipeline and Terraform state complexity influenced several of the design decisions discussed later in this article.

Module 1: Azure resources — Deploys and configures Azure infrastructure.

Depending on your platform requirements, Azure resources can be organised into a single module or multiple Azure modules. As the primary focus of this blog is Databricks, we keep all Azure resources in a single module and only cover the common Azure resources required to support the Databricks platform.

Assumptions

• Azure subscription is created.
• Deployment CI/CD identity is created and granted Contributor and RBAC Administrator roles, or Owner, at the required scope.
• Virtual network is created.

Resources

Resource groups are logical containers for Azure resources that simplify organisation, access management, and separation of responsibilities. For example, network resources can be grouped together to simplify administration and permission management (resource: azurerm_resource_group).

Two subnets (container and host) are required for VNet-injected Databricks deployments for classic compute (resource: azurerm_subnet). VNet-injection is the recommended deployment model for classic compute as it provides greater control over networking and security. At least one additional subnet is typically required for private endpoints that enable secure connectivity to Azure resources that are not publicly accessible. Databricks recommends using a /26 or larger address space for each workspace subnet, and subnet sizing should be planned carefully to accommodate the expected cluster capacity.

Two key Databricks-related resources are deployed in this module: the Databricks Workspace (resource: azurerm_databricks_workspace) and the Databricks Access Connector (resource: azurerm_databricks_access_connector). Depending on the deployment approach, the creation order of these resources may be important. We will discuss this in more detail in a future blog.

The Databricks Access Connector provides a managed identity that Databricks can use to securely access Azure resources such as storage accounts, key vaults, and other services. Configuring access consists of two steps: granting the managed identity the required Azure permissions through role assignments (resource: azurerm_role_assignment) and granting Databricks principals access to these resources through Unity Catalog.

We recommend using separate access connectors for different resource domains. For example, if we create a storage account (resource: azurerm_storage_account) and an Azure Key Vault (resource: azurerm_key_vault), we should create two access connectors to improve security isolation and simplify permission management. If private access to these resources is required, private endpoints can also be created (resource: azurerm_private_endpoint).

When it comes to observability, there are two common options. Databricks system tables are often sufficient for Databricks-specific monitoring, while Log Analytics (resource: azurerm_log_analytics_workspace) together with diagnostic settings (resource: azurerm_monitor_diagnostic_setting) can be used when logs from Databricks and other Azure resources need to be stored centrally.

Finally, as a security best practice, separate identities should be used for Terraform infrastructure deployments and Databricks workloads. This helps split responsibilities and enforce the principle of least privilege. Depending on the use case, dedicated service principals or user-assigned managed identities (resource: azurerm_user_assigned_identity) can be used for Databricks jobs and other platform workloads.

Important considerations: Unity Catalog automatic enablement and identities

The usual workflow, marked with the blue line in Figure 1 and previously discussed in the MDT blog, is that after a Databricks workspace is created, Unity Catalog is enabled by attaching a metastore to the workspace at the account level. Once Unity Catalog is enabled, principals (users, groups, and service principals) are added to the workspace.

The challenge we faced at a client was that the private CI/CD runners used for Azure resource deployments did not have connectivity to the Databricks account endpoint. This would have required splitting the CI/CD pipeline into three parts: (1) private runners for Azure resource deployments, (2) public agents for Databricks account-level operations, and (3) private runners for Databricks workspace and Unity Catalog operations. In addition, this would have required maintaining three separate Terraform states, significantly increasing the complexity of the solution.

To keep the CI/CD process as simple as possible, we decided to avoid Databricks account-level operations during this phase (see the red line in Figure 1) and instead use a dedicated pipeline for Databricks account administration. This approach was possible because Databricks now supports automatic metastore assignment during workspace creation. With this option, Unity Catalog was enabled automatically, and principals could subsequently be added from the workspace itself, provided that automatic identity management was enabled at the account level. In this configuration, Entra ID principals become automatically available within Databricks.

However, there is an important catch. Although any principal can be added to the workspace through the Databricks UI, the same approach does not work directly through Terraform. The reason is that Entra ID identities must first be assigned a Databricks identity before they can be referenced in Terraform resources.

Fortunately, Databricks now provides identity resolution APIs that can be used at the workspace level. For example, service principals can be resolved using the resolveByExternalId API together with their Entra ID object ID. Similar APIs exist for users and groups, both at the workspace and account level.

To invoke these APIs from Terraform, the terraform_data resource can be combined with an idempotent local-exec provisioner that runs the Databricks CLI or directly calls the API. Alternatively, identity resolution can be executed as a CI/CD pre-deployment step before principals are referenced by Terraform.

Module 2: Unity Catalog objects — Manages data objects.

When it comes to Unity Catalog objects, we first need to define the boundary between resources managed through Terraform and resources managed through DABs (Declarative Automation Bundles, formerly known as Databricks Asset Bundles). We use Terraform for Unity Catalog objects that define the platform foundation, such as credentials, external locations, catalogs, schemas and volumes. Objects closely tied to applications or data products are better managed through DABs alongside the workloads that use them.

Assumptions

• Unity Catalog is enabled in the workspace.
• Deployment CI/CD identity is either a Metastore Admin or has the appropriate Unity Catalog privileges required to manage the target resources.
• Principals are resolved in Databricks.

Resources

Previously, we granted the Databricks Access Connector managed identity access to Azure storage accounts and services such as Azure Key Vault. The next step is to register this identity as either a storage credential or a service credential (resource: databricks_credential), depending on whether it will be used to access an Azure storage account or another service.

External locations (resource: databricks_external_location) and catalogs (resource: databricks_catalog) are the primary building blocks for organising data and defining access patterns and should therefore be created next. We define external locations at the storage container level because this keeps the storage boundary clear and easy to govern. Catalog design should primarily follow business boundaries such as environment, domain, or ownership, while medallion layers can be represented as schemas within those catalogs.

We can create catalogs as either open or isolated. Open catalogs are available from all workspaces, while isolated catalogs are available only from the workspace in which they were created. If a catalog needs to be shared with additional workspaces, it can be exposed using the databricks_workspace_binding resource in either read-only or read-write mode.

Once the Unity Catalog foundation is in place, additional resources such as schemas (resource: databricks_schema) and volumes (resource: databricks_volume) can be created. Schemas provide logical organisation within catalogs, while volumes store non-tabular data and files in Unity Catalog. Tabular data objects, such as tables and views, are typically created and managed as part of DAB deployments.

Module 3: Unity Catalog privileges — Assigns privileges to Unity Catalog objects.

This is likely the module whose resources will change most frequently. Separating Unity Catalog privileges from Unity Catalog object creation allows both modules to evolve independently, as access requirements typically change much more often than the underlying data structure.

Assumptions

• Unity Catalog objects exist.
• Principals are resolved in Databricks.
• Deployment CI/CD identity has the required privileges to grant permissions.

Resources

In addition to the privileges management concepts already covered in the MDT blog, we recommend using the databricks_grants resource wherever possible. Unlike individual grant resources, databricks_grants manages the complete set of privileges for a securable object and overrides privileges granted manually or outside of the current Terraform configuration. This helps ensure that the actual privileges remain aligned with the desired state defined in Terraform.

To simplify privileges management, we recommend defining Unity Catalog privileges in an easy-to-read format such as YAML and then ingesting that configuration into Terraform. This allows privilege changes to be made without modifying Terraform code and enables platform administrators to manage access without requiring deep Terraform knowledge.

Example:

catalogs:
  bronze:
    grants:
      - principal: data-engineers
        privileges:
          - USE_CATALOG
      - principal: data-analysts
        privileges:
          - USE_CATALOG

schemas:
  "bronze.sales":
    grants:
      - principal: sales-team
        privileges:
          - USE_SCHEMA
          - SELECT

Terraform can then load the file and create the corresponding databricks_grants resources dynamically.

Related to secret management, there are now two additional approaches besides the secret scopes discussed in the MDT blog:

• Using service credentials, as explained in Hubert Dudek’s blog. Access to Azure Key Vault secrets is controlled through the Unity Catalog ACCESS privilege granted on the corresponding service credential. This is the recommended approach when integrating Azure Key Vault with Databricks.
• Databricks also supports managing secrets as Unity Catalog objects (catalog.schema.secret). This enables fine-grained access control at the individual secret level and provides a Unity Catalog-native approach to secret management (see documentation).

Module 4: Databricks workspace governance — Manages workspace access, entitlements, and settings.

Once the Unity Catalog foundation is in place, we can start onboarding users and configuring workspace governance settings.

Assumptions

• Deployment CI/CD identity is a Workspace Admin.
• If certain workspace settings are managed, the deployment identity may also need to be granted Databricks Account Admin privilege.

Resources

We first add principals (users, groups, and service principals) to the Databricks workspace and assign them either user or Workspace Admin permissions. This can be achieved using the databricks_permission_assignment resource.

To better control what users can do within a workspace, the databricks_entitlements resource can be used to manage capabilities such as access to the Workspace UI, Databricks SQL, and Genie One (previously known as Databricks One). This allows different experiences to be provided for data engineers, data scientists, data analysts, and business users.

Databricks workspaces expose many configuration settings that can now be managed through the databricks_workspace_setting_v2 resource, which at the time of writing is available in Public Preview. Compared to the older databricks_workspace_conf resource, it offers several advantages:

• Settings are officially documented and supported.
• Available settings can be easily discovered and managed.
• Preview workspace settings are supported.
• Each setting is managed as a separate resource, providing better granularity and control.

More details on important Databricks workspace settings will be covered in a future blog focused on secure Databricks workspace deployments.

Module 5: Databricks workspace objects — Creates operational objects inside a Databricks workspace.

After users have been added to the workspace and their way of working has been defined in the previous module, the final ingredient required for development is providing access to compute resources.

Assumptions

• Deployment CI/CD identity is a Workspace Admin.

Resources

Although Databricks increasingly promotes serverless compute because it simplifies management and enables advanced AI capabilities, most clients still use a combination of serverless, all-purpose, and job compute.

One trend we increasingly see at clients is controlling access to compute through cluster policies (resource: databricks_cluster_policy). This approach simplifies cluster administration compared to creating dedicated clusters for individual users (resource: databricks_cluster). Job workloads should generally use job clusters governed by cluster policies, as this approach typically results in better resource utilisation and lower costs. For personas that primarily require SQL access, such as data analysts, SQL warehouses can be created (resource: databricks_sql_endpoint).

The reason we suggest creating Unity Catalog objects before workspace objects is that cluster policies can use Unity Catalog volumes to store cluster logs, making them easier to govern and share with users.

Additional information about cluster policies and compute management can be found in the previously published blog Unity Catalog migration best practices: Lessons learned from two implementations.

Compared to the MDT blog, we would highlight two additional recommendations:

1. Cluster policy definitions should be kept outside of Terraform code in JSON files. This keeps the configuration transparent for end users and easier to manage for the platform team. The only value that typically needs to be specified outside the JSON definition is the maximum number of clusters per user, which is configured directly on the cluster policy resource. The example below illustrates a development-focused cluster policy that enforces Unity Catalog security, centralised logging, and cost controls while still allowing flexibility in runtime selection and compute configuration:

{
  "cluster_type": {
    "type": "fixed",
    "value": "all-purpose"
  },
  "data_security_mode": {
    "type": "fixed",
    "value": "USER_ISOLATION",
    "hidden": true
  },
  "runtime_engine": {
    "type": "allowlist",
    "values": ["STANDARD", "PHOTON"],
    "defaultValue": "STANDARD"
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "autoscale.max_workers": {
    "type": "range",
    "maxValue": 4,
    "defaultValue": 2
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 30,
    "hidden": true
  },
  "dbus_per_hour": {
    "type": "range",
    "maxValue": 15
  },
  "custom_tags.Environment": {
    "type": "allowlist",
    "values": ["dev", "test", "prod"]
  },
  "cluster_log_conf.type": {
    "type": "fixed",
    "value": "VOLUMES",
    "hidden": true
  },
  "cluster_log_conf.path": {
    "type": "fixed",
    "value": "/Volumes/platform/observability/cluster_logs",
    "hidden": true
  }
}

2. We recommend keeping permissions on workspace objects, such as folders, cluster policies, and other workspace-level resources, within the same module where those objects are created. Unlike Unity Catalog permissions, these permissions usually change infrequently and can therefore be managed together with the underlying resources using the databricks_permissions resource.

Module 6: Databricks account — Provides account-level administration capabilities across all Databricks workspaces.

The modules discussed so far focus on Azure resources, Unity Catalog, and workspace-level configuration. However, Databricks also includes a separate account layer responsible for managing identities, workspaces, networking, and account-level governance and security. Because these resources require different privileges and are often operated by a separate platform team, they are managed in a dedicated module.

Additionally, as mentioned earlier, Databricks account-level operations may require a separate CI/CD process. In our case, private runners used for Azure and workspace deployments did not have connectivity to the Databricks account endpoint, making separate deployment pipelines a practical solution.

Assumptions

• Deployment CI/CD identity is a Databricks Account Admin.
• Databricks account endpoint is reachable from the CI/CD pipeline.
• Principals are resolved in Databricks.

Resources

In the MDT blog, several account-level resources were distributed across multiple modules. Although Databricks-managed account groups created with the databricks_group resource can be useful in certain situations, they add complexity on top of Microsoft Entra ID group management. Microsoft Entra ID-managed account groups should therefore be the default authorisation principals, while Databricks-managed groups should be reserved for explicit Databricks-only requirements.

Other account-level resources were related to Network Connectivity Configurations (NCC), which enable private connectivity between serverless compute and external resources (resources: databricks_mws_network_connectivity_config, databricks_mws_ncc_binding, databricks_mws_ncc_private_endpoint_rule).

If Databricks is deployed in a way that outbound connections from standard clusters are controlled and restricted, it is usually desirable to apply similar controls to serverless compute. Network policies (resource: databricks_account_network_policy) provide an additional layer of security by controlling outbound network access for serverless workloads. To assign a network policy to a Databricks workspace, the databricks_workspace_network_option resource can be used.

As recommended earlier, dedicated service principals should be used for Databricks workloads. To control which users or groups can use these service principals, the databricks_access_control_rule_set resource can be used.

Finally, similar to the databricks_workspace_setting_v2 resource discussed in Module 4, Databricks provides the databricks_account_setting_v2 resource for managing account-wide settings in a consistent and declarative manner. At the time of writing, this resource is in Public Preview.

Production Terraform essentials

Although these controls are not Databricks-specific, they are important for production Terraform deployments:

• Use a secure remote backend with state locking.
• Define explicit Terraform and provider version constraints.
• Commit the .terraform.lock.hcl file to source control.
• Use OIDC or workload identity federation for CI/CD authentication instead of long-lived secrets.
• Apply prevent_destroy selectively to critical resources.
• Promote reviewed and versioned modules through deployment environments.
• Run scheduled Terraform plan executions to detect configuration drift.

The proposed module structure is not intended to be a strict blueprint but rather a starting point that can be adapted to project requirements. In our experience, aligning Terraform modules with provider, ownership, lifecycle, and security boundaries simplifies CI/CD processes, improves governance, and makes Databricks platforms easier to operate as they grow.

Building a Databricks Lakehouse Platform with Terraform Modules was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why this pipeline needed more than standard batch processing

Telecom networks generate data continuously. Calls, messages, and data sessions produce millions of records per hour, every hour of the day. But in this project, volume was only half the challenge. The harder constraint was time.

On a telecom project I worked on, every event had to be fully processed within one hour of arriving in the data platform. This was not a soft target. If the pipeline missed the window, downstream systems for billing, analytics, and compliance would work with stale data. At terabyte scale, across two Databricks workspaces, that SLA became the forcing function behind almost every architectural and operational decision.

This article is not a general introduction to Databricks or the Medallion architecture. It is a practical walkthrough of the problems we hit while trying to meet that SLA, and the optimisations that made it achievable. Some of the solutions are Databricks-specific. Most of the thinking applies to any high-volume, latency-sensitive data pipeline.

If you’re building or operating a pipeline where processing latency actually matters, there’s something here for you.

Pipeline overview

The platform followed a Medallion architecture: data moved through three progressive layers (Bronze, Silver, and Gold), each adding structure, quality, and business meaning. (See figure below.)

Incoming files were continuously landing in our Azure Data Lake Storage account in zipped Protobuf format, compact and efficient for transmission, but requiring custom deserialisation before the pipeline could work with structured, queryable columns.

The pipeline was split across two dedicated production workspaces:

• Ingest workspace — handled Raw → Bronze → Silver. Auto Loader detects incoming files, after which a custom deserialisation step converts the Protobuf binary content into structured columns before appending records into the Bronze Delta table. A further process then deduplicates and merges records into Silver.
• Transformation workspace — handled Silver → Gold. Updates to the Silver table trigger a Structured Streaming job that applies business-level transformations and writes to the Gold layer.

This two-workspace separation was a deliberate architectural choice: keeping ingestion and transformation concerns isolated means a problem on one side doesn’t destabilise the other.

Beyond production, dedicated development and testing workspaces mirrored the same topology, allowing end-to-end validation before anything reached production.

How the pipeline runs:

Two things worth noting from this table:

• The ingest job ran every 30 minutes on a fixed schedule, leaving enough time for Bronze, Silver, and Gold processing to complete within the SLA window. Running once per hour wouldn’t have been enough — a record landing just after a run completed would have had to wait up to 60 minutes before being picked up by a new run, leaving no time to meet the SLA.
• Within each job, availableNow=true was used as the Structured Streaming trigger mode. Instead of running continuously, each job processed all currently available data in a single microbatch and then self-terminated.

The SLA budget is worth making explicit. Sixty minutes sounds generous until you account for: Auto Loader discovery latency, Protobuf deserialisation, Bronze append, Silver deduplication and merge, cross-workspace trigger delay, Gold microbatch processing, and cluster startup time. Every optimisation in this article targets one or more of these stages.

The breakdown above reflects typical observed timings from the production pipeline. Individual run times varied depending on data volume, but these figures represent a realistic baseline for a normal run.

Reducing file discovery latency with Auto Loader events

Why directory listing became a bottleneck

Auto Loader supports two file discovery modes: directory listing and file notification. Directory listing is the default, working by periodically scanning the source directory for new files. Simple to set up, no infrastructure dependencies. But at scale it has a serious flaw: as the number of files in the directory grows, each scan takes longer. On a high-volume pipeline where files land continuously, directory listing latency quickly eats into the SLA budget before a single record is processed.

The fix: switch to managed file events

File events mode uses cloud storage events for file discovery. Azure Databricks automatically configures Azure Event Grid and Azure Storage Queues on your behalf, and Auto Loader reads from the managed file events cache to identify newly arrived files, instead of relying on periodic directory scans. Discovery latency drops from minutes to seconds or near-instant, regardless of how many files are in the directory.

Databricks now recommends cloudFiles.useManagedFileEvents as the standard way to enable this. The code change itself is straightforward — a single option added to the stream configuration. However, the prerequisite is having file events enabled on your Unity Catalog external location, which requires coordination with your cloud and platform teams to configure the underlying storage event infrastructure.

.option("cloudFiles.useManagedFileEvents", "true")

On first run, Auto Loader performs a full directory listing to synchronise with the file events cache and establish a read position in the checkpoint. Every subsequent run reads directly from that cache, with no directory scanning and near-instant file discovery.

One step further: Tuning maxFilesPerTrigger on the Bronze side

Once file discovery is fast, the next tuning parameter to consider is maxFilesPerTrigger, which controls how many files Auto Loader processes per single run. On the Raw → Bronze Auto Loader read we set this to 10000, a deliberate cap that keeps individual microbatch size predictable.

This is a genuine tradeoff: too low and the stream becomes inefficient due to many small batches; too high and a batch occasionally grows large enough to threaten your latency budget. The right number depends on your file sizes and cluster configuration — and it requires measurement under realistic load rather than guessing.

Making MERGE INTO fast enough for the Silver layer

Why MERGE INTO became expensive at scale

The Bronze → Silver job relied on MERGE INTOto deduplicate and upsert records into the target table. At small scale, merges are fast and largely unnoticeable. At TB scale, however, they become one of the most expensive operations in the pipeline, and a significant risk to SLA compliance.

The cost comes from two places. First, Spark has to scan the target table to find matching records, and without good data organisation, that means scanning a lot of data you don’t need. Second, Delta Lake rewrites entire files even when only a few rows within them are affected by the merge. The more files touched by the merge operation, the higher the cost.

Optimisation 1: use liquid clustering on Silver

Liquid clustering (CLUSTER BY) organises data based on columns frequently used in filters, joins, or merge conditions, meaning Delta can skip large portions of the table that don’t match the query predicate rather than scanning everything. For a high-cardinality key like a unique event identifier, the scan reduction is dramatic.

Unlike traditional partitioning, liquid clustering can be added or changed on an existing table at any time via ALTER TABLE — no need to recreate it. New data immediately follows the updated clustering keys. However, existing data is not automatically reclustered. To force reclustering of all historical data you need to run OPTIMIZE FULL.

It’s also worth noting that Databricks now supports CLUSTER BY AUTO, which selects and adapts clustering keys automatically for Unity Catalog managed tables.

Optimisation 2: deduplicate before the merge

The more rows you feed into a merge, the more expensive it gets. If your incoming microbatch contains duplicates, the merge has to resolve them, adding complexity and cost to an already expensive operation. Deduplicating the incoming dataset before it hits the merge reduces the number of rows Spark has to match and write.

This also has a correctness benefit: duplicates that survive into Silver create downstream problems in Gold that are much harder to fix than catching them before the merge.

Optimisation 3: drop unneeded columns early

Every column that travels from Bronze through the merge is a column Spark has to read, compare, and potentially rewrite. Dropping columns that aren’t needed in Silver (at the Bronze → Silver boundary) means less data flows into every merge operation. Fewer columns means smaller files, faster scans, and less data rewritten.

Processing only what changed

Why full reprocessing was not an option

The easiest pipeline design is also the most expensive: read the entire dataset, run every transformation, and merge all results back into the target tables on every run. That can work at small scale. At terabyte scale, it quickly destroys the SLA budget. The real goal was not just to process faster, but to avoid processing unchanged data at all.

Optimisation: use Change Data Feed for Gold

Silver was a merge target, not an append-only table, which means rows can be inserted, updated, or deleted as new data arrives. If Silver were append-only like Bronze, the Gold job could simply process newly appended data. But because rows can change after they’ve already been written, a simple “read what’s new” approach wasn’t enough. The Gold job also needed to know when an existing row was updated or removed.

This is exactly the problem Change Data Feed (CDF) solves. Rather than reading the full Silver table on every microbatch, the Gold job read only what changed since the last run.

CDF is not enabled by default. You turn it on at the table level:

ALTER TABLE silver_table
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

Once enabled, Delta maintains a change log for every write to the table. Each record includes a _change_type column with one of four values: insert, delete, update_preimage (row before the change), or update_postimage (row after the change).

The Gold job read from this change log using readChangeFeed:

silver_changes_df = spark.readStream \
.option("readChangeFeed", "true") \
.table("silver_table")

Combined with the availableNow=True trigger, each Gold job run processed only what changed since the last run. The job fired, processed the change set, and terminated. No redundant scanning, no reprocessing records that hadn’t changed. For a full reference on CDF configuration and options, see the Databricks Change Data Feed documentation.

Removing idle wait between pipeline layers

Why scheduled handoffs added risk

Optimising individual processing stages is only part of the latency equation. Delays between layers can be just as impactful. Scheduled jobs are a prime example: when a downstream job runs on a fixed schedule, newly available data may sit idle until the next execution window. Across several pipeline stages, these waiting periods add up quickly, turning minutes of scheduling latency into a meaningful threat to end-to-end SLA compliance.

Optimisation 1: trigger Gold when Silver updates

The direct answer to unpredictable inter-layer latency is eliminating the schedule entirely and replacing it with a table update trigger. This is a native Databricks feature that automatically launches a job shortly after a source table is updated, usually within a few seconds to one minute.

To set it up, open your Databricks job’s trigger settings, select “Table update” as the trigger type, and choose the table you want to observe.

In our pipeline, the Gold job in the transformation workspace used a table update trigger watching the Silver table in the ingest workspace. Once the Silver job completed and committed its changes, the Gold job fired automatically. That removed wasted schedule wait time and reduced cross-workspace handoff latency from minutes to seconds.

For a full reference on configuring table update triggers, see the Databricks documentation.

Optimisation 2: use cluster pools to reduce startup time

One hidden cost of trigger-based execution is cluster startup time. If each run starts a fresh cluster, several minutes can be lost before any data is processed, reducing the time left within the SLA.

Cluster pools solve this by maintaining a set of pre-warmed instances ready to be claimed immediately when a job fires. The cost model is worth understanding: DBU charges are pay-per-use, only accumulating when a job is actively running, while Azure VM costs apply even when instances are idle. You are essentially trading a continuous but predictable VM cost for the elimination of startup latency.

We configured pools on both the ingest and transformation workspaces, ensuring every job across the pipeline claimed a pre-warmed instance rather than spinning up from scratch. Without pools, cluster startup alone took at least 5 minutes; with pools, that dropped to a couple of minutes. On a pipeline with a 60-minute SLA, that difference mattered.

Putting it all together

Every optimisation in this article targets a specific slice of the 60-minute SLA budget. No single optimisation was sufficient on its own. Instead, the SLA was achieved by eliminating waste at every layer and compounding small gains into a pipeline that consistently met its deadline.

Here’s a consolidated view:

One pattern stands out immediately: the majority of high-impact optimisations are also low effort to implement. Adding liquid clustering, configuring a table update trigger, and setting up cluster pools — none of these require significant engineering investment. They are configuration and design decisions more than they are development projects. A few others, like useManagedFileEvents and CDF, need a bit more setup but are still well worth the investment given their impact.

What moved the needle most: the combination of Auto Loader file events, liquid clustering on Silver, CDF on Gold, table update triggers, and cluster pools. Each of these targeted a distinct and significant source of latency in the pipeline. Everything else was incremental improvement on top of a foundation that already worked.

Conclusion

A 60-minute SLA on a TB-scale pipeline may look like a performance problem, but in practice it is an architecture problem. The biggest wins did not come from low-level Spark tuning. They came from decisions about how layers interact, what data is allowed to flow forward, and when jobs should run.

Although the specific stack here is Databricks, Delta Lake, and Azure, the thinking applies broadly. If you’re operating any high-volume, latency-sensitive pipeline, the questions are the same: is time being wasted in file discovery? Are you processing more data than necessary? Are scheduled jobs introducing idle wait that an event-based trigger could eliminate?

The SLA was a hard constraint, but it was also the best forcing function we had for building a pipeline that was fast, correct, and easy to operate at scale. Constraints, it turns out, can be useful.

Processing high-volume event data within a 60-minute SLA on Databricks was originally published in Valcon on Medium, where people are continuing the conversation by highlighting and responding to this story.