AI-assisted data engineering at Expedia Group — and less squinting at Spark execution plans

Anyone who has actually opened a Spark execution plan to figure out why a job is slow or expensive knows what the pain looks like. It’s not one screen — it’s the Spark UI’s stage DAG, then the SQL tab’s physical plan with nested Exchange, BroadcastHashJoin, and SortMergeJoin operators stacked five levels deep, then a separate executors tab to check whether any of that actually correlates with memory pressure, then cross-referencing timestamps against a job's config history to rule out "did someone change something last week." Reading a plan correctly means holding all of that in your head at once, and most of it is stored in different tabs that don't talk to each other. Doing this well takes real practice, and doing it under time pressure — a pipeline failed overnight, someone needs an answer before standup — is exactly when people take the fastest wrong guess instead of the right slow one.

This is the part of the job that AI turned out to be genuinely good at, and it’s the part of the last two years worth focusing on.

A brief origin point: Iceberg V3

The first place this paid off concretely was migrating tables to Apache Iceberg’s V3 format after it shipped in mid-2025. The work itself was conversational — reading a table’s delete-file accumulation, reading the actual scan plan against that table, deciding whether a merge-on-read migration to deletion vectors was worth it, then checking the resulting plan again to confirm it worked. Daily compute cost on a couple of pipelines that got this treatment dropped by more than 90%. That’s a good result, but the more important thing it proved was narrower: that reading a real execution plan and matching it against real table metadata, rather than trusting what the DDL or the job config claimed, was something an AI could do reliably enough to act on. (The deletion-vector mechanics themselves are covered in more depth separately: Deletion Vectors and Puffin Files: MERGE in the New V3 Iceberg Format.) That one-off, table-by-table exercise is what eventually got packaged into a repeatable skill — but the skill’s value was never really about Iceberg. It was about not having to read the plan by hand anymore.

Where this actually shows up: reading the execution plan instead of guessing from configuration

Two skills now do this reading for us — one after a job has finished, one while it’s still running — and both are built around the same premise: the plan and the execution trace tell you what happened; the config and the code only tell you what someone intended.

The cost-analysis skill reads the plan after the fact, on every expensive job. For the two or three most costly jobs in a pipeline, it pulls the actual stage summary (shuffle and spill volumes, task skew), the actual executor summary (peak memory against what was allocated), and the literal SQL execution plan the engine produced — the same tabs a human would have to cross-reference by hand, done automatically and correlated in seconds instead of the better part of an hour. That turns into diagnoses that are hard to make correctly by eye:

• Skipped stages vs. wasted stages. A stage the plan skips is a cache hit doing its job, not idle compute — easy to misread as “this stage never ran” if you’re scanning the DAG quickly, and the skill is built to recognize the difference rather than flag it as an anomaly. It’s the same judgment call discussed separately in Optimizing CTEs on Apache Spark: When to Cache and When Not To: a repeatedly-rescanned CTE is real cost worth fixing, but reflexively caching everything isn’t free either, and telling the two apart from a plan requires actually tracing which stage re-executes the same subtree.
• Near-capacity vs. already spilling. Peak executor memory at 99% of a container with zero spill and the same peak with active spill look nearly identical on a summary chart — one is a warning for next week, the other is money being spent right now. Distinguishing them means reading the actual spill metric per stage, not the aggregate memory graph.
• Full-scan MERGE INTO detection. This is the plan-reading catch I'd put ahead of the others: the skill checks whether a merge statement's target-side condition is filtered the same way its source is. A plan with an unfiltered target ON clause against a windowed source looks completely unremarkable at a glance — it's the kind of thing a human reviewer skims past — but it forces a full-table scan every run, and where a WHEN NOT MATCHED BY SOURCE clause is present, it can delete rows outside the intended window. This was caught as active, ongoing data loss on more than one pipeline during a routine cost sweep, because the skill compared the plan's actual scan boundaries against what the source windowing implied instead of trusting that the code did what its comments said. More on why this specific plan shape is so costly on Iceberg tables specifically: Boost Iceberg Performance and Cut Compute Costs with Well-Scoped MERGE Statements.
• Driver-side vs. executor-side memory failures. Recurring out-of-memory kills are the textbook trigger for “give it more executor memory” — a reasonable guess from the error message alone. Reading the actual heap trace changes the diagnosis: if executor heap usage at the moment of failure was near zero, the failure is on the driver side (in one case, file-transfer buffering), and more executor memory would have done nothing.
• JVM heap pressure vs. off-heap memory pressure. A separate pipeline kept failing with the same OOM error after someone manually doubled executor memory — because in both configurations, JVM heap usage topped out at only 11–23% of what was allocated. The memory being exhausted was Python process memory outside the JVM, invisible to anyone reading the Spark memory config and assuming that’s where the ceiling was.
• Stale tables, caught through metadata rather than assumption. Not every finding comes from a Spark plan — some come from asking the catalog itself a question a human would rarely think to ask on a routine pass: when did this table actually last get written to? Through a Trino MCP connection, the skill queries a table’s Iceberg snapshot history directly — no ad hoc SQL session, no remembering the right metadata table name — and checks the most recent commit timestamp against the analysis window. A table with no write in the last 30 days but ongoing shared maintenance cost is pure waste hiding in plain sight, invisible unless something actually checks the snapshot log instead of assuming a table is “probably fine” because nobody’s complained. The same Trino MCP path is used to reconcile a repo’s DDL against what genuinely exists in the live catalog — turning up tables that were dropped in one environment but never removed from source, or tables the catalog thinks are current but haven’t been touched in months. Once a table is confirmed genuinely stale or corrupted, the lowcode deployment skill takes over: registering it cleanly as a trash table with the right ownership tags before dropping it, or, for a table stuck in a corrupted metadata state, running the unregister-and-re-register recovery sequence — again pulling the current state through the same metadata queries rather than guessing at what’s safe to touch.

Reading a plan well enough to catch any one of these by eye is a real, learned skill — the kind senior engineers build over years of exactly this kind of incident. Having it done automatically, correctly, and consistently on every top-cost job is what turned troubleshooting from “hope the person on call has seen this exact failure before” into a repeatable ten-minute pass. The limits get documented rather than papered over: a pre-signed dashboard URL that’s already been used needs a fresh request, and a job older than roughly 30 days has no server-side trace left to read at all — a hard platform limit, not something worth retrying around.

The lowcode deployment skill reads the plan while the job is still running — and, increasingly, before it’s even committed. Rather than deploying straight to production and waiting for a downstream alert to say something went wrong, it can run the same job as a live dry run first: executing the actual statement against a non-production table or branch (the same Write-Audit-Publish pattern behind Iceberg’s branch-based validation — see Chill Your Data With Iceberg: Write-Audit-Publish), watching the real plan and stage progress it produces, and confirming the change behaves as expected before anything touches production data. For a risky operation — an Iceberg format upgrade, a DDL change with ownership implications — that dry run is what turns “we think this is safe” into “we watched it run safely.” Once a deployment does go to production, the skill keeps watching: it polls the running job’s stage progress every ten seconds or so for the duration of the deployment, the same live equivalent of the plan a human would otherwise only get to inspect after the fact. That’s the difference between discovering a failure an hour later and starting a fresh debugging session, versus watching it fail in the same session it was deployed in and fixing it before moving on. The same live view — from the dry run or the real deployment — is what lets it pick the right recovery procedure — a corrupted table metadata state, a write that committed conflicting deletion vectors, a stale table confirmed through the catalog checks above — from the actual job trace rather than guessing from an error message.

What actually changed

Not the model getting smarter — the willingness to go read the plan instead of guessing from the config, every single time, on every job that matters, instead of only when someone finally has an hour free to dig in properly. That’s the same instinct that started with one engineer squinting at a scan plan to decide whether an Iceberg table was worth migrating, now running automatically across a whole portfolio of pipelines — querying the catalog directly through Trino MCP, dry-running risky changes before they touch production, reading real execution traces, and fixing what it finds rather than just reporting it. The dollar savings and the incident catches are both downstream of that one habit, not the point of it.

Related reading:

• Deletion Vectors and Puffin Files: MERGE in the New V3 Iceberg Format
• Optimizing CTEs on Apache Spark: When to Cache and When Not To
• Boost Iceberg Performance and Cut Compute Costs with Well-Scoped MERGE Statements
• Chill Your Data With Iceberg: Write-Audit-Publish

How I Taught an AI to Read My Logs was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How Expedia Group compressed an estimated two years of iOS modularization into about three months while shipping weekly

The first wave of coding assistants trained us to think in prompts: describe a task, generate code, review the result.

Reusable skills made that interaction more repeatable. They packaged instructions, conventions, and tools so an agent could perform a familiar task more consistently.

The hard part of large-scale engineering sits between code changes: hidden dependencies, incomplete context, build failures, test regressions, architectural decisions, ownership boundaries, and a codebase that keeps moving while the work is underway.

Generating a change is only the first step. The system still has to carry progress across hours of execution, repeated failures, changing state, and human decision points.

For that, we need a Loop.

A prompt provides direction. A skill packages reusable capability. A sub-agent provides bounded execution and context isolation. The Loop is the operating model.

At Expedia Group™, that distinction became the foundation for a major modularization program inside our Shared iOS Components Stack. Work initially estimated at roughly two years was completed in about three months. We did it without a code freeze, and our Monday release cadence stayed intact.

In one complex-domain migration, the loop ran autonomously for more than eight hours before requesting its first human architectural decision.

We got there by treating coding agents as one component of a deliberately engineered system.

Three engineering north stars shaped the program:

• Prepare workspaces for the agentic era: create smaller, faster, isolated workspaces with clear ownership and dependency boundaries that can compile independently
• Shorten every feedback loop: let developers and CI build and test the smallest meaningful piece
• Improve developer flow: turn faster build cycles into less coding time, less waiting, and faster delivery

The goal: transform the architecture without stopping delivery

An important distinction: our iOS repository is a monorepo, but it is not a monolith. It uses a layered, modular, white-label architecture designed for delivery at scale. Its shared foundation supports more than ten Expedia Group brands, with many domain teams contributing concurrently.

This program focused on one of its largest and most interconnected surfaces: the Shared iOS Components Stack. Teams build domain-specific experience components and logic here, then compose them into screens and end-to-end flows across brands. The stack is large, heavily reused, and always changing.

In this context, a domain is a group of packages that exposes multiple related components and experiences.

The target for each domain was explicit:

• Build, run, and test in a focused domain component workspace app
• Depend only on what the domain requires
• Expose clean protocol-based seams to its consumers
• Remain composable across existing screens, flows, and brands

Before the migration, domain folders lived behind one Shared Components Stack package boundary. Direct cross-domain imports were easy, but focused compilation, testing, and automated reasoning were harder. The stack had a focused all-component app for component validation, but it still compiled every domain.

The target state gave each domain an implementation package and a small API package for protocols, state, and dependency-injection entry points. Application packages could consume both. Domains could depend on one another through API packages, breaking compile-time cycles while keeping implementations decoupled.

Each domain team could instead compile and run a focused domain component workspace app containing only its components and required dependencies. That made component validation much faster locally and in CI.

The migration contract mattered as much as the end state. There could be no code freeze, Monday releases had to continue, and product work could not stop.

This involved much more than moving files. Each migration had to expose implicit dependencies, restructure implementation code, update generated-code infrastructure, and prove compatibility at progressively wider boundaries.

Before this approach, a medium-to-complex domain could require approximately four months of incremental work. Faster typing by an engineer or an agent addressed only a small part of the problem. The end-to-end flow of work needed a redesign.

Find the loop before designing the automation

“Modularize this domain” is a goal, not an executable plan.

We began by studying successful migrations and identifying the sequence experienced engineers repeated. Beneath the domain-specific details was a stable control loop:

Analyze → Decide → Act → Verify → Memory → Route → Repeat

• Analyze: collect facts about the domain, its size, complexity, dependencies, consumers, tests, and generated code
• Decide: select the next transformation using known patterns or fresh reasoning
• Act: change package structure, seams, sources, dependencies, and generated-code configuration
• Verify: compile and test at progressively wider boundaries
• Memory: record failure signatures and successful resolutions
• Route: continue automatically, invoke reasoning, or request a human architectural decision
• Repeat: use the new verified state as the starting point for the next cycle

This loop became the design center for the system. Scripts, agent skills, orchestration, builds, tests, memory, and human review all existed to serve one or more stages of it.

We stopped asking how to write a better modularization prompt. Instead, we asked how to build a system that could keep modularizing safely after the initial prompt had disappeared from view.

Design the system around the loop

At repository scale, sending every operation through a model would have been slow, expensive, and unnecessarily variable. We separated deterministic work from judgment and connected the two through a purpose-built Model Context Protocol (MCP) server.

Set the modularization goal before the loop starts

For each domain, the Orchestrator Entry Skill set a bounded goal:

• The target domain and desired package and API seams
• The infrastructure transformations the workflow could perform
• The ordered build and test boundaries required for completion
• The cross-domain decisions that required human architectural direction

The Main Agent could choose how to reach the next verified state, but it could not redefine the target architecture or bypass a verification gate.

Analyze with deterministic tooling

Through the Modularization MCP, deterministic scripts inventoried the repository, mapped dependencies in both directions, measured complexity, and ranked migration order. The analyzer worked at the domain or portfolio level and wrote a structured findings document. No model reasoning was needed, and the agent did not have to rediscover the repository on every turn.

Execute through the Modularization MCP

The Orchestrator Entry Skill accepted the modularization request, established guardrails, and configured the Main Agent. The Main Agent defined the loop’s semantics and made decisions from the structured plan and MCP state, including when to continue or exit.

Guided by the analyzer-driven plan, the Modularization MCP checked and persisted state, managed tasks and completeness, created package infrastructure and protocol seams, moved sources, updated dependencies and code generation, and drove compilation and testing. After each build, it queried Xcode’s index for precise symbols and references instead of using broad text searches.

The index gave the Main Agent smaller, more accurate context when reasoning was required. The skill made the workflow discoverable, the MCP made it durable and token-aware, and the Main Agent’s loop provided autonomy.

Verify against executable truth

Build and test evidence, rather than the Main Agent’s judgment alone, determined completion.

Completion came from a hierarchy of builds and tests:

1. Build the focused domain component workspace app with only that domain and its dependencies.
2. Run the domain’s tests until they pass.
3. Compile and test the legacy workspace against the domain’s new public seams.
4. Wire the modularized domain into the main application.
5. Build and test the shippable application.
6. Prepare the complete change for review as one coherent pull request.

Each green boundary expanded the part of the system we knew was safe. Each failure returned control to the loop.

Builds and tests provided feedback throughout the workflow. Their results supplied the evidence the Main Agent used to choose the next state.

Use memory before reasoning again

When a build failed, the Main Agent classified the error while the MCP checked persistent memory for a matching pattern. If an earlier domain had produced the same failure, the MCP applied the recorded resolution and compiled again.

Only unfamiliar failures required fresh reasoning. The Main Agent examined the dependency and ranked possible resolutions by architectural fit and blast radius. It selected the best option and returned the workflow to verification.

Memory made later migrations cheaper. Each verified fix became available to later runs, which reduced cost and uncertainty as the work moved across domains.

Treat token cost as a system constraint

Efficiency was part of the architecture. We reduced unnecessary model work in five ways:

• The MCP handled deterministic analysis, infrastructure changes, compilation, and testing.
• Structured findings and Xcode index lookups supplied precise facts and symbols without broad repository searches.
• Focused domain workspaces bounded both source context and build feedback.
• Memory reused verified fixes before requesting new reasoning.
• Main Agent routing reserved deeper reasoning and human attention for genuinely new or high-impact decisions.

We did not optimize for the fewest possible model calls. We optimized for the least expensive safe path to the next verified state.

Put humans at decision boundaries, not inside every step

Human involvement remained essential, but its placement changed.

Engineers no longer needed to supervise routine file movement, known build errors, or repeated compile-and-test cycles. The Main Agent routed work to a person only when it discovered a consequential architectural fork. One example was a deep cross-domain dependency with several credible resolutions and a large potential blast radius.

At that point, the workflow paused with evidence: the dependency path, the affected domains, the failure, and ranked options. An engineer provided architectural direction, and the Main Agent resumed from the state preserved by the MCP.

One especially large and complex domain ran through the workflow for more than eight hours before requiring its first human decision.

Useful autonomy keeps work moving safely within clear decision boundaries. It is measured by how far the system can progress between decisions that genuinely require human judgment.

The outcome: months became days

The results moved together across program delivery, domain delivery, build feedback, CI, and developer flow.

A medium-to-complex domain moved from about four months to under three days. Focused domain builds fell from about 15 to 2 minutes, while CI build and test time fell from 45 to 11 minutes. P75 cycle time moved from about 5 to 2 days, and P75 coding time from about 4 to 1 day.

The architecture changed as well:

• Domains became independently executable and testable.
• Dependency boundaries became enforced by builds rather than described only in documents.
• The continuous integration system could perform more selective work.
• Smaller build and test surfaces improved reliability and developer feedback.
• Product development and weekly releases continued throughout the transformation.

The migration finished faster, and it left behind a more modular architecture.

Why the loop scales where prompts do not

Prompts are temporary. They depend on the context available at one moment.

Skills last longer and usually describe how to perform a task. State management, verification, recovery, memory, cost controls, and escalation boundaries sit outside them.

A production engineering loop brings those pieces together.

It scales because it has:

• State: it knows what has completed, what failed, and what must happen next
• Evidence: builds, tests, and dependency analysis determine progress
• Memory: successful resolutions become reusable operational knowledge
• Economics: deterministic programs perform deterministic work, reserving model reasoning for ambiguity
• Boundaries: rules define what may proceed automatically and what requires human judgment
• Recovery: failure is an expected input to the next iteration, not the end of the run

This changes the model’s role. The Main Agent does not simply receive a task and generate a change. It defines and follows explicit loop semantics, while the MCP supplies durable state, deterministic execution, and evidence. Reasoning is applied only where it has the highest value.

Lessons for organizations operating at scale

Six principles from this work apply beyond modularization or mobile development:

1. Start with the human success pattern. Before automating, understand how experts recognize completion, including the evidence, exceptions, and escalation points.
2. Make success executable. Compilation, tests, dependency rules, and generated artifacts give the loop a machine-verifiable definition of done.
3. Separate certainty from ambiguity. Put inventory, graph traversal, validation, and pattern matching in deterministic tools. Reserve model reasoning for multiple plausible solutions.
4. Design memory as part of the product. Chat history is not operational memory. Persist failure signatures, resolutions, context, and verification outcomes in a form the next run can query.
5. Engineer the escalation path. Trigger human involvement when a decision requires authority. Supply the evidence and preserve the state needed to resume.
6. Optimize the whole system. Model speed is only one variable. At organizational scale, optimize elapsed time, token consumption, human attention, recovery cost, release safety, and architectural quality together.

The next era is about stateful loops

Prompts provide direction. Skills package capability. Sub-agents provide bounded execution. Autonomy becomes reliable when the Main Agent defines explicit loop semantics and the MCP preserves the state needed to run the loop.

The loop provides motion. The state machine provides control. The MCP tracks progress and preserves memory. The Main Agent uses that evidence to repeat, advance, route, or escalate.

Once that architecture is explicit, a coding agent can serve as both controller and reasoning component inside a larger engineering system. The system can run for hours, improve across projects, and make progress while engineers retain control of architectural decisions.

The question for engineering organizations is no longer, “What prompt should we write?”

It is, “What stateful loop should we engineer?”

Learn about life at Expedia Group

Beyond the Prompt: Engineering the Loop for Software Change at Scale was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building a command-line interface to standardize machine learning model delivery at Expedia Group

At Expedia Group™, machine learning (ML) teams design, build, and deploy models that use optimization techniques and artificial intelligence to help travelers find the best option for every trip.

In practice, a significant share of time goes to everything around models: continuous integration and continuous delivery (CI/CD) pipelines, automated retraining, inference setup, access permissions, environment targeting, and cluster and resource sizing on Databricks and Kubernetes. These tasks can’t be skipped, and every moment a model waits on them is time that model isn’t helping travelers. This phenomenon isn’t specific to Expedia Group; it shows up in any growing organization that runs several ML teams.

Inside these ML teams, ML scientists, on the one hand, want to iterate on optimization techniques, features, architectures, and training strategies. ML engineers, on the other hand, want to work on novel engineering initiatives, challenging problems, and partner closely with the scientists they support to serve ML models at scale. This organisation and split, although easy to define, is not perfect. Often, teams spend their time running repetitive, ad-hoc deployment work one model at a time. Both groups end up doing the same setup work again and again, on every new model.

This repetitive setup work is called the quiet tax, and eventually, it becomes a structural cost. Without standardisation, the engineering effort required to deploy and maintain N models grows roughly linearly with N. In other words, every new model brings its own workflow setup, environment alignment, debugging cycle, among other efforts and tasks.

This article describes a problem space many ML teams will recognise, the trade-offs in solving it, and one pattern that works well: a command-line interface (CLI) that standardises the setup work that is the same for every model, while leaving the model itself, i.e., code, dependencies, and framework, fully in the hands of the scientist.

A familiar but taxed workflow

To make the problem concrete, it helps to walk through a workflow that will be familiar to most ML practitioners. The exact shape varies by team, by model, by personal preference and seniority; however, the general pattern is broadly familiar across the industry.

An ML scientist typically starts in a notebook environment, working interactively against a cluster configured ad hoc for their use case. They load data, prototype features, train models, and iterate until they have something worth promoting. Up to this point, the experience is fast and interactive, which is exactly what notebook development is good at.

The friction begins on the way to production. The working notebook has to be migrated into a version-controlled repository, restructured into importable code, and wired to run outside the interactive cluster. Cluster configurations drift between environments, so settings that worked in development do not always work in staging or production. The list of things that need to be aligned is long and rarely documented in one place. For instance, identity and access management (IAM) roles, service accounts, secrets, and table permissions.

On top of that, the ML scientist has to set up a workflow orchestrator for training and inference, and decide how it will be scheduled. For models that require hyperparameter optimisation, there is the additional problem of running the search in parallel across multiple executors. This, for example, is something most platforms support, but that teams rarely set up because the implementation effort is not worth it for a single model. None of these tasks are conceptually hard. However, their context must be reconstructed from memory, institutional knowledge, or another model’s repository, every time.

The outcome is predictable: a gap of days to weeks between “the model works in the notebook” and “the model is running reliably in production”, with most of that gap spent on setup work rather than on the model. And that gap is paid twice. Once in engineering time, and again in deferred impact: a finished model that is not yet in production is delivering nothing.

Three common classes of failure

Without a standardised setup, the same categories of problem recur across models, more often quietly, but sometimes loudly. Any one of these is manageable. When all three happen at once, across, for example, thirty models, it is a tax.

Silent failures. The dangerous ones. For instance, version mismatches in ML frameworks or dependencies could silently corrupt predictions. Another example: Spark configuration drifting between development and production, altering data processing behaviour in ways that passed initial testing.

Visible failures. The time-consuming ones. These failures are loud, which makes them tractable; however, the cumulative cost of debugging them, model by model, is significant. For instance, runtime errors from forgotten version pins; authentication failures from misconfigured IAM roles or service accounts; or incompatible package combinations.

Operational inefficiencies. The slow-burn ones. These cause friction for both new and experienced ML teams, where the friction can slow down processes for weeks at times. For instance, configuration drift between models without a single source of truth; knowledge silos where certain engineers understand specific deployment patterns; or manual approval bottlenecks for routine tasks.

The tension to resolve: standardisation against flexibility

The naive solution of “everyone uses the same template” constrains legitimate scientific variation. ML scientists work in different ML frameworks, including TensorFlow, PyTorch, and custom in-house frameworks. Business logic also varies significantly across use cases. Compute profiles differ from one model to the next. A rigid template collapses that variation into a lowest common denominator, which is not a victory. A more useful question is: which components should be standardised, and which should remain flexible?

The components that should remain flexible are the ones where variation is the source of value: the model code, framework choice, dependencies, business logic, compute requirements. The components that should be standardised are the ones where variation is only a source of drift: CI/CD pipelines, development quality standards, testing frameworks, deployment configurations, and governance integration. Put differently, the differentiator is the model, not the deployment pipeline.

A command-line interface, not a platform

A pattern that works well at this scale is a CLI that scaffolds and manages production-ready ML project structures. The syntax of the CLI is deliberately uniform: every command follows the shape <resource> <operation>, where the operation is one of create, update, validate, or delete. A scientist starting a new model runs two commands to get a working setup generated for them: one to bootstrap the repository, one to add a model to it.

Drawing the line: project versus model

The CLI is built around a deliberate split between the business domain a repository serves (the project) and the individual ML model inside it (the model).

A project is a repository scoped to a business domain. It owns the shared infrastructure, CI/CD, and environment configuration. It is expected to hold multiple models over its lifetime.

A model lives inside a project. It owns its own config, dependencies, notebooks, and orchestration workflows, and inherits the project’s shared infrastructure.

This split matters for two reasons. First, it prevents repo sprawl; i.e., adding a new model to an existing domain is strictly preferred over creating a new project. Second, it simplifies the organization of business domains, i.e., a project is the single place where all the models and configurations of a business domain are located.

The rule that follows: a new project is justified only when a new model belongs to a genuinely different business domain than anything in any existing project.

Resources

Everything the CLI manages is modelled as a resource. Some resources are scoped to a model, some to the project, and most are created by default; a few are optional and only added when a team opts in. A representative catalogue covers:

• Project and model containers: the top-level scoping primitives.
• Training and inference entry points: generated notebook templates, source modules for testable code, and bindings to a workflow orchestrator.
• Workflow files: auto-generated orchestration definitions for each model, regenerated whenever config or resources change.
• Model registry integration: uploading and loading model artefacts from a model store.
• Experiment tracking integration: built-in hooks into a tracking server.
• Table writer utilities: a standardised path for persisting outputs with consistent schemas and governance defaults.
• Notebook bootstrap utilities: so notebooks behave consistently across clusters and environments.
• Config schema and interface contracts: the loaders and contracts that make model code pluggable.
• Validation step: automated acceptance checks (metric thresholds, schema checks, distribution drift) between training and inference that fail the workflow when the model regresses.
• Human-in-the-loop gate: a configurable gate baked into training and inference that pauses execution before persistence, notifies a reviewer with the run artefacts, and only saves the result on approval.
• Hyperparameter search: a standardised parallel search harness wired to the experiment tracker.

Each resource is documented individually, and the list grows over time. Crucially, every new resource inherits the same lifecycle: create, update, validate, delete. ML scientists do not have to learn a new set of commands every time the platform gains a capability.

Two non-obvious design choices

Shared infrastructure code is copied into each project, not hidden behind a package. The alternative, i.e., publishing shared code as an internal dependency, would make updates trivial. The heavier path is the right one at small-to-medium scale because visibility matters more than update ergonomics. When an ML scientist hits an error in shared code, they can read it, add print statements, and debug it themselves. Stack traces point to local files, which shortens the feedback loop and reduces the support burden on engineers. Past a certain scale, the trade-off inverts and a published package becomes the right answer; until then, visibility wins.

The CLI retains ownership of infrastructure files after scaffolding. Running update pulls the latest templates for managed files while respecting local changes to user-owned files. This means the CLI is not a one-shot code generator; it is the durable owner of the undifferentiated parts of every project, with a clear boundary around the parts the team owns.

The result is that standardisation applies where it should, and flexibility survives where it should.

What comes next

The resource model is explicitly open-ended. Several capability classes sit naturally on the roadmap of any tool of this shape:

• Real-time model deployments: Some models need to run as microservices behind an HTTP or gRPC endpoint rather than as scheduled batch workloads. Built-in support for serving frameworks (KServe and similar) is a natural extension for teams moving in that direction.
• Multi-model orchestration: Production-grade real-time ML pipelines rarely runs a single model in isolation. They typically run an orchestrator that routes traffic across multiple models, run experiments, and call feature stores. The CLI can add project-scoped resources to manage orchestrator configuration alongside the models it serves.
• Hooks for downstream monitoring: The CLI itself does not need to manage monitoring dashboards, but there are upstream prerequisites worth standardising: consistent training-baseline tagging, run-id propagation, schema fingerprints, among others. Monitoring tools can consume them without bespoke per-model wiring. Getting these hooks right makes monitoring setup a configuration task rather than an integration project.
• Feature cataloguing: A standard format for declaring the features a model consumes, alongside lineage and ownership metadata, would make cross-model reuse practical rather than aspirational.
• Automated profiling and bottleneck identification: Model training and inference code regularly hides performance problems that only surface in production, slow data loaders, unnecessary collects, inefficient joins, memory blow-ups. Standardised profiling hooks that run alongside training and inference, surface the hot paths, and flag likely bottlenecks would turn performance work into a routine output of every run rather than an ad-hoc investigation triggered by an incident.

The pattern across all of these is the same one that drives the tool in the first place: identify a capability that every team would benefit from, standardise the setup, and leave the modelling to the scientist.

What changes when this works

The change is best described in terms of the people the tool serves.

For the ML scientist: initialisation in minutes, not days. A scientist starts a new model by running two commands, gets a production-ready project, and is iterating on the model itself within minutes. Production setup is no longer a multi-day project that competes with modelling time; it is a small step that happens once and is then taken for granted.

For the new joiner: one structure to learn, not many. Every project the team owns shares the same skeleton. A new team member who learns one project effectively understands all of them. Onboarding stops being a survey of historical decisions across many repositories and becomes the much shorter task of learning a single, current standard.

For everyone: silent drift goes away. Because the code running during development is generated from the same source of truth as the code running in deployment, configuration drift between environments stops being a recurring class of bug. The “silent failure” category — where dev and prod look the same to the eye but behave differently in subtle ways — shrinks materially.

The compound advantage

The first-order benefits are what sell the tool. The second-order effects are what make it pay back over years.

• Maintenance compounds: A bug discovered in shared infrastructure is fixed once, in the CLI, and propagates to every project on the next update. Without the CLI, the same fix is repeated by hand across N repositories — and missed in some of them.
• Improvements compound: A new capability becomes available to every model the moment it ships in the CLI. The fixed cost of building the capability is amortised across the entire portfolio.
• Reasoning compounds: A reviewer, an on-call engineer, or a new joiner who looks at one project recognises the patterns in all of them. Operational reasoning is no longer a per-project skill; it is a team-wide one.
• Scaling compounds: Adding a model no longer adds proportional engineering load.

The hypothesis worth betting on is that these second-order effects matter more than the first. If the tool works as intended, every new model joins a standardised baseline instead of starting from a blank page; every CLI update propagates across the portfolio at once; and every engineering improvement lands on every model rather than just one. That is the outcome the investment is for, and the reason standardisation compounds over time.

Final thoughts

The broader principle is worth extracting. A template is a one-shot starting point; a CLI is a durable tool that continues to own the infrastructure it generates. Every fix, every new capability, every updated default propagates across the portfolio on the next update. Maintenance and improvements compound.

The approach assumes a bounded number of projects, on the order of ten, not hundreds. At larger scale, the shared-code visibility argument inverts, and the right answer becomes a published package. Knowing that the architecture can evolve in that direction is part of the case for starting with a CLI.

For any team carrying the quiet tax described above, the useful exercise is simple: look at your ML delivery pipeline and separate what is genuinely differentiated from what is undifferentiated plumbing. The differentiated work is why your team exists. The plumbing is the tax, and it does not have to be paid model by model.

Acknowledgements

Thanks to the Expedia Group engineers and scientists who provided their feedback, code, reviews, and design input to the CLI. Special thanks to our manager, Marco Rodrigues, for backing the initiative, and to Fernando B. Pérez Maurera, who will lead the project going forward.

Edited by Anantharaman Janakiraman from Expedia Group Technology editorial team

Learn about life at Expedia Group

The Quiet Tax on Machine Learning Teams was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How Expedia Group is getting organizationally and operationally ready for agents

At Expedia Group™, agents are showing up across the business: traveler-facing trip planners, partner tools, internal copilots, log readers, and a growing number of workflows that make teams say, “We tried this, and it was surprisingly useful.”

That energy is exactly what AI should unlock. But without shared foundations, it can also create fragmented technology choices, partial governance stories, and promising experiments that never become durable capabilities.

The question is not how to stop teams from building agents. It is how to let teams explore locally while building toward shared standards, reusable patterns, clear accountability, and a coherent experience for travelers and employees.

At Expedia Group, the answer is a connected operating model: communication and culture, platform and tooling, governance, and enforcement and recommendations. Together, these layers help us move quickly without becoming fragile.

What “agentic development” means at Expedia Group

“Agentic” is a term that can quickly become too broad to be useful. We should be precise.

Deterministic workflows, AI-assisted workflows, and agents all have a place in our technology landscape. A deterministic workflow follows explicit, repeatable rules. An AI-assisted workflow may call a model within an otherwise scripted process. Both can be valuable, reliable, and often the right solution for a defined problem.

But neither should automatically be called agentic.

At Expedia Group, an agent is an AI-powered system that works toward a goal. It can use context and memory, draw on skills, reason across multiple steps, and use tools to gather information or make progress. Some tasks will be long-running: they require the system to maintain relevant context, adapt to changing conditions, and work through a sequence of decisions rather than execute a fixed path.

This definition is not about giving systems unlimited autonomy. It is about deliberately designing the capabilities that make an agent useful: a clear goal, bounded memory, well-defined skills, orchestration, secure tool access, evaluation, and observability. The more capable the system, the more intentional its design needs to be.

The practical principle is simple: choose the least complex approach that can deliver the outcome — but use true agentic systems when a goal requires reasoning, context, tool use, and adaptive multi-step work.

From specialized agents to a whole-trip experience

We do not believe the future is one monolithic agent that attempts to do everything. The future of travel AI is a composed, multi-agent system.

Specialized agents can be excellent at discrete jobs: understanding traveler intent, helping with discovery, comparing properties, building an itinerary, answering questions about booking, or supporting a traveler when plans change. Each should deliver value on its own. Over time, orchestrating those specialized agents can create a continuous, end-to-end whole-trip experience.

This is both an ambitious product vision and a practical engineering approach. Narrowly focused sub-agents are easier to test through robust evaluations, easier to secure, and easier to improve. Their focus helps teams understand failure modes, set task-specific quality bars, and ensure agents do not go off the rails. That focus does not make them deterministic. They still reason, use context, and adapt; they simply have a clear purpose and well-designed boundaries.

The orchestration layer then becomes a product and systems-design challenge. How do specialized agents share the right context and memory? How do they hand off cleanly? How do they follow common principles for tone, transparency, privacy, and traveler control? The traveler should not have to understand the architecture. They should simply experience Expedia Group as a trusted companion that understands their trip and remains helpful when plans change.

Evals are the new PRD

For agentic systems, evaluations cannot be a final compliance check after the product is designed. They are how we encode what the product is meant to do: the same role a Product Requirements Document (PRD) has traditionally played.

That means shifting evals left. Before building, teams should define the traveler or business outcome, the behaviors they expect, the failure modes they will not accept, the security and privacy requirements, and the risk-based release criteria. Those expectations belong in the product design itself.

This is the core idea behind “evals are the new PRD.” A strong evaluation suite turns expectations into something we can test, debate, monitor, and improve. It creates a shared language for quality and makes a faster feedback loop possible: production signals should feed back into the evaluation suite, which helps teams identify, fix, and prevent issues.

This is how we build velocity rather than merely speed. Teams need room to experiment, but they also need the shared mechanisms to learn quickly, reuse what works, and converge with confidence.

For consequential traveler actions, the quality bar should be appropriately high. Agents can inform, compare, recommend, and help a traveler decide. The traveler should retain the final click for decisions such as completing a booking. That is a trust principle as much as a product one.

Delivering that standard consistently cannot depend on individual teams solving for quality, safety, and scale on their own. It requires an operating model that turns these principles into shared capabilities and everyday development practices.

The operating model

1. Communication and culture: creating shared visibility before standardizing

The first layer of readiness is not a platform or a policy. It is the way people share what they are doing.

Agentic work at Expedia Group is intentionally distributed. Different organizations face different problems, so experiments should not all look the same. The important pattern is what happens next: teams fan out to learn locally, then fan back in by turning useful discoveries into shared guidance and assets.

That connective tissue includes:

• AI Champion groups embedded in teams to support adoption, repository readiness, and local sharing of tools and agents.
• Cross-team guilds, including the Expedia Group Agentic Leads Guild, where senior practitioners compare patterns, surface failure modes, and develop standards that can travel across organizations.
• Engagement forums — AI Slack channels, office hours, Show and Tell sessions, and events such as the Expedia Group Intelligence Conference — where teams share both the business story and the implementation story.
• Organization and company hubs that bring together tools, patterns, and channels so teams can find existing work before creating another version from scratch.

The goal is more “we adapted another team’s pattern” and fewer surprises where multiple groups solve the same problem in incompatible ways. Internal AI adoption is not separate from the traveler vision; it is how we build the speed, creativity, and operating muscle to deliver it.

2. Platform and tooling: making the right path reusable

Teams should not have to rebuild the foundations for every use case. Expedia Group is building a common GenAI and agentic fabric that provides governed, observable access to models; shared evaluation and experiment tooling; tool and Model Context Protocol (MCP) governance; and integration with logging, alerting, and incident-management systems.

The platform must also support model optionality. The model landscape is changing too quickly for any one model, vendor, architecture, or cost curve to remain optimal for every task. Optionality is not abstraction for its own sake. It is the ability to evaluate, route, and change models based on quality, latency, cost, risk, and the needs of the experience.

The same is true of the assets that make travel AI valuable: trusted supply, rich and current content, marketplace signals, first-party traveler understanding, and accountable servicing. Generic AI can generate generic travel answers. Expedia Group’s opportunity is to build agents that are more accurate, current, useful, and grounded in the real traveler experience.

Platform investment should turn local experimentation into shared capability. It is how the work of one team can make every subsequent team faster.

3. Governance: making safe development easier to follow

Governance starts with Responsible AI and is reinforced by security standards for agents and tools. Every AI initiative that goes live, including agents, should be registered and risk rated.

Governance needs to correlate to risk. A lightweight internal helper and a traveler-facing trip-planning agent do not carry the same consequences, so they should not face the same path. Lower-risk use cases should move quickly with a lighter process; higher-risk traveler or partner capabilities should receive more scrutiny.

Shared standards should make expectations clear around:

• Ownership and accountability.
• Approved access paths to models and data.
• Agent identity and permissions.
• Logging, traceability, and observability.
• How systems behave when they cross a defined boundary.

The goal is not to bolt requirements on at the end. It is to design them into the tools and workflows teams already use, so safe development is the easiest path — not a separate document or a late-stage obstacle.

4. Enforcement and recommendations: turning principles into practical adoption

Our ML/AI Model Principles set the common standard for how we build across Expedia Group. They apply equally to classical machine learning, generative AI, and agentic systems: pursue measurable business impact and long-term marketplace health; earn complexity against strong baselines; evaluate both offline and online; build on shared foundations; make systems traceable; govern proportionally to risk; and monitor continuously in production. These principles have also been shared publicly because they are not abstract aspirations — they reflect how we intend to build reliable AI at scale.

Agent Release Tollgates are the primary mechanism for turning those principles into action for agent development. They are not a final checklist at the end of a project. They are progressive, risk-tiered checkpoints that complement the Software Development Life Cycle (SDLC) and shift agent-specific practices — evaluation, operational readiness, security, cost effectiveness, and observability — left into the development process. The goal is to anticipate issues early, so teams can move faster once they are on the road to production.

We are also automating this process as part of the Agentic SDLC. Tollgates can already be created and tracked as work items through Jira automation, and the broader direction is to automate their execution with reusable checks, evaluation assets, metrics, and dashboards rather than ask each team to build its own bespoke pipeline. That is essential if these standards are to scale across agents, skills, tools, and multi-agent systems.

The paved road includes:

• Agent Release Tollgates that connect Responsible AI risk classification to the requirements that are recommended or required, and make readiness visible throughout development rather than only at launch.
• Standard implementation patterns in agent-development handbooks and guidance, including how to structure tools, avoid burying business logic in prompt spaghetti, and keep behavior observable.
• AI-ready repositories developed with Developer Experience and AI Champions, so documentation, configuration, and automated checks live in the repository rather than in someone’s inbox.
• Shared observability patterns that make agent behavior inspectable through standard tracing, logging, and dashboards rather than bespoke tooling that only one person can operate.

The paved road should be opinionated, but not restrictive. It does not close every other path; it makes the common path smoother, better supported, and easier to operate. In practice, it asks focused questions throughout development: Are we evaluating the behavior that matters? Is anyone watching this? Do we understand cost and latency? If it misbehaves, who owns the response?

Measuring impact, not novelty

Agentic approaches are already being applied to Expedia Group’s operations. For example, Developer Experience (DevEx) is investing in reusable agents that reduce recurring “keep the lights on” work across many repositories — not just a single product team.

But a plausible response is not enough. Agents should be evaluated on whether they complete tasks correctly, reduce manual effort or cycle time, are adopted and relied on by intended users, operate within sensible cost and latency bounds, and improve a real traveler or business metric.

A clever agent that nobody uses — or that costs more to run than the work it replaces — is a useful experiment, not a template to scale.

How this changes the enterprise

Taken together, these layers add up to more than tools and documents. They represent an operating-model shift.

Agents are becoming part of how Expedia Group thinks about how work moves through the organization, how decisions are supported, how knowledge is captured and reused, and how responsibility is shared across domains and teams. Experiments are encouraged, and so is documenting what was learned. Local innovation is valued, and so is connecting successful work back into shared rails and standards.

The story of agentic AI at Expedia Group is not simply that there will be more agents. It is that there will be more structure underneath them: a clear definition of what an agent is, specialized agents that can become a coherent whole, evaluations designed in from the start, and shared foundations that help the company innovate with speed, direction, and trust.

Learn about life at Expedia Group

Building the Operating Model for Agentic AI at Expedia Group was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Missing context used to slow a team down, with agents in the loop it speeds up the wrong work instead

When we started using agents more seriously inside the team, the first bottleneck was not code generation —

It was context coordination.

That sounds like a documentation problem, but it did not feel like one. It felt like a delivery problem. A human engineer who is missing context usually slows down. They ask someone, search through old notes, or wait for the next sync. An agent does not always slow down. It can keep moving and produce work that is technically plausible, well formatted, and aimed at the wrong reality.

That is the agentic velocity trap: unclear context does not just delay work; it can accelerate the wrong work.

Rick Fast recently wrote about the broader Expedia Group™ platform shift toward agent-friendly interfaces and operating surfaces. This is the ground-level version from one team: what did we have to change in our own planning loop so humans and agents could work from the same understanding?

The problem was hidden context

One of the clearest examples came from multi-environment handling in AI Workbench (our web console for machine learning (ML) artifacts and workload management).

From the outside, an AI Workbench URL appeared to represent one backend environment. Under the hood, it was backed by another. A team saw their model in the UI, reasonably assumed it existed in the environment implied by the URL, and then hit “model not found” when downstream jobs queried that expected backend.

Internally, the concrete case was a machine learning scientist team onboarding a model. It appeared in the .prodA AI Workbench URL, but that deployment was serving model registry prodB data rather than model registry prodA data. The platform services were not simply broken. The hidden semantic mapping was because of networking and access nuances.

That distinction matters. A human team lost time because the environment model was tribal knowledge. An agent asked to write onboarding or deployment automation against the same assumptions would not know to pause and ask, "Does this URL actually map to the environment it appears to represent?" It would likely generate confident, tidy, wrong work.

This is the kind of problem that made us rethink what "context" needs to mean in an agentic development cycle. It is not enough for the context to exist somewhere. It has to be explicit enough that people can align on it, and structured enough that agents can use it without inventing missing pieces.

This is not just more documentation

The obvious push-back is fair: engineers already do not have enough time to write docs.

The difference we are experimenting with is timing and function. Traditional documentation is often archival. It records what happened after the fact so a human can remember or discover it later. The context we need for agentic development has to be written closer to the beginning. It defines the semantic boundary of the work before the team accelerates.
This comes from the same instinct as Infrastructure-as-Code or Configuration-as-Code: move important intent into declarative files, keep them close to the work, and make them reviewable. We are not claiming the maturity of Terraform-style state, plans, CI enforcement, or drift detection. We are not there. But the instinct is similar.

In our team’s context repository, proposal documents capture the problem, scope, constraints, risks, trade-offs, and decisions. A small STATUS.yaml file carries the machine-readable part: current stage, priority, affected services, OKR alignment, target date, Jira, and stage notes. Markdown gives humans a readable narrative. YAML gives agents and scripts a compact routing layer.

For example, our in house Model Monitoring proposal status tells you, without reading the full Product Requirements Document (PRD), that the work is in implementation stage, affects services such as AI Workbench, Workflow Orchestration, Model Registry, and Deployments, aligns to embedded Model Monitoring, targets June 2026, and is tracked under a specific Jira.

That is not a replacement for judgment. It is a shared starting point.
The payoff is that agents do not start from a vague chat prompt. They can read the same structured context that product, engineering, and architecture review. Repository-level instructions define the rules of the road. Declared AI skills operate on those files to carry context forward, enrich proposals, validate gates, or generate next-stage drafts. The agent becomes less like a magic text box and more like a runner operating inside the same workflow as the team.

AI skills deserve a short explanation, because they are what makes any of this repeatable rather than a one-off prompt someone got right on a Tuesday. An agent skill is a folder containing a SKILL.md file: YAML frontmatter with a name and a description, then the procedure itself written in plain markdown.

The description does more work than it appears to. An agent loads only the name and description of every installed skill up front, and reads the full body only when it judges the skill relevant to the task at hand. The description is effectively the trigger condition; the body is the procedure. Here is a trimmed skill that checks whether a document is complete enough to build against:

---
name: check-context
description: >-
  Verify that a design document states everything needed to build against it.
  Use before generating code, tickets, or automation from a document.
---

# Check Context

1. Read the status file beside the document. Note the stage it is in and
   the systems the change touches.
2. Confirm the previous stage was approved. If it was not, stop and say
   which approval is missing.
3. For each system listed, confirm the document says what that system is
   expected to do. List any that it does not.
4. For each environment named, confirm the document states which backend
   it maps to. Do not infer the mapping.
5. Report what is missing and stop. Do not fill in the gaps yourself.

How it changed the work

The biggest shift was not that agents wrote more code. It was that we put more attention on what happens before code.

Inception became the place to answer: what problem are we solving, for whom, and what is out of scope? PRDs became less like static approval documents and more like structured context for later decisions.

Implementation plans became a way to separate work that needs human judgment from work that is safe to delegate, automate, or parallelize.
That sounds close to normal product development, and in some ways it is. The difference is the second reader. The document is no longer only for a PM, engineer, or architect. It is also for the agent that will later be asked to summarize trade-offs, identify missing context, generate tickets, draft implementation steps, or review whether the plan still matches the stated goals.

That changes the quality bar.

A vague “support multiple environments” requirement is hard enough for humans. For agents, it is an invitation to fill in the blanks. A structured proposal that names the confusing environment topology, the affected services, the user friction, the success criteria, and the out-of-scope areas gives both humans and agents a better boundary.

It also changes stakeholder alignment. A product lead or engineering lead does not need to read every page to answer basic state questions. The dashboard and status files show what stage a proposal is in, what services it touches, which OKR it supports, and which epic tracks delivery. The longer documents remain available for the why.

The cross-proposal links are where this starts to feel less like documentation and more like a working context system. Model Monitoring metrics are environment-scoped in our data lake. AI Workbench needs an environment selector so users can view the right monitoring data from the right environment. That dependency is not just in someone’s head; it is captured in proposal context where humans and agents can discover it.

What happened when we used it

In-house Model Monitoring is the cleanest positive example. The problem was concrete. We had 100s of models on the platform, but only a handful were onboarded to a vendor solution. That left a lot of opportunity on the table without standardized monitoring by default. Teams could build their own bespoke checks, but the platform had no consistent baseline for model health. The first useful agent contribution was not code. It was research compression.

The team already had important context scattered across previous monitoring work, Vendor evaluation notes, lessons from previous iterations of Model Monitoring and vendor experience, internal docs, and tool comparisons. Instead of asking an engineer to manually reassemble all of that from memory, search, and old links, the agent helped pull in the background through the enrichment workflow.

That enrichment saved hours of manual research but more importantly, it changed the shape of the discussion. The PRD records a comparison of nine alternatives across drift detection, data quality, evaluation, explainability, scalability, integration, and cost. It also captures licensing risks that matter in a platform setting: Deepchecks carried an AGPL 3.0 risk, while Seldon Alibi Detect used a BSL 1.1 license that would require care for production use.

Those details did not remove human judgment. They made the judgment easier to exercise.

Engineers and leads could spend more time deciding what mattered: time to market, reuse of existing vendor patterns, avoiding previous operational failure modes, data lake as the metrics store, and AI Workbench as the visualization surface. The agent helped surface and organize the terrain.

Humans still owned the decision.

The status layer then made the work legible beyond the immediate authors. A reader could see that Model Monitoring was P1, tied to the embedded monitoring OKR, linked to its delivery epic, and touching multiple services. That is the alignment benefit. The two hours saved is useful, but the repeatable pattern matters more: every proposal has a research phase and an alignment phase. Structured context makes both easier to reuse.

What is still hard: context drift

Context drift is what happens when the shared context was once correct, but reality moves on. If a decision happens in Slack, Zoom, or a hallway conversation and the proposal never changes, the agent is immediately behind. It will continue to reason from the stale file. The output may look clean, but it targets an older version of the team’s understanding.

This is where the model is still immature. We have conventions, templates, status files, and skills. We do not yet have perfect drift detection. We do not have a reliable way to fail a build because a hallway decision was never committed to the context. The current answer is discipline, lightweight rituals, and making the cost visible.

Too much structure becomes bureaucracy. Too little structure becomes noisy agent output. The balance we are trying to find is the smallest amount of context that keeps humans aligned and agents pointed at the right problem.

Start with one workflow

The context repository is not the product. The habit is.

For another team starting this journey, I would not begin with a full operating model. Pick one workflow where agents already help, or already create noise. Move the important context for that workflow close to the work. Make it structured enough to review, reuse, and carry forward.
Then treat context freshness as part of delivery, not a documentation chore after delivery.

In an agentic development loop, the limiting factor is not always how fast code can be generated. Often, it is how clearly the team can express what should be generated, why it matters, and what must not be lost along the way.

Learn about life at Expedia Group

The Context Tax of Agentic Development was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

What happened when we treated a framework migration as an architecture modernisation — and cut P99 inference latency by two-thirds

Expedia Group™ has always been a market leader in providing personalised search experiences for travellers. As our ranking models evolved, we saw an opportunity not just to migrate to Keras 3, but to modernise the broader stack around it so we can better serve travellers. This led us to rewrite key parts of our pipelines that made model training 30% faster and cut P99 inference latency by two-thirds.

Our main focus in this blog is to discuss our improvements to lodging ranking — our service which handles users’ search requests and returns a personalised property ranking on top of a lightweight candidate generator. This service is frequently retrained to improve customer experience as we get new signals over time.

As such, our velocity to make changes needs to be fast, but more importantly we need to be able to build models which are state-of-the-art, enabling customers to easily find and book the most relevant property for their trip.

The problem with Keras 2, eager training and combined serving

The challenge was not that the old stack was broken. It was that it had gradually become a limiting factor.

On the training side, we were carrying dependencies on older components and missing out on modern Keras APIs, new optimisers, new layers and cleaner support for accelerator-oriented training. On the serving side, we had a path that was simple and operationally familiar, but not especially well suited to aggressively optimised inference for heavier ranking architectures.

This blockage on utilising new layers and optimisers has become more pronounced as LLMs see blistering amounts of innovation. Naturally, these new features are only available in Keras 3 or as separate backends from our default TensorFlow-only serving. As Machine Learning Scientists, this significantly impacted our innovation velocity, as we either had to recreate layers — such as mixture of experts implementations in TensorFlow — or we could not simply utilise the functionalities (such as new optimisers like Muon).

That mattered more as we started thinking about models whose compute is dominated by dense matrix operations. Mixer- and transformer-style ranking models benefit heavily from GPU execution, graph compilation, kernel fusion and mixed precision. If the stack cannot take advantage of those capabilities cleanly, latency becomes the main constraint on what you can train and serve.

The prior state: a coupled preprocessing and scoring graph

In our ranking stack, all the feature transformations happen in preprocessing layers generated by Kamae. Kamae is our open source preprocessing repository, covered in our 2025 RecSys paper, that maps TensorFlow operations to Spark operations. This enables fast, distributed, and one-time preprocessing before training, as well as produces a single preprocessing graph for serving. Kamae handles operations such as categorical lookups, one-hot encoding and other transformations over raw request features, including string-heavy inputs such as country codes.

That design led naturally to a coupled serving graph: as TensorFlow supported the string transformations we needed, the simplest production architecture was to bundle preprocessing and the neural network together into one fused TensorFlow graph. That made testing and deployment straightforward, and for earlier model generations, it was a very reasonable trade-off.

But the same architecture also created a ceiling

Because preprocessing and scoring were tightly coupled, we could not easily move the neural network into more aggressively optimised runtimes without first separating those graphs. The fused graph was stable, but it limited how far we could push compiled serving and hardware-aware optimisation.

Our string-based operations in preprocessing could only be run with eager training in contrast to JIT (Just-In-Time compilation).

JIT support is not new to Keras 3 but is a major part of our migration. JIT means Keras traces the model into a static computation graph and hands it to XLA (accelerated linear algebra, an ML compiler), which can fuse operations together enabling more efficient hardware utilisation. The main catch is that compiled execution is built around numeric tensor operations, not string processing, so preprocessing stages that depend on things like one-hot encoding from raw strings or other vocabulary-based transformations cannot cleanly be compiled into the same optimised graph.

What Keras 3 brought — multiple backends and new operations

Keras 3 was a fairly substantial shift for the ecosystem because it moved Keras away from being treated primarily as tf.keras and back toward being its own higher-level framework with a cleaner import keras API, a stronger backend-agnostic model definition style and support for writing more logic in framework-neutral primitives such as keras.ops rather than directly in TensorFlow ops.

In practical terms, that meant cleaner imports, more emphasis on portability, a newer native .keras serialisation format and some breaking API cleanup around how models, variables, reductions and layer resolution are handled, compared with older tensorflow.keras patterns.

Alongside that, the broader ranking stack around Keras also changed. Keras 3 made it much more natural to target newer execution paths and backends, while older TensorFlow-coupled utilities became less attractive to depend on long term.

One important example is TensorFlow Ranking: its deprecation was not a feature of Keras 3 itself, but it landed in the same general modernisation wave, and many teams migrating to Keras 3 have used that moment to drop the dependency entirely and move ranking losses, metrics, and supporting logic into their own libraries or directly onto standard Keras APIs.

This has also opened new opportunities for us to move towards more backend-agnostic code which could enable the usage of PyTorch or JAX. This means, should we wish to use highly optimised kernels which do not have native bindings in our chosen backend, our layers can easily migrate to match. We still predominantly use TensorFlow, as our preprocessing for most models still requires strings, but we have added flexibility.

Beyond Keras 3 migration — speeding up training and inference

Rather than treating Keras 3 as just an API update, we used the migration to modernise the core ranking stack. That included rewriting layers, losses, metrics, and training logic to be Keras 3 native and XLA-safe, while preserving parity with existing production behavior.

This allowed us to run testing within our existing training and ensure performance parity while benefitting from the speed-ups and the cleaner, more backend-agnostic APIs.

Kamae operations were rewritten to use backend-agnostic operations, with the exception of string operations that are maintained for legacy support in TensorFlow only. For our current models which rely on string inputs, nothing changes and we can still produce parity. For future models we can now choose XLA-safe preprocessing layers and as a result, support different backends should we want the entire model graph in JAX for instance.

Once the neural-network portion of the stack was cleanly XLA-safe and Keras 3 native, a more optimised serving design became possible. The key architectural move was to split the old fused graph into two stages to retain compatibility with string-containing preprocessing. Our new separate graphs consist of:

• Preprocessing in TensorFlow, where the string-heavy transformations still belong
• Neural network in Keras, comprising purely numeric operations and enabling the graph to be exported to an optimised runtime

We currently make use of ONNX and TensorRT to provide our faster runtime for the trained graph.

Speeding up the graph

ONNX is an open-standard intermediate representation for machine learning models. ONNX Runtime takes an ONNX graph and applies hardware-specific optimisations at load time (e.g. collapsing a MatMul + BiasAdd + ReLU) into a single fused kernel. Because ONNX Runtime is a purpose-built inference engine rather than a full training framework, it carries none of the overhead that TensorFlow or PyTorch bring to serving.

TensorRT is NVIDIA’s GPU inference optimiser and runtime, designed to take a neural network graph and compile it into a form that runs more efficiently on specific NVIDIA hardware. It improves latency and throughput by doing things like kernel fusion (see example illustrated below), hardware-specific optimisation, and lower-precision execution such as FP16 where accuracy allows. It further refines ONNX graphs in our case and enables us to optimise for the GPU we specifically are serving on.

Serving multiple graphs

With these more optimised runtimes, we can ensemble the preprocessing and model graph together at serving time with Triton Inference Server. Triton Inference Server is a serving platform which enables users to utilise multiple graphs with low cost memory passes and supports TensorRT natively. Triton configurations are able to be autogenerated as we already know how the graphs relate from preprocessing to training.

This separation matters because ONNX and its runtimes are much better suited to aggressively optimised numeric-inference than to vocabulary based string-transformation. ONNX Runtime can apply graph optimisations such as operator fusion and hardware-specific execution-provider selection, TensorRT can then go further on NVIDIA GPUs through device-specific compilation, kernel selection, and mixed-precision execution.

So the migration ultimately did two things at once: it modernised the training stack and it created a cleaner boundary that made compiled serving practical but still able to combine both graphs should we need singular graph TF-Serving for legacy support.

What we gained in training

Some of the biggest wins from the migration showed up before inference even began.

Our earlier staging pipeline was slow and heavily constrained by serial work. In the Keras 3 rewrite, our data handling was redesigned around more parallel Spark-based handling and improved shard management. This aligned with our design philosophy behind Kamae of having these heavy preprocessing ops parallelized by Spark. On representative production workloads, that represented a 75% speed up in writing our datasets from preprocessing.

Training also became substantially faster. On a representative production ranking model, training has become 30% faster since enabling JIT and rewriting our graph to be JIT friendly without any cost to our metrics.

As part of the rewrite, some of the legacy code which handled callbacks such as variable batch size was refactored to enable multi-GPU training. This has allowed us to throw much more compute at problems — for some of our models this means training is down from a working week to a day.

Those changes are important not just because they save time, but because they change the cost of experimentation. Faster staging and shorter training cycles mean teams can evaluate more ideas, iterate on larger models quicker, and spend less of their time waiting on experiments to complete.

What we gained in serving

The serving gains were just as important.

In isolated neural-network benchmarks, each step toward a more compiled runtime improved latency. On our old CPU-oriented serving model we saw ONNX providing major latency reductions both when we tested the model graph in isolation and tested it end to end.

We can see that while our preprocessing cost has not changed at all, we observe a reduction of 35% in total compute time due to the model graph speeding up. This is encouraging and with some of our model architectures being far more complex within the model graph than the preprocessing graph, which should enable us to better utilise our resources.

Utilising TensorRT means we can make use of hardware optimisation for mixed precision. Mixed precision means running most inference in lower precision where it is safe to do so, and in our Triton setup we use TensorRT with FP16 so the neural network can make better use of its resources. One question that naturally comes up with compiled GPU serving is whether FP16 changes ranking quality in ways that matter.

In practice, mixed precision looked like a safe trade-off for this class of model. Across our simulations, FP16 inference achieved an RBO (Rank-Biased Overlap, a measure of similarity of two ranked lists where disagreements at the top of a list matter much more than disagreements at the bottom) greater than 99% relative to the higher-precision baseline, indicating that ranking order stayed effectively unchanged while still benefiting from faster TensorRT execution.

We broadly see a substantial improvement across the board as we move to better optimised runtimes. Our P99 is higher than expected on our Triton ONNX serving pattern but still substantially lower than TF-Serving.

We then deployed our new service and compared it end to end, including preprocessing, hitting it with real traffic from Expedia®.

That kind of improvement matters because it reduces tail latency and creates room for more expressive architectures. In this case our prior P50 is now our P99. Further improvements can be seen on TensorRT FP16 where our P50 latency is reduced to only 10% of our original P99 latency on TF-Serving.

Why this mattered beyond one migration

The most important outcome of this work is more than any one benchmark. It is that the stack now has a clearer shape while retaining total parity with our old stack.

Preprocessing and neural-network execution are separated to keep strings isolated. Training is built around Keras 3 native, XLA-safe components. Serving still supports our old bundled TF-Serving approach but now has practical faster compiled paths through Triton. The same pattern can be reused for future ranking models that need better accelerator utilisation and lower latency than the previous architecture could comfortably provide.

Once a stack can train efficiently, export separate numeric model graphs cleanly and serve them through a compiled inference path, it becomes much easier to operationalize models that would otherwise be too latency-sensitive to deploy.

Further, we have been unblocked on utilisation of code from other backends now and are much more capable to run experiments with newer features such as optimisers like Muon. This has massively increased the velocity of experimentation.

Modernisation is often described as maintenance work. Sometimes, it is. But sometimes, modernisation is what makes the next generation of systems possible. For us, Keras 3 was exactly that kind of migration.

Acknowledgements

This would not have been possible without the fantastic work of the Expedia Group Search Ranking AI team as a whole. However, a special shoutout must go to George Barrowclough, James Shinner and Daniele Donghi who actively assisted with the Keras migration alongside maintenance of our core ranking repositories. A further thanks to Rajat Malhotra who assisted with productionalisation and testing of the new serving pattern. Finally thanks to Anne Morvan, Hafiz Tiomoko Ali, Youssef Achenchabe and Zoe Yang for their careful review of the article and valuable feedback.

[1]: Deploying Deep Neural Networks with NVIDIA TensorRT (Gray et al., 2017)

How Keras 3 Helped Modernise Expedia Group’s Lodging Ranking Stack was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Combine a GraphQL schema, hints and a LLM to generate contextual GraphQL mock responses

A product developer on our navigation header team spent an afternoon hand-typing a 200-line GraphQL JSON mock response so they could keep building the UI while a resolver was pending to be implemented. The schema changed the next morning. We threw the mock away. That is the boring tax on every GraphQL prototype: mocks that drift, fixtures that rot, frontends blocked on backends, and demos slipping because nobody wanted to update the same mock data again. Mock data is just data. It should not be the most expensive part of trying an idea.

GenAI + GraphQL: a match made in heaven

Most tools that promise “AI generates an API” share the same flaw: the model invents a shape from scratch, and you spend the afternoon reshaping its output to match your actual types.

GraphQL flips that. The selection set is the spec. The schema is the type system. Hand an LLM a query and it already knows the exact JSON it must return, field by field, type by type. That’s the unlock: LLMs are bad at inventing shapes and great at filling them in, and GraphQL hands them a bounded shape for free.

mockql-rs is what happens when you take that pairing seriously: mark what you want mocked directly in the query with a “@mock” directive:

query LoyaltyRewards @mock {
  loyaltyRewards {
    heading @mock(hint: "Platinum member") { 
      text 
    }
    subtitles @mock(hint: "At least 4 items") { 
      text 
      theme 
    }
  }
}

Annotate the field, add a hint, keep building. mockql-rs parses the operation, validates it against the schema, and prompts an LLM to fill in only the fields you annotated with “@mock”.

{
  "data": {
    "loyaltyRewards": {
      "heading": { "text": "Welcome back, Platinum Member" },
      "subtitles": [
        { "text": "2,450 points until your next reward", "theme": "HIGHLIGHT" },
        { "text": "3 nights earned this quarter", "theme": "STANDARD" },
        { "text": "Breakfast included on your next stay", "theme": "POSITIVE" },
        { "text": "Suite upgrade available at checkout", "theme": "POSITIVE" }
      ]
    }
  }
}

No handwritten fixtures. The mock lives in the query and the response matches the schema, always. GraphQL stays the contract. The LLM never invents an API, it fills in the shape that the schema has already drawn.

Contextual mocking

The real power of mockql-rs shows up when you need most of a response from your actual backend but one field isn’t ready, or even available yet.

query TripDetails($id: ID!) {
  trip(id: $id) {
    property {
      name
      address
    }
    recommendations @mock(hint: "5 most popular nearby restaurants") {
      title
      description
      distance
    }
  }
}

{
  "data": {
    "trip": {
      "property": { // <- Real upstream data
        "name": "Hotel Palazzo Pischedda",
        "address": "Via Roma, 09089 Bosa OR, Italy"
      },
      "recommendations": [ // <- Context-aware GenAI mock data
        {
          "title": "Ristorante Sa Pischedda",
          "description": "Located directly at the hotel, famous for its authentic Sardinian seafood dishes, fresh handmade pasta, and excellent local Vermentino wine.",
          "distance": "0.0 miles"
        },
        {
          "title": "Locanda di Corte",
          "description": "Charming restaurant nestled in the historic medieval center, serving traditional Bosa cuisine with a modern twist in an intimate courtyard.",
          "distance": "0.3 miles"
        },
        {
          "title": "Ristorante Borgo Sant'Ignazio",
          "description": "Cozy, rustic eatery offering hearty Sardinian meats, local cheeses, and traditional dishes inside a beautifully restored stone building.",
          "distance": "0.2 miles"
        },
        {
          "title": "Aldualis Ristorante",
          "description": "Slightly elevated restaurant outside the historic center offering a refined contemporary take on regional recipes with stunning views over Bosa.",
          "distance": "0.6 miles"
        },
        {
          "title": "Ristorante Ponte Vecchio",
          "description": "Lovely riverside venue near the old bridge, specializing in fresh catch-of-the-day fish and traditional Sardinian lobster dishes.",
          "distance": "0.1 miles"
        }
      ]
    }
  }
}

The real backend resolves the trip property, the LLM resolves real recommendations for that particular property. Your frontend gets a completely cohesive, realistic data without a single hardcoded fixture or configuration.

How it works ?

mockql-rs sits between your GraphQL client and your GraphQL server, as a thin layer, a standalone CLI.

Why a CLI ?

a CLI is the smallest integration surface. Any client can use it because every language, test runner, CI job, Makefile, npm script, or demo environment can execute a process. You do not need an SDK, Apollo Client link, Relay or any backend framework. You hand to a process an operation, a schema, an upstream GraphQL URL, and a provider, and it returns a GraphQL response.

Execution flow

When mockql executes a GraphQL request it will:

1. Parse: the operation is parsed and validated against your schema using the apollo-compiler.
2. Split: fields annotated with “@mock” are separated from real fields. Real fields are forwarded to the upstream server as normal. For those familiar with GraphQL Federation, this process is very familiar to query planning.
3. Prompt: the operation, mocked fields, hints, and the subset schema are assembled into a structured prompt. The operation and the schema constrain the LLM’s output shape; it can’t hallucinate fields that don’t exist or return a string where the schema expects an enum.
4. Merge: real upstream data and LLM-generated mock data are stitched back into a single response.

Why now ?

GraphQL response mocking is finally practical. LLMs are exceptionally good following schemas and rust + apollo-compiler makes parsing and validation blazingly fast.

Try it

mockql oneshot \
  --operation ./my-query.graphql \
  --graphql-url https://your-graphql-server/graphql \
  cli --provider claude

Pick one handwritten fixture blocking your prototype. Replace it with “@mock”. Ship the demo on Friday.

• Repository
• @Mock Spec
• Getting Started

Learn about life at Expedia Group™

Focus on the Feature, Not the Fixture: GenAI powered GraphQL mocks was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A framework for how we build, deploy, and evolve AI systems for impact and scale

There’s an important distinction between Artificial Intelligence (AI) that just works today and AI that lasts at scale. Many companies optimize hard for the first one without ever asking whether they’re building the second.

Velocity without discipline and strategic direction is a liability, not an asset. The hardest part of building AI at scale isn’t getting a model to work once. It’s building systems that continue to work, scale beyond individual teams and use cases, and improve consistently over time.

Today’s AI systems do more than just predict and optimize. They converse, reason, and increasingly take action. An autonomous system making decisions on a traveler’s behalf creates a very different set of expectations around reliability, governance, and accountability. As AI takes on more of those roles, the principles behind how these systems operate matter more than ever.

At Expedia Group™, we have spent years applying AI and machine learning across the traveler journey from personalization, ranking, and recommendations, to fraud prevention, customer support, and, more recently, generative and agentic AI experiences. That depth of experience is what led us to develop a set of machine learning and AI principles to guide how we build, deploy, and evolve AI systems across the company.

The goal is simple: make sure the systems we build create real business value, scale across the company, and operate safely. These principles define how we measure, design, govern, and operate the systems we use.

From principles to practice

Publishing principles is the easy part. The harder and more important work is turning them into operating mechanisms: recommendations, requirements, tooling, and release processes that teams actually use. At Expedia Group, we have started doing this through Agentic Release tollgates: a set of recommended and, in some cases, required checks before launching agentic AI features. These tollgates translate principles like clear ownership, risk-based governance, evaluation, safe rollout, and monitoring into concrete expectations for teams. Some of these recommendations and requirements are already being automated and integrated into the software development life cycle (SDLC). Over time, the goal is for these expectations to become embedded in how we design, evaluate, approve, launch, and monitor AI systems from the start.

Outcomes: measuring what actually matters

The first test for any model at Expedia Group isn’t whether it improves a technical metric. It’s whether it improves a business outcome and, ultimately, the traveler experience.

• Align models to metrics with business impact: Every machine learning effort must tie directly to a key business outcome or traveler experience metric. Technical optimizations are useful midpoints, not end goals.
• Optimize for return on cost: The value a model creates has to justify what it costs to develop, train, and monitor it, plus the operational complexity it adds. Favor solutions that deliver lasting impact relative to what they cost to run.
• Justify complexity against strong baselines: Complexity should be earned, not assumed. Start with a strong baseline: an existing general model, a simple heuristic, an off-the-shelf solution. Reach for specialized models or more complex architectures only when simpler options genuinely can’t meet the bar.
• Require both offline and online evaluation: No model goes to broad deployment on offline validation alone or jumps straight to A/B testing. Every model must perform in both offline and online evaluations. Over time, our offline evaluations should reliably predict what we see online.

Design: building systems that scale beyond the teams that build them

Getting a model to work is one challenge. Making its value extend beyond a single team or use case is the harder one.

• Build on shared foundations; specialize only when justified: Favor shared, platform-wide foundations for core capabilities, data representations, and model building blocks. Specialization should build on those foundations, not spin up isolated stacks, so when the foundation improves, the gains flow across the organization.
• Treat data as a core product: A model’s quality is bounded by the quality of its data. We need to maintain robust pipelines, clear lineage, reproducibility, and reusable features built with documented ownership, clear schemas, and SLAs that other teams can rely on.
• Prioritize generality over local optimization: When two approaches perform similarly, favor the one whose learnings, assets, and operating patterns can be reused across teams, brands, and use cases. We should optimize not just for local performance, but for how quickly improvements can diffuse across the company and compound over time.
• Minimize and sunset manual business rules: Manual rules are sometimes necessary for policy, safety, or compliance, but they should be explicit and reviewed regularly, never silent patches for weak models or a source of permanent maintenance debt.
• Reproducibility and traceability by default: Training data, features, configurations, evaluation results, deployment versions, and key decisions should all be documented and recoverable. That’s what lets you debug a production issue months later and hand off ownership without losing institutional knowledge.

Trust: ownership, governance, and operating responsibly at scale

The bar for deploying AI isn’t just “does it work?” It’s “can we stand behind it?” Trust isn’t something you add at the end; it’s earned over time and maintained across the full lifecycle of every model we ship.

• Assign clear ownership and accountability: Every model needs defined ownership across its lifecycle: a business owner, a product owner, an AI owner, and an operational owner. These don’t need to be four people, but the responsibilities must be explicit. Who’s accountable for outcomes? Who responds if the model drifts? Who answers the incident at 2 a.m.? Without this in place, models become orphaned and problems surface with no one to own them.
• Adhere to standards and governance: AI and machine learning models must use approved platforms and comply with established company standards, release gates, and governance processes. Operating outside these guardrails requires a clear, defined path to remediation or deprecation, rather than an open-ended exception.
• Govern proportionally to risk: The level of review, evaluation rigor, and human oversight should scale with a model’s impact. A customer-facing model that affects pricing or availability for millions of travelers demands a far higher bar than an internal tool used by a small team. For high-impact, safety-sensitive, or highly autonomous systems, human-in-the-loop checkpoints are built in from the start.
• Design for fairness, privacy, and transparency: We build with traveler and partner trust in mind: actively testing for unintended bias, practicing strong data guardrails, and favoring explainability when decisions meaningfully affect users. These are designed from the start, not added on.
• Design for safe rollout, rollback, and control: Deployments are progressive, with rollback paths, fallback mechanisms, and circuit breakers ready before launch. The ability to safely undo a deployment matters as much as the ability to ship it.
• Monitor continuously and adapt: Once live, teams need to actively monitor quality, drift, latency, cost, and business performance and retrain or recalibrate when the data shifts. A team should always be able to explain how its model is performing now, not just how it performed when it launched. For example, we recently introduced algorithm score cards. Updated regularly, these scorecards provide a simple, consistent framework for reviewing how key marketplace algorithms support business goals, perform in practice, and contribute to marketplace health.

These principles do more than define how we build. They define what we’re willing to ship and how we stand behind it. In a world where AI systems are increasingly consequential and make real decisions for real travelers and partners, these standards matter. Applied consistently, they build responsible AI that lasts.

Learn about life at Expedia Group

How Expedia Group Builds AI That Lasts at Scale was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using large language models to reveal bottlenecks in Spark SQL execution plans

If you’ve ever stared at a 300-plus-node physical plan at 2 a.m. trying to spot a missing broadcast or one cursed skewed partition, this is for you.

Spark makes it deceptively easy to write complex SQL that looks correct but quietly turns into a performance and cost problem at scale. A query that runs fine on day one can slow to a crawl as data grows, joins get wider, and aggregations become more nested. Suddenly, jobs take hours instead of minutes, clusters need to be over-provisioned, and costs climb with no clear explanation.

When things go wrong, the situation gets worse. Debugging Spark SQL often means wading through deeply nested physical plans, fragmented UI metrics, and stage-level statistics that don’t clearly point to the root cause. Engineers are left guessing whether the issue is a bad join strategy, skewed data, excessive shuffling, or memory pressure — turning optimization into a time-consuming, trial-and-error exercise.

So we built an automated, LLM-powered workflow that reads Spark SQL plans, flags performance anti-patterns, and suggests concrete fixes. Here’s why we built it, how it works, and what we learned after running it on real workloads.

Why we even needed this

Across teams we kept bumping into the same pain points:

• Slow or failing jobs that needed deep Spark expertise to debug
• SQL that didn’t behave the way people assumed
• Skew, spills, or partition issues hiding beneath the surface
• Tuning efforts that were reactive and inconsistent

At some point, the obvious question came up:

Can an LLM look at a Spark SQL plan and tell you what’s likely to go wrong — where cost will spike, where execution will drag, and where seemingly harmless SQL hides real operational risk?

Turns out: yes

How we approached it

1. Bring in real context with the Spark MCP server

We used an open-source Spark MCP server that exposes:

• SQL execution metadata
• Final and physical plans
• Stage and task metrics
• Shuffle, spill, and I/O stats

This gave us clean, structured insight into the slowest SQL statements for each app — perfect for an LLM to work with.

Reference: GitHub — kubeflow/mcp-apache-spark-history-server (list_slowest_sql_queries).

2. Replace free-form chat with pattern-guided prompting

Early attempts with generic prompts failed.

“Analyze the SQL plan and suggest optimizations.”

The prompt was too general, and produced inconsistent results, hallucinated insights, and missed actual issues.

So we stopped being vague.

We defined clear, rule-driven anti-patterns with strict criteria. The model’s job became simple: detect only these patterns, nothing else. This made the analysis consistent and grounded in real metrics.

3. Enforce traceability

Every finding has to point to actual evidence: stage IDs, task IDs, node IDs, operator names, and table references. All output goes into a strict JSON format so downstream systems can consume it without guessing.

What the model detects

We focused on a small set of high-impact Spark SQL execution risks. What makes this different from traditional tooling is traceability: the LLM doesn’t just flag a problem — it ties it back to specific stages, plan nodes, and data characteristics, explains why Spark behaved that way, and points engineers to the exact lever to pull.

In every case below, the workflow was the same:

The LLM surfaced a concrete risk → we verified it in the physical plan → we made a targeted change → we measured the impact.

Missed Broadcast Joins

At first glance, the SQL looked reasonable. Nothing obviously wrong.
Yet one of our job consistently ran for over an hour.

What the LLM surfaced

The model traced the bottleneck to a SortMergeJoin and highlighted something Spark had quietly ignored:

One side of the join was ~96 MB — small enough to broadcast — yet Spark still chose a shuffle join.

{
  "name": "Missing Broadcast Joins",
  "detections": [{
    "stage_id": "10.0",
    "join_details": {
      "right": {
        "name": "destination_attribute",
        "size_human": "96.6 MiB"
      },
      "chosen_strategy": "SortMergeJoin",
      "expected_strategy": "BroadcastHashJoin"
    },
    "notes": "Broadcast threshold likely too low; AQE skew handling favored SMJ."
  }]
}

How we validated it

We went back to the physical plan. The screenshot below confirms exactly what the LLM predicted: a shuffle-heavy join where a broadcast should have been possible.

Impact:

We increased the broadcast threshold and the Runtime dropped from ~1 hour to ~30 minutes — with zero query rewrites.

Skewed partitions

Another job technically “succeeded”, but ran for more than three hours with a long tail of straggler tasks.

What the LLM surfaced

The model didn’t just say “there is skew.”
It pinpointed where and how bad the skew was:

• Median partition size: 0 B
• Largest partition: ~280 GB

{
  "name": "Skewed Partitions",
  "detections": [{
    "stage_id": 35,
    "node_type": "SortMergeJoin(skew=true)",
    "partition_sizes": {
            "min_human": "0.0 B",
            "med_human": "0.0 B",
            "max_human": "279.7 GiB",
            "partition_count": 16
          },
    "likely_cause": "Extreme skew detected in join output."
  }]
}

How we validated it

Looking at the SQL plan and join keys, the issue became obvious:
a left join with null-heavy keys was funneling almost all data into a single partition.

Impact

We fixed the join logic to handle null keys correctly. A job that previously ran 3+ hours brought back to expected runtimes.

Full table scans from missing filters

This pattern is especially painful — and easy to miss.

What the LLM surfaced

The model flagged a table scan in one of our jobs that was silently reading 23.5 TB of data — with no partition or date filters applied.

{
  "name": "Missing Partition or Date Filters",
  "detections": [{
    "table": "lodging_content_vrbo_room_type_profile_history",
    "input_size_human": "23.5 TiB",
    "partition_filters": [],
    "pushed_filters": [],
    "issue": "Full table scan without partition filters."
  }]
}

How we validated it

The physical plan confirmed it: a full scan of a massive Iceberg table, triggered by a missing predicate.

Impact

We added the correct partition predicates.⚡ Runtime dropped from ~20 minutes to ~1 minute.

Oversized broadcasts

Broadcast joins are usually good — until they aren’t.

What the LLM surfaced

The model detected multiple multi-GB broadcasts happening in the job, something that doesn’t always fail immediately but creates driver pressure and unstable performance.

{
  "name": "Oversized Broadcasts",
  "detections": [
    {
      "node_id": 55,
      "broadcast_details": {
        "table_name": "supply.lodging_profile_vrbo",
        "broadcast_size_human": "2.7 GiB",
        "join_type": "BroadcastHashJoin (feeds node 13)"
      },
      "impact": "Large broadcast may increase driver memory pressure and prolong broadcast/build time."
    },
    {
      "node_id": 25,
      "broadcast_details": {
        "table_name": "supply.lodging_profile_eg",
        "broadcast_size_human": "2.2 GiB",
        "join_type": "BroadcastHashJoin (feeds node 24)"
      },
      "impact": "Multiple >1 GiB broadcasts in the same query amplify driver/network overhead."
    }
  ]
}

How we validated it

The Spark UI confirmed long broadcast built times and elevated driver memory usage.

Impact

We reworked the join strategy to avoid broadcasting large datasets. 🚀 Runtime dropped from ~20 minutes to ~1 minute, with improved stability.

Other patterns the model detects

The model also identifies subtle performance anti-patterns that often go unnoticed but significantly impact runtime and cost.

Aggressive coalesce operations

The model flags overly aggressive AQE coalescing when shuffle partitions grow excessively large (for example, >2 GB or >10× the advisory partition size) and overall parallelism drops below available executor cores. This typically results in underutilized clusters and long-running straggler tasks.

High spill volumes

Excessive memory or disk spills are detected when total spill size exceeds 1 GB, spills account for a large fraction of input data (≈20%+), or when spills occur across a majority of tasks in a stage. Higher severity is assigned when spill volumes exceed 5 GB, indicating sustained memory pressure.

Data explosion due to joins

The model identifies joins where output rows grow disproportionately — 2× or more than the largest input — often caused by duplicate keys, missing predicates, or incorrect join grain. These joins can silently inflate data volume and downstream shuffle costs.

Each pattern is backed by concrete metrics and physical plan operators, enabling the model to convert complex Spark execution details into clear, actionable guidance.

The prompting strategy that finally worked

The structured prompt tells the model to:

• Analyze only the slowest SQL in the application
• Cross-check evidence before reporting anything
• Ground every observation with stage/task/node IDs
• Output only structured JSON with validated detections

This eliminated false positives and made the analysis predictable.

Prompt Used:

https://gist.github.com/bhuvanaSagi/149518ddeabc05020cafd95a3c19e83d

What we saw in real workloads

Across multiple Spark workloads, the LLM-based analysis consistently identified all the specified patterns in the SQL Plans.

We used these insights to tune Broadcast thresholds, add partition filters, and adjust AQE settings.

• Hours of manual debugging saved per incident
• Runtime dropped by 40–95% on long-running jobs
• Compute cost dropped by 50–90% per job

The exact numbers vary by workload, but the pattern was clear: faster debugging, fewer surprises.

How you can extend this approach

You can take this further in a bunch of ways:

• Add new rules as you see more performance issues
• Introduce Delta or Iceberg-specific checks
• Map slow stages to individual plan nodes
• Adjust thresholds for skew, spills, and broadcasts
• Feed detections into dashboards or alerts for proactive tuning

Once the context is structured, extending the system becomes easy.

Closing thoughts

This isn’t about replacing Spark experts. It’s about giving them leverage.

LLMs handle the repetitive triage so engineers can focus on the fixes that matter. By combining structured plan data, execution metrics, and domain-guided prompting, we automated a debugging workflow that used to demand deep Spark internals knowledge.

If you operate large-scale Spark workloads, LLM-assisted SQL plan analysis is a genuinely useful addition to your toolbox.

References:

https://github.com/kubeflow/mcp-apache-spark-history-server

Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How a screen-level performance metric reshaped platform decisions, engineering ownership, and release discipline

For the last few years, my responsibility has been straightforward to state but hard to execute — owning the traveler login experience across mobile platforms.

Not just whether a feature works, but whether it feels responsive, predictable, and trustworthy in the moments that matter most. During login, those moments are unforgiving: if a login screen hesitates travelers don’t interpret it as ‘a slow render’, they interpret it as risk.

And when the majority of travelers interact through our mobile apps, performance stops being a technical concern and becomes a product promise.

This post is a case study of how, within Expedia Group™’s login domain, we extended Native Time to Interactive (NTTI) across login screens — moving performance from a late-stage check to a first-class signal we can use to validate technology investments, compare platform behavior, and prevent silent regressions as we ship.

The Problem: Mobile reliability rarely fails loudly

Mobile performance rarely breaks with a crash.

It usually degrades quietly,

• a button takes a beat longer to respond
• a screen looks ready, but taps don’t register
• the UI stutters just enough to feel “off”

Those are the expensive milliseconds — because they erode trust without triggering obvious alarms.

In our login flows, we were shipping consistently, evolving a major part of our stack, and supporting increasing product complexity. Yet we didn’t have a consistent way to answer the user’s real question — “When can I actually use this screen?”

Why our existing signals failed

We were not blind. We tracked many useful things,

• crashes and ANRs
• backend latency and service SLIs
• some component-level timing signals
• limited Time to Interactive tracking on onboarding and initial login

But we had a gap — we had visibility into system health, not screen readiness.

Once travelers moved beyond the initial login screen into deeper login flows, performance became harder to compare release-to-release. We could detect obvious failures, but we lacked a consistent, screen-level metric that told us,

• Did this flow get slower after shipping feature X?
• Did the new tech stack improve what travelers feel?
• Are iOS and Android behaving similarly, or are we averaging away differences?

And during a major tech stack migration, leadership asked fair questions — Are we actually delivering a better traveler experience ? Or just different architecture?

We needed something more aligned with user-perceived readiness.

What we chose to measure

Expedia didn’t find an industry-standard metric that captured native screen interactivity the way Web Vitals capture LCP and TTI on the web. So we coined a native-inspired equivalent: Native Time to Interactive (NTTI).

Definition: All components that are visible in the initial viewport are loaded and available for interaction.

NTTI measures:
The time from when the screen starts loading to when its key components have loaded.

Key terms

• Screen Start: when a new screen starts rendering and the user can’t interact yet (user is effectively blocked)
• Key Components: essential UI components that must be ready for meaningful interaction (typically in the initial viewport)
• Last Key Component: when the final key component completes render and is ready for interaction

Formula

NTTI = Last Key Component − Screen Start

How we instrumented NTTI across login screens

Performance instrumentation wasn’t absent before — but it wasn’t consistent across the login journey.

We had:

• component-level signals on both platforms
• limited screen-level visibility on a few launch flows
• gaps across deeper login screens

Extending NTTI across login screens required more than “adding timers.” We had to make deliberate decisions.

Why NTTI worked for us

NTTI anchors to a human outcome:

A screen isn’t “fast” when it renders. It’s fast when it responds.

A screen that draws quickly but doesn’t respond to touch feels broken. A slightly slower screen that responds reliably feels fast.

NTTI helped us measure what travelers actually experience: readiness to interact, not just rendering completion.

The work broke down into four steps

1. Define “interactive” from a traveler perspective

We aligned on what “usable” means for each screen — not in theory, but in practice.
This forced healthy product + engineering conversations: what’s essential vs what’s nice-to-have.

2. Choose key components per screen

We selected a subset of components that represent real readiness.
Not every UI element should gate interactivity; if everything counts, NTTI becomes noisy and less actionable.

3. Standardize measurement across platforms

We ensured the measurement approach was comparable across iOS and Android even though runtime behavior differs.

4. Make the data reliable enough for decisions

A metric is only useful if people trust it. That meant validating instrumentation, watching for gaps, and ensuring consistency across releases.

This wasn’t just observability work. It was building an operational contract: if we ship UI changes, we must be able to measure the impact.

What we learned: iOS and Android are not symmetric

One of the most important things NTTI made visible is what experienced mobile engineers already know, but organisations often ignore — iOS and Android behave differently even when the UI looks identical.

iOS issues often surfaced as

• subtle UI delays
• main-thread contention
• strict lifecycle + rendering constraints

Android issues often surfaced as

• device + OS fragmentation
• OEM-specific behavior
• hardware-dependent variance

Before NTTI, it was easy to blend results and assume parity. After NTTI, we could see platform deltas clearly and evaluate them fairly.

The same backend and product flow could yield different NTTI uplift across platforms. That wasn’t a product inconsistency — it was platform behaviour and implementation nuance.

NTTI made those differences measurable instead of debatable.

The unexpected outcome: Better ownership

One of the most valuable outcomes had little to do with the metric itself.

Once NTTI was introduced, teams naturally started to

• instrument what they shipped
• read dashboards with intent instead of curiosity
• connect code changes directly to traveler experience
• identify and fix regressions earlier in the cycle

Observability stopped feeling like a downstream activity. It became a core engineering discipline. And because the metric reflected real user experience, performance stopped being “just an engineering concern.” It became a shared responsibility. Product teams aligned on performance as a release guardrail and part of the user experience discussion. Design helped define what “ready” actually meant from an interaction standpoint. Engineering made those expectations measurable and actionable.

Over time, ownership shifted from “Someone will catch this later.” to “If I ship it, I monitor it.” That kind of cultural shift is difficult to enforce top-down.
NTTI helped create it through consistency, visibility, and repeated feedback loops.

How this changes release conversations

Today, NTTI is one of the first signals I look at

• before features ramp to wider production
• while validating major technical initiatives
• during release readiness discussions

We haven’t yet fully adopted NTTI as a formal rollout blocker, but that’s the direction we’re moving toward.

What we intend to do next

• Treat NTTI degradation as a “pause and investigate” signal
• Require explicit conversations before ramping when performance moves outside expected thresholds
• Protect hard-earned performance gains from slowly regressing over time

To be clear, NTTI won’t be the only release signal, but it will become an important guardrail because it forces the right conversations to happen early, while fixes are still practical and inexpensive.

What NTTI tells us — and what it doesn’t

Being explicit about scope has helped us use the metric responsibly.

What we know

• NTTI measures screen readiness from an interaction perspective
• the same change can yield different outcomes across platforms
• it works as an early regression guardrail
• it improves release conversations through concrete signals
• it increases engineering ownership of performance

What we don’t claim

• NTTI doesn’t explain user intent or behavior
• it isn’t used in isolation
• it doesn’t prove causality beyond what it measures
• platform parity is not assumed or enforced

Closing thought

NTTI didn’t just provide another metric, instead it gave us

• earlier visibility into traveler friction
• a concrete way to validate tech stack investments
• a growing culture of engineers owning performance
• a shared language for cross-platform performance conversations

During login where trust, speed, and reliability are inseparable this discipline matters. Performance isn’t something we optimize later. With NTTI, it’s something we design for from day one.

Learn about life at Expedia Group

The Most Expensive Milliseconds Are Unmeasured was originally published in Expedia Group Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.