<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DT Engineering on Medium]]></title>
        <description><![CDATA[Stories by DT Engineering on Medium]]></description>
        <link>https://medium.com/@dt_engineering?source=rss-25681449fffc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5bm3WUGRlFqRGpe_8zXSCg@2x.png</url>
            <title>Stories by DT Engineering on Medium</title>
            <link>https://medium.com/@dt_engineering?source=rss-25681449fffc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:46:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dt_engineering/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building the DT Semantic Layer: Creating a Single Source of Truth with Agent-Driven Data Modeling]]></title>
            <link>https://medium.com/dt-engineering-blog/building-the-dt-semantic-layer-creating-a-single-source-of-truth-with-agent-driven-data-modeling-fa7eaedd5459?source=rss-25681449fffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa7eaedd5459</guid>
            <category><![CDATA[semantic-layer]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[DT Engineering]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 15:02:09 GMT</pubDate>
            <atom:updated>2026-10-20T09:39:06.985Z</atom:updated>
            <content:encoded><![CDATA[<p><em>by Dan Ferrante, Krystian Kurek, Kyrylo Yershov, Sarit Banerjee, VJ Davey</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6tJQ41mXYxvjFMvKSUhAOA.png" /></figure><p>At <a href="https://www.digitalturbine.com/">DT</a>, we serve ads at a scale where the data challenges are as real as the business challenges. Across our platform, teams were independently answering the same questions with different numbers because they were pulling from different tables, applying different transformations, and working from different assumptions. Upstream schema changes broke pipelines silently. Data scientists spent more time wrestling raw joins into shape than building models.</p><p>Rather than patching individual issues and failures, we designed and built a shared logical foundation that every team, engineering, data science, analytics, and business users, works from. One definition of a user. One definition of a device. One way to calculate a device match rate or resolve a geographic hierarchy. Entities defined once, versioned, tested, and reused across every downstream system that needs them (Reporting, Data Science, Analytics, Engineering). The semantic layer that establishes a single source of truth by standardizing how core business entities are defined, built, and consumed. It ensures that analytics, machine learning models, and downstream applications all interpret data the same way. We’re still in the early stages of broad adoption, but the foundation is in place and teams are already building on top of it.</p><p>One of the clearest early signals of that payoff is in our audience pipelines. We feed semantic layer data as input to the pipeline that produces audience lists for users we bid on through our demand-side platform (DSP). The pipeline already had a solid feature set, but we wanted to give our per-app classifiers a richer signal about who a user actually is. With a clean, stable semantic foundation in place, we’re building dense user and app embeddings that encode behavioral and preference signals and feed them into those models as additional features. This work would have been far more challenging to do reliably against raw, ungoverned data.</p><p>This post covers what we built, how we built it, why we made the decisions we made, and what we’re learning along the way.</p><h3>DT Semantic Layer: A Primer</h3><p>Prior to the existence of the semantic layer, data at DT lived in silos. Each team curated and propagated its own datasets, but — more than that — the datasets lived across different buckets, different technologies, and different warehouses. The semantic layer we are building is driving consumers towards adopting a better option and working with a clear set of rules to build out downstream users’ dataset needs. Previously, even when the underlying raw data was the same, the interpretation rarely was.</p><p>This interpretation issue led to:</p><ul><li>Conflicting metrics across teams</li><li>Repeated transformation logic</li><li>Fragile pipelines and inconsistent definitions</li><li>Reduced confidence in analytics and models</li></ul><p>As part of the semantic layer design process, the first step taken was to gather a group of 15+ engineers, business intelligence SMEs, data scientists, and architects from across the company together to establish a data model and agree to foundational technologies used to implement that model. With the model agreed to, the Data Engineering Team set out to build the structure around which the dataset could be derived. Prior to those discussions and agreements, the Enterprise Data Team had already applied a commonly used data architecture pattern starting with a <a href="https://www.databricks.com/blog/what-is-medallion-architecture">base medallion architecture</a> and building up from there. Below is a view illustrating the overall architecture of the enterprise lakehouse for DT and where the semantic layer fits in around the Gold layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*JZMD_sKmgsMyEoQj.png" /><figcaption>Medallion Architecture example</figcaption></figure><h3>‍Building the DT Semantic Layer: A Phased Approach</h3><p>We took a phased approach to building the semantic layer. First defining entities, then compiling entities into data pipelines based on data stored in the metadata registry, scheduling and running data pipeline jobs to create entities.</p><p>The DT Semantic Layer was built against a modern data lakehouse stack. Rather than adopting an off-the-shelf orchestration framework, we built purpose-specific layers on top of each technology to serve the specific demands of entity compilation and deployment:</p><ul><li><strong>Databricks:</strong> We built a config-driven job factory that packages all entity ETL jobs into a single deployable Python wheel. Each entity gets its own Databricks job definition, and the factory instantiates the correct job at runtime from a shared registry meaning one wheel ships the entire set of semantic layer pipelines.</li><li><strong>Delta Lake: </strong>We built a three-strategy write system on top of Delta’s MERGE capabilities: full rebuild for reference tables, incremental merge for append-heavy entities, and a two-stage SCD2 merge for slowly changing dimensions (SCD). The SCD2 implementation in particular relies heavily on Delta’s ability to atomically match, update, and insert rows in a single operation.</li><li><strong>PySpark</strong>: We built a transform-chain ETL pattern where each pipeline step is a pure DataFrame → DataFrame function, as well as a foreign key resolution layer that handles deterministic joins across bridge tables without embedding that logic into individual jobs.</li><li><strong>Apache Airflow</strong>: We built auto-generated DAG tasks for each entity, with upstream and downstream dependencies wired at compile time based on the entity’s declared source relationships.</li><li><strong>Terraform</strong>: We built per-entity job provisioning so that adding a new entity to the semantic layer automatically provisions its compute resources, cluster configuration, and wheel reference in the production Databricks workspace.</li></ul><p>Let’s step through the construction phases one by one and describe the pieces with examples.</p><h3>Phase 1: Defining Entities</h3><p>We created a separate repository for building and storing entity definitions apart from the compilation into data pipelines to separate concerns between logical and physical models. We followed the approach that this first phase would be a low- to no-code generation process using Claude as the backing agentic AI assistance technology. To keep costs lower and because we were satisfied with its behavioral characteristics, Sonnet 4.6 was used as the LLM.</p><p>Our first step on the roadmap to automatically building entities was defining a CLAUDE.md file, the rulebook that tells Claude how to work within this project. The file includes Makefile targets for entity setup, validation, testing, and creation, along with schema conformity rules that verify generated pipeline code actually matches what the entity definition declares. That last piece matters more than it might seem: given an entity such as Device, an LLM asked to build the pipeline could silently hallucinate different relationships, wrong column mappings, or a mismatched grain. The conformity checks catch that drift by comparing generated output against the canonical entity YAML before anything reaches production.</p><pre># CLAUDE.md<br><br>This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.<br><br>## Commands<br>```bash<br>make setup-uv       # Create .venv and install deps (recommended, requires uv)<br>make setup-venv     # Create .venv and install deps (pip)<br>make validate       # Validate all entity files against schema + custom rules<br>make test           # Run pytest test suite<br>make lint           # Run flake8 on scripts/ and tests/<br>make test-watch     # Run pytest in watch mode<br>make status         # Show environment status<br>```<br>Run a single test:<br>```bash<br>.venv/bin/pytest tests/test_entities_conform_to_schema.py::TestRealEntities::test_all_entities_are_valid -v<br>```<br>Without make (after activating venv):<br>```bash<br>python scripts/validate_entities.py   # Validate entities<br>pytest tests/ -v                      # Run tests<br>```<br>...</pre><h3>‍Building Skills</h3><p>Building skills came next. The agent needed a structured way to extract information before generating an entity. Essential questions — such as the entity’s domain, build strategy, and ownership — had to be answered upfront. Rather than relying on ad-hoc prompting, we built an add-entity-definition skill that codifies this questionnaire and drives the authoring process through Claude.</p><pre>---<br>name: add-entity-definition<br>description: Add or edit semantic-layer entity definitions (YAML). Follow the entity implementation process: gather input, draft/apply, validate, run pre-merge checklist, and ask until 100% clear. Use when the user wants to add a new entity or change an existing one.<br>---<br><br># Add entity definition<br>Use this skill when the user wants to **add a new entity** or **edit an existing entity** definition in the semantic layer. Follow the process below; do not guess - ask until everything is clear.<br>## When to use<br>- User says they want to **add** a new entity, create an entity definition, or add an entity definition for X.<br>- User says they want to **edit**, change, update, fix, or modify an existing entity; or has an `entities/**/entity.yaml` file open and asks for changes.<br>If the entity does not exist yet, use the **add** flow. If it already exists, use the **edit** flow.<br>---<br></pre><p>These are the skills that we built for the definition phase:</p><ul><li>add-entity-definition. Main skill for adding or editing entity YAML definitions. Orchestrates the full workflow from gathering input through validation and pre-merge checklist.</li><li>add-entity-definition-reference. Companion questionnaire template and quick-reference for add-entity-definition. Not independently triggered; called from the main skill.</li><li>resolve-entity-foreign-key. Sub-skill for FK resolution on entity relationships. Handles deterministic (bridge table, hash computation, waterfall) and probabilistic strategies. Invoked by add-entity-definition when a relationship requires FK resolution.</li></ul><p>With skills and agent rules in place, the first phase became simply a matter of prompting. A single input such as “I want to define an entity named Creative” kicks off the agentic workflow. Claude then drives the conversation, asking the questions needed to capture the entity’s domain, relationships, build strategy, and structure. The result is a completed entity.yaml artifact.</p><p>From there, the author commits the entity YAML file, opens a pull request, and once merged to main, Phase 2 begins automatically: the CI pipeline verifies the entity definition and stores it in the metadata layer.</p><h3>‍Phase 2: Building the Entity Metadata Registry</h3><p>The REST API handles write operations and CI/CD validation. But for the AI assistant Claude, we needed a different interface — the MCP.</p><p>We built an MCP server exposing exactly four tools:</p><ul><li>list_entities. Returns the full topology graph: entities + directed join edges.</li><li>list_tables. Cheap discovery; filter by source, target, or name.</li><li>get_entities. Batch entity fetch (one call for <em>N</em> names).</li><li>get_tables. Batch table fetch (one call for <em>N</em> names).</li></ul><h3>MCP Features: Right Context For Right Audience</h3><p>One of the most interesting design elements in the MCP layer is the ability to limit context based on who the audience is (who is interacting with the service).</p><p>The default audience returns a public metadata summary: column names, types, nullability, descriptions, and a small number of representative sample values per column. Profiling statistics and internal implementation columns are omitted. The goal is a clean, concise overview suited to exploration and discovery.</p><p>The full audience must be requested explicitly. It returns the complete context: all columns, all profiling statistics, and the full sample set. This level of detail is appropriate when the output will drive code generation or query optimization — tasks that require null rates, cardinality estimates, and concrete value examples to make correct decisions.</p><p>The controlling point here is that the full audience is never the default. An interactive agent asking “what entities exist?” doesn’t need 60 columns × 10 profiling fields × 10 sample values per entity. That’s hundreds of kilobytes of context the model doesn’t need and will likely misuse. The audience lens keeps token consumption proportional to the task. Keeping costs low in development required this segregation of duties between returning consumer based context.</p><pre>def get_entities(<br>    names: list[str],<br>    audience: Audience,<br>):</pre><p>When you call the discovery tool (list_entities), the response isn’t a flat list of names, it’s a graph. Consider a simplified response with three entities:</p><pre>{ #-- example of how entities could tie to joins --<br>  &quot;entities&quot;: [ #nodes<br>    { &quot;name&quot;: &quot;order&quot;,&quot;type&quot;: &quot;fact&quot;,&quot;domain&quot;: &quot;Commerce&quot; },<br>    { &quot;name&quot;: &quot;customer&quot;,&quot;type&quot;: &quot;dimension&quot;,&quot;domain&quot;: &quot;Commerce&quot;},<br>    { &quot;name&quot;: &quot;product&quot;,  &quot;type&quot;: &quot;dimension&quot;, &quot;domain&quot;: &quot;Catalog&quot;}<br>  ],<br>  &quot;joins&quot;: [ #edges<br>    { &quot;from&quot;: &quot;order&quot;, &quot;via&quot;: &quot;customer_id&quot;, &quot;to&quot;: &quot;customer&quot; },<br>    { &quot;from&quot;: &quot;customer&quot;, &quot;via&quot;: &quot;customer_id&quot;, &quot;to&quot;: &quot;order&quot; },<br>    { &quot;from&quot;: &quot;order&quot;, &quot;via&quot;: &quot;product_id&quot;,  &quot;to&quot;: &quot;product&quot;  },<br>    { &quot;from&quot;: &quot;product&quot;, &quot;via&quot;: &quot;product_id&quot;,  &quot;to&quot;: &quot;order&quot;  }<br>  ]<br>}</pre><p>Because edges are emitted in both directions, an agent planning a query that starts at a customer can immediately see that the order is reachable, without scanning the full edge list looking for any edge whose field happens to be a customer. The join path is readable directly from the node’s neighbors (i.e., join → order to customer via customer_id).</p><p>An agent planning a multi-entity query can call list_entities once to discover the full graph, identify which entities it needs, then call get_entities once with all <em>N</em> names. Two calls total, regardless of how many entities exist. It’s about saving time and tokens through the full roundtrip.</p><p>Entity definitions live in a separate Git repository as mentioned above. When an engineer opens a pull request adding or modifying an entity, we want to validate the definition against the real <a href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/">Unity Catalog</a> schema before merging so as not to discover that a referenced column doesn’t exist when the PySpark job runs in production.</p><h3>‍MCP Features: Dry Run Entity Validation and Testing</h3><p>The metadata registry supports an entity validation and testing feature on the generation endpoint (i.e., dry-run mode).</p><p>In dry-run mode, the engine performs the full harvest and strict-mode validation, confirming that the source table exists in our catalog and that every alias, column attribute reference resolves to a real column, but writes nothing to the context store. A non-empty failed array or a failed status fails the deployment pipeline.</p><h3>‍MCP Features: Draft Entity Support</h3><p>The service also supports “draft entities.” Engineers often need to merge an entity definition for review before its physical target table exists in Unity Catalog (often created by a later migration). By using the lifecycle: draft descriptor, the engine skips Unity Catalog introspection for the target table. It safely harvests sources, validates mappings, and writes the entity document without failing because the target does not yet exist.</p><p>The registry is the foundation for AI-driven semantic layer development. With it in place:</p><ul><li>A Claude session can call list_entities to understand the full topology, then get_entities on the entities it needs, and generate a correct multi-entity PySpark join with accurate column references and realistic filter values.</li><li>A CI pipeline validates entity definitions against live schema on every pull request, catching column mismatches before they ever reach production.</li><li>Entity context is always fresh. If a source table schema changes, the next entity read automatically includes the updated schema without regenerating the entity document.</li></ul><p>The broader principle is that schema metadata is the infrastructure for AI‑assisted data engineering. Without it, AI assistants are working blind. With a well-structured registry, they have the context they need to generate code that runs.</p><h3>Phase 3: Pipeline Creation and Deployment</h3><p>A semantic definition on its own produces no data. The entity compiler agent is what closes that gap, transforming entity definitions into physical tables that live and are queryable in the lakehouse. An entity definition describes what a concept is: its attributes, its sources, and how it changes over time. The compiler takes that definition and generates everything needed to run it, the PySpark job, the Delta table schema, the Databricks compute configuration, the Airflow orchestration task, and the unit tests. By the time compilation is done, the entity is ready to deploy with no further engineering work.</p><p>The compiler translates the business entity design into six concrete, consistent artifacts for each entity:</p><pre>Artifact: ETL job<br>File: jobs/build_&lt;entity&gt;_entity.py<br>Purpose: PySpark DataFrame transformations<br>────────────────────────────────────────<br><br>Artifact: Column model<br>File: model/&lt;entity&gt;.py<br>Purpose: Type-safe column constants (single contract)<br>────────────────────────────────────────<br>Artifact: Delta sink<br>File: delta/table/delta_&lt;entity&gt;_table.py<br>Purpose: Target schema and write strategy<br>────────────────────────────────────────<br>Artifact: Runtime config<br>File: resources/test.yaml, prod.yaml<br>Purpose: Source/sink bindings by environment<br>────────────────────────────────────────<br>Artifact: Terraform job<br>File: ci/terraform/environments/prod/terraform.tfvars<br>Purpose: Job provisioning<br>────────────────────────────────────────<br>Artifact: Airflow task<br>File: airflow/dags/.../build_semantic_layer.py<br>Purpose: Orchestration with upstream dependencies<br>────────────────────────────────────────</pre><p>Keeping these artifacts consistent automatically rather than by convention was the core motivation for building the compiler. The two systems (entity design and compilation) can also evolve independently: entity definitions can gain new fields or profiling annotations without touching pipeline code; the compiler can upgrade its merge mechanics without touching entity semantics.</p><h3>Compiling an Entity: What the Compiler Actually Does</h3><p>When a new entity is compiled, it fetches the full definition from the entity MCP server and works through five phases in sequence:</p><ul><li><strong>Validate</strong>. Verify all source tables are registered; confirm any required bridge tables already exist as compiled entities.<br><em>Output:</em> Pass / Halt with error.</li><li><strong>Generate core artifacts.</strong> Write ETL job, column model, and Delta sink, all sharing the same Column column constants.<br><em>Output: </em>3 Python files</li><li><strong>Sync config and deployment.</strong> Terraform job definition, and Airflow DAG with correct upstream dependencies.<br><em>Output:</em> 4 files updated</li><li><strong>Generate test fixtures.</strong> On sample data covering every scenario for the entity’s build strategy.<br><em>Output:</em> Test file + JSON fixtures</li><li><strong>Verify &amp; version.</strong> Run <em>n</em> (MINOR for new entities, PATCH for fixes).<br><em>Output:</em> Version string updated in 2 files</li></ul><h4>‍Entity Build Strategies</h4><p>Every entity declares a build strategy in its definition. The compiler generates different merge mechanics depending on which one is chosen:</p><ul><li>full_rebuild. Overwrites the entire table on every run.<br><em>Best for:</em> Small reference table lookup lists.</li><li>incremental. MERGEs on keys and updates a row when the incoming version is newer.<br><em>Best for:</em> Larger tables where history is not needed.</li><li>scd2. MERGEs while preserving history; old versions are closed and new versions are inserted.<br><em>Best for:</em> Slowly changing dimensions where knowing what was true at a point in matters.</li><li>passthrough. Compiler registers the entity for FK resolution and context generation but emits no ETL job. The target object is assumed to exist and is managed outside the compiler.<br><em>Best for:</em> Reference/lookup tables owned by another system.</li></ul><h3>Regression Testing</h3><p>Even well-designed models drift. Upstream changes happen, new values appear, relationships break. Without something actively checking, you usually find out weeks later when someone questions a downstream result or report. The regression process we built rests on a simple truth: the data contract itself is what needs to be tested. A versioned set of SQL assertions covers the four things that actually break trust: uniqueness, referential integrity, valid values, and freshness. When a pipeline change silently drops a column, it surfaces as a test failure. When an entity relationship breaks, it shows up before it reaches a consumer. When a new entity ships, its tests ship with it, so coverage compounds rather than decays. The shift is from reactive fixes to preventative guardrails. Single source of truth only works if it stays true on every run.</p><h3>A Note on CI/CD</h3><p>The compilation process is what closes the loop between intent and execution. A data modeler defines what an entity is, the compiler generates everything needed to run it, and the deployment pipeline carries it the rest of the way. When a merge lands on main, the full test suite runs, the wheel is published to Google Cloud Storage, Terraform applies the updated Databricks job definitions to production, and the Airflow DAGs are deployed to Cloud Composer, all without a manual step. From that point, the entity is considered “published”. It runs on its scheduled cadence, its history is tracked, and its data is queryable in the lakehouse. The semantic layer was designed so that the distance between defining a business concept and having it reflected in production data is as short as possible and the compiler is what makes that distance manageable.</p><h3>In Summary</h3><p>Taken together, the three phases form a closed loop: a designer defines a business concept through a guided agentic workflow, the definition is validated and stored in the metadata registry, and the compiler transforms it into a fully deployed, tested, production pipeline with multiple human-in-the-loop review processes to enable any code to deploy into a production environment. The entity authoring process stays low-code and intent-driven; the compilation and deployment process is deterministic and auditable. The diagram below captures how these pieces connect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5aldRd395kyQQ90GAsZ0tA.png" /><figcaption>Semantic Layer Design Overview</figcaption></figure><p>Here are the major features of the three-phased DT Semantic Layer build system:</p><ul><li><a href="https://mentormate.medium.com/designing-and-building-domain-specific-languages-for-business-flexibility-ce46b30398b0">Entity DSL</a>. YAML-based schema for defining dimensions and facts &amp;emdash; grain, attributes, sources, relationships, and build strategy in a single file.</li><li>Entity Compiler. Reads entity YAML and generates PySpark jobs plus Unity Catalog comments, keeping the physical catalog self-documenting.</li><li>MCP Server. Exposes the semantic layer as a graph API for AI tools —entity nodes type, domain, and target table; join edges are bidirectional for direct neighbor lookup.</li><li>Adoption Tracking. KPI dashboard measuring entity completion, DS user adoption, legacy-to-entity migration rates, and scheduled job consumer count.</li></ul><h3>The Results Are In: Data Science Team Sees 10‑20% Improvement Using Semantic Entities for Audience Embedding</h3><p>An audience embedding is a way to turn user behaviors, traits, and contexts into a list of numbers (a vector) so machine learning models can easily understand them. Instead of grouping users into rigid, manually created categories (like “Age 25–34”), embeddings map users into a multi-dimensional mathematical space based on how they actually behave.</p><p>To evaluate the embedding features built from user–app interaction data sourced through the semantic layer, we used AUC (Area Under the Curve): a standard metric that measures how well a model separates users who will take a target action (install, in-app event, revenue) from those who won’t. An AUC of 0.5 is random guessing; 1.0 is perfect. Even small gains in the 0.70–0.90 range translate directly into better-targeted campaigns and higher return on ad spend (ROAS).</p><p>We tested across 6 apps and three model types, keeping all settings identical to production and adding only the embedding features. The results were clear:</p><ul><li>Post-install KPI models showed the largest gains (+0.09 to +0.15 AUC — 21% gain). App1 went from 0.72 to 0.87, App2 from 0.70 to 0.80. These are exactly the models where the existing feature set had the most room to grow, and where embeddings ranked #1 or #2 in feature importance.</li><li>Revenue models improved by +0.09 to +0.14 AUC (20% gain).</li><li>Install models started from a higher baseline (0.85–0.98), so gains were smaller (from +0.02 to +0.10–2%10% gain) there’s simply less headroom for growth.</li></ul><h3>Driving Adoption</h3><p>A semantic layer only succeeds if teams actually use it. We focused on three primary levers:</p><ul><li><strong>Visibility</strong>: Exposing these entities through a Data Catalog (Unity Catalog) makes the entities easy to discover, understand, and utilize in production capacity. The Data Catalog serves as the discovery and governance backbone for the semantic layer.</li><li><strong>Proof of Value</strong>: Demonstrating improved performance and consistency in data science models and analytics. Previously, data science teams relied on legacy datasets built outside the semantic layer process. Once they migrated to the new semantic entities, their model results improved significantly, yielding a 10% to 20% performance increase across various use cases.</li><li><strong>Adoption Measurement: </strong>Building agents to create adoption measurement statistics from data lakehouse logs and downstream (non-semantic layer) job sources. This provides the transparency needed to establish where improvements need to be made.</li></ul><p>By aligning both engineering and data science workflows around shared definitions, we’ve seen improvements in the data science team’s ability to build embeddings for audiences. We are actively working cross functionally to continue driving adoption by making it easier for downstream consumers (think reporting, BI, and product analytics).</p><h3>Closing Thoughts and Considerations</h3><p>The DT Enterprise Data Platform touches billions of user records and active devices. At that scale, a misapplied filter or wrong join doesn’t produce a mildly inaccurate report, it produces a confidently wrong one, and decisions about which campaigns to fund, which markets to grow, and which device segments to target get made against it. The cost of ambiguous data isn’t theoretical; it compounds quietly across every team that builds on top of it.</p><p>The semantic layer is how we address that at the source. The geo entity means a location isn’t recorded as “India” for one team, “IN” for another, and “IND” for a third — it is a standardized identifier used universally across all teams. Similarly, the device entity means analyzing Android versus iOS performance no longer requires querying messy, raw manufacturer strings; instead, teams simply join on clean, unified identifiers. It is a complete standardization of how data across DT is connected and extended. These aren’t cosmetic improvements. They’re the difference between data that is merely available and data that is trustworthy.</p><p>What we’ve built is more than a data modeling initiative. Treating entity definitions as versioned, tested, executable assets — and using AI-driven automation to make authoring them fast and consistent — changes the relationship between data producers and consumers across the organization. Teams stop rebuilding the same concepts independently and start building on a shared foundation. The semantic layer is how we’re ensuring that as DT’s scale grows, so does our confidence in the data underneath it.</p><h4>This piece was written by:</h4><ul><li><strong>Dan Ferrante</strong>— Director, Data Engineering at DT</li><li><strong>Krystian Kurek </strong><em>— </em>Senior Data Scientist at DT</li><li><strong>Kyrylo Yershov</strong>— Lead Data Engineer at DT</li><li><strong>Sarit Banerjee </strong>— Quality Strategy Lead at DT</li><li><strong>VJ Davey </strong>— Senior Data Engineer at DT</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa7eaedd5459" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dt-engineering-blog/building-the-dt-semantic-layer-creating-a-single-source-of-truth-with-agent-driven-data-modeling-fa7eaedd5459">Building the DT Semantic Layer: Creating a Single Source of Truth with Agent-Driven Data Modeling</a> was originally published in <a href="https://medium.com/dt-engineering-blog">DT Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ONNX at Auction Speed: Part 2 — Building the Pipeline for Real-Time Bidding]]></title>
            <link>https://medium.com/dt-engineering-blog/onnx-at-auction-speed-part-2-building-the-pipeline-for-real-time-bidding-4280a425cf9c?source=rss-25681449fffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4280a425cf9c</guid>
            <category><![CDATA[onnx]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[programmatic-advertising]]></category>
            <category><![CDATA[real-time-bidding]]></category>
            <category><![CDATA[ml-model-deployment]]></category>
            <dc:creator><![CDATA[DT Engineering]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 13:52:50 GMT</pubDate>
            <atom:updated>2026-10-20T09:38:53.058Z</atom:updated>
            <content:encoded><![CDATA[<p><em>by Gabriel Gurvich, Julia Stachowiak, Ron Elias, Yuval Eckstein</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0-_6eThezK63HNllAe7Dg.jpeg" /></figure><p>In <a href="https://medium.com/dt-engineering-blog/onnx-at-auction-speed-part-1-why-we-rebuilt-our-bidding-pipeline-around-one-file-15381c39d7ca">Part 1</a>, we covered why we moved to ONNX: a single portable pipeline file eliminates the training-serving skew that comes from reimplementing preprocessing across languages, and it consolidates ownership so data science ships one validated artifact instead of a handoff. Here, we build one of those pipelines end to end, on real (anonymized) auction data from our exchange.</p><h3>The Dataset: DT Auction Bid Price Prediction</h3><p>For this walkthrough we use the dataset from our public <a href="https://www.kaggle.com/competitions/digital-turbine-auction-bid-price-prediction/overview">Kaggle competition</a>. This is real (anonymized) auction data from our exchange, and it gives us a natural use case: predicting the winning bid price (winBid) for an auction given the features known at bid time.</p><p><strong>Key columns:</strong></p><ul><li>‍unitDisplayType, string. Ad placement type: banner, interstitial, or rewarded.</li><li>‍countryCode, string. ISO country code of the user (e.g., US, DE, JP).</li><li>‍eventTimestamp, long. Epoch timestamp in milliseconds.</li><li>‍sentPrice, double. The bid floor price sent by the exchange.</li><li>‍winBid, double. Target: the actual auction clearing price.</li></ul><p>This is a regression problem: predict a continuous bid price value. The task sounds simple, but the raw distributions are far from well-behaved — a reality we address with feature engineering.</p><p>Note that the model we build here is a toy example designed to illustrate the ONNX pipeline pattern. It is not intended to represent our production bid-price model, which incorporates many additional signals and a more sophisticated architecture.</p><p><strong>From “Why” to “How”:</strong> Having covered the problem — training-serving skew, ownership fragmentation, and the latency wall — the rest of this post is a concrete technical walkthrough. We have deliberately kept the example simple to make the ONNX workflow as legible as possible. Our production pipelines are significantly more complex, but the same patterns apply at scale. Along the way, we highlight Key Takeaways and common pitfalls encountered in large-scale data environments.</p><h3>Feature Engineering in Spark</h3><p>All feature engineering runs inside a Spark Pipeline on a Databricks cluster. This is important for two reasons: (1) the data volume requires distributed processing, and (2) the fitted pipeline stages (e.g., StringIndexerModel, OneHotEncoderModel) carry the vocabulary and encoding metadata that we will later bake directly into the ONNX graph.</p><h3>Time Features: Cyclic Hour Encoding</h3><p>To illustrate how time-based features integrate into the pipeline, consider a common ad-tech challenge: representing time-of-day in a way that preserves its cyclic nature. Many techniques exist (embedding lookup, linear binning, learned positional encodings), but we demonstrate a sine/cosine approach that is simple, compact, and native to ONNX operators. We extract the hour of day from eventTimestamp (milliseconds epoch) and convert it to two cyclic features:</p><pre>hour_sin = sin(2π × hour / 24)<br>hour_cos = cos(2π × hour / 24)</pre><p>Why cyclic? A naive integer hour (0–23) encodes hour 23 and hour 0 as far apart when they are actually adjacent. Sine and cosine together form a continuous cyclic representation: the angular distance between any two hours is proportional to their true temporal distance on a 24-hour clock. Using periodic functions as feature representations for time is <a href="https://arxiv.org/abs/1907.05321">well-motivated in the ML literature</a>.</p><pre># From FeatureEngineering.add_time_features() in utils.py<br>df = (<br>    df<br>    .withColumn(sin_col, F.sin(F.lit(2.0 * math.pi) * F.col(hour_col) / F.lit(24.0)))<br>    .withColumn(cos_col, F.cos(F.lit(2.0 * math.pi) * F.col(hour_col) / F.lit(24.0)))<br>)</pre><p>These two features encode both <em>amplitude</em> and phase of the daily cycle, so XGBoost can learn patterns like “high bids in the late evening” without needing to hard-code time boundaries.</p><h3>Power Transform: Taming Skewed Bid Distributions</h3><p>To illustrate how feature engineering handles skewed distributions — a challenge endemic to ad-tech data — consider the winBid and sentPrice columns. While many normalisation techniques exist (log transform, quantile transform, rank-based scaling), we demonstrate a power transform approach that fits naturally into our ONNX postprocessing story: because the transform is invertible, we can encode both the forward and inverse steps into the same graph. Auction bid prices are heavily right-skewed. A small number of premium placements command prices an order of magnitude higher than the median. This kind of distribution is a known challenge for tree-based models: the model can spend most of its splits trying to resolve the long tail rather than learning general patterns in the bulk of the distribution.</p><p>The raw winBid distributions by placement type (top row) show exactly this shape: nearly all values cluster near zero with a long tail to the right. The power-transformed versions (bottom row) are far more symmetric and easier for a model to learn from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GYFu_17DOUKhuf--.png" /><figcaption>‍<em>Figure 1: Raw winBid distributions by placement type — severely right-skewed.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M0Je5PLAS3a3rclS.png" /><figcaption>‍<em>Figure 2: Power-transformed winBid distributions — approximately symmetric.</em></figcaption></figure><p>We apply a per-placement-type power transform of the form <em>x^(1/n)</em> where <em>n</em> is tuned per placement category. This is a restricted form of the <a href="https://www.jstor.org/stable/2984418">Box-Cox power transform family</a>, constrained to positive-only inputs (which bid prices always are). <a href="https://doi.org/10.3389/frai.2022.877569">Recent work</a> on applying power transforms in machine learning contexts confirms that such transformations can significantly improve model accuracy by normalizing the input distribution.</p><p>The exponents are defined in config.py and can be fine-tuned as part of a hyperparameter tuning process:</p><pre>POWER_N: Dict[str, float] = {<br>    &quot;banner&quot;:       4.0,<br>    &quot;interstitial&quot;: 3.0,<br>    &quot;rewarded&quot;:     3.0,<br>}<br>POWER_N_DEFAULT: float = 3.0</pre><p>The transform is applied both to the input feature sentPrice and to the target winBid. Critically, since the model learns in transformed space, any prediction it outputs is also in transformed space — we must invert the transform at serving time to get a prediction in the original bid-price scale. This is the postprocessing step we will encode into the ONNX graph.</p><pre># From FeatureEngineering.power_transform() in utils.py<br>for c in cols:<br>    df = df.withColumn(f&quot;{c}_orig&quot;, F.col(c))  # keep original for comparison<br>    exponent = F.lit(1.0 / default_n)<br>    for group, n in power_n_by_group.items():<br>        exponent = F.when(<br>            F.col(group_col) == F.lit(group),<br>            F.lit(1.0 / float(n))<br>        ).otherwise(exponent)<br>    df = df.withColumn(<br>        c,<br>        F.pow(F.greatest(F.col(c), F.lit(0.0)), exponent)<br>    )</pre><p>‍<strong>Categorical Encoding</strong></p><p><strong>Placement type </strong>(unitDisplayType) goes through StringIndexer → OneHotEncoder (with dropLast=True to avoid dummy variable collinearity). Because the Spark encoder uses handleInvalid=&quot;keep&quot;, it reserves an extra bucket for unseen labels at serving time — a useful safety net in production where new placement types may appear.</p><p><strong>Country code </strong>(countryCode) is encoded with StringIndexer only — producing a float index (<em>0, 1, 2, …</em>) rather than a one-hot vector. Given that there can be 200+ distinct countries in our data, one-hot encoding would add excessive dimensionality for what is essentially a ranking signal. The index encoding still lets XGBoost learn country-specific effects via its tree splits.</p><h3>Feature Vector: A Deterministic, Ordered Contract</h3><p>The assembled feature vector follows a fixed order defined as a single source of truth in config.py:</p><pre>FEATURE_INPUT_COLS_ORDER: List[str] = [<br>    &quot;placement_ohe&quot;,   # one-hot vector, variable width depending on # placement types<br>    &quot;country_idx&quot;,     # scalar float<br>    &quot;hour_sin&quot;,        # scalar float<br>    &quot;hour_cos&quot;,        # scalar float<br>    &quot;sentPrice&quot;,       # scalar float (power-transformed)<br>]</pre><p>This order is critical. The Spark VectorAssembler produces features in this order, the XGBoost model is trained on features in this order, and the ONNX preprocessing graph must produce a features tensor in exactly this order. Any mismatch would silently feed the wrong values to the wrong tree splits — a bug that would be very hard to detect.</p><p><strong>Key takeaway:</strong> Define feature order once, in a config, and reference it everywhere: training, metadata export, and ONNX graph construction.</p><h3>Training a Distributed XGBoost Model</h3><p>With features engineered, we train a gradient-boosted tree model — specifically SparkXGBRegressor from the XGBoost PySpark API. We chose this model family because it represents the “classical ML” archetype well: non-neural, tree-based, and <a href="https://dl.acm.org/doi/10.1145/2020408.2020604">widely used in ad-tech prediction tasks</a>. The ONNX export pattern we demonstrate here applies equally to other classical ML frameworks. Our production pipelines mirror these same patterns at a larger scale.</p><p>The full pipeline is:</p><pre>Pipeline(stages=[<br>    StringIndexer(inputCol=&quot;unitDisplayType&quot;, outputCol=&quot;placement_idx&quot;, handleInvalid=&quot;keep&quot;),<br>    OneHotEncoder(inputCols=[&quot;placement_idx&quot;], outputCols=[&quot;placement_ohe&quot;], handleInvalid=&quot;keep&quot;),<br>    StringIndexer(inputCol=&quot;countryCode&quot;, outputCol=&quot;country_idx&quot;, handleInvalid=&quot;keep&quot;),<br>    VectorAssembler(inputCols=FEATURE_INPUT_COLS_ORDER, outputCol=&quot;features&quot;),<br>    SparkXGBRegressor(<br>        features_col=&quot;features&quot;,<br>        label_col=&quot;winBid&quot;,<br>        validation_indicator_col=&quot;is_validation&quot;,<br>        objective=&quot;reg:squarederror&quot;,<br>        max_depth=6,<br>        eta=0.08,<br>        num_round=200,<br>        subsample=0.8,<br>        colsample_bytree=0.8,<br>        callbacks=[CollectEval(period=25)],<br>    )<br>])</pre><p>The validation_indicator_col mechanism deserves mention. Rather than a random train/val split, we use a time-based split: the most recent 20% of events (by eventTimestamp) become the validation set. This mimics the real-world scenario where a model trained on historical data is evaluated on future auctions. Train/test splitting is a domain unto itself — random splits, time-based splits, group-based splits, and stratified variants each reflect different assumptions about the problem. We use a time-based split here because it best reflects our production environment and the real-world temporal dynamics discussed earlier. The CollectEval callback captures RMSE at every boosting round so we can visualize the training curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VA8d2j1ZRr_OISAz.png" /><figcaption>‍<em>Figure 3: Train and validation RMSE over 200 boosting rounds. Validation loss stabilizes around round 40, indicating good generalization.</em></figcaption></figure><h3>Exporting the Preprocessing Metadata</h3><p>Before building the ONNX graph, we export a metadata JSON file that captures all the information needed to reconstruct the preprocessing steps exactly. This file is the contract between the Python training code and the ONNX builder.</p><p>From PreprocMetadataWriter.build_metadata() in utils.py:</p><pre>{<br>  &quot;opset&quot;: 17,<br>  &quot;feature_order&quot;: [&quot;placement_ohe&quot;, &quot;country_idx&quot;, &quot;hour_sin&quot;, &quot;hour_cos&quot;, &quot;sentPrice&quot;],<br>  &quot;power_transform&quot;: {<br>    &quot;enabled&quot;: true,<br>    &quot;group_col&quot;: &quot;unitDisplayType&quot;,<br>    &quot;power_n_by_group&quot;: {&quot;banner&quot;: 4.0, &quot;interstitial&quot;: 3.0, &quot;rewarded&quot;: 3.0},<br>    &quot;default_n&quot;: 3.0,<br>    &quot;cols&quot;: [&quot;sentPrice&quot;, &quot;winBid&quot;]<br>  },<br>  &quot;target_normalization&quot;: {&quot;enabled&quot;: true},<br>  &quot;notes&quot;: {<br>    &quot;placement_ohe_expands&quot;: true,<br>    &quot;country_idx_is_float&quot;: true,<br>    &quot;hour_features&quot;: [&quot;hour_sin&quot;, &quot;hour_cos&quot;]<br>  }<br>}</pre><p>Every coefficient and label mapping the ONNX builder uses comes from this file (and from the fitted pipeline stages). Nothing is hardcoded in the ONNX builder itself. If you retrain the model on new data with different country vocabulary or different power-transform exponents, you regenerate the metadata and rebuild the graph — the ONNX construction code stays unchanged.</p><p><strong>Key takeaway:</strong> The metadata JSON is the single source of truth for all preprocessing parameters. It bridges training and graph construction and should be versioned alongside the model artifact.</p><h3>Building the ONNX Graph: Four Stages</h3><p>Now for the heart of the post. We construct the ONNX graph in four stages, each building on the previous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/0*oOG_Uu6IhC4yzm5g.png" /><figcaption>‍<em>Figure 4: Four-stage ONNX graph construction: preprocessing (Stage 1) feeds into the XGBoost tree ensemble (Stage 2); the two are merged into a single graph (Stage 3); the inverse power transform is appended to produce final bid prices (Stage 4).</em></figcaption></figure><h3>Stage 1: Preprocessing Graph</h3><p>The preprocessing graph is built by hand using the ONNX helper API, directly translating each Spark pipeline stage into ONNX operators. The OnnxBlogpostGraphBuilder class in utils.py handles this.</p><p><strong>Label encoding (placement type):</strong></p><p>The fitted StringIndexerModel carries the vocabulary as an ordered list (labels). We create an ai.onnx.ml.LabelEncoder node that maps each string to a float index, with known labels assigned indices <em>0, 1, …</em> and unseen strings assigned a default_float index (equal to the vocabulary size — the “unknown” bucket):</p><pre>helper.make_node(<br>    &quot;LabelEncoder&quot;,<br>    inputs=[&quot;unitDisplayType&quot;],<br>    outputs=[&quot;placement_idx&quot;],<br>    domain=&quot;ai.onnx.ml&quot;,<br>    keys_strings=[&quot;banner&quot;, &quot;interstitial&quot;, &quot;rewarded&quot;],<br>    values_floats=[0.0, 1.0, 2.0],<br>    default_float=3.0,   # unseen label → index 3<br>)</pre><p>‍<strong>One-hot encoding:</strong></p><p>We Cast the float index to int64, Reshape to a 1-D vector, then apply the ONNX OneHot operator. If dropLast=True (our default, to avoid collinearity), we append a Slice node that removes the last column:</p><pre># Cast → Reshape → OneHot → Slice (dropLast)<br>helper.make_node(&quot;Cast&quot;,    [&quot;placement_idx&quot;],        [&quot;placement_idx__i64&quot;],   to=TensorProto.INT64)<br>helper.make_node(&quot;Reshape&quot;, [&quot;placement_idx__i64&quot;,    &quot;shape_minus1&quot;],          [&quot;placement_idx__1d&quot;])<br>helper.make_node(&quot;OneHot&quot;,  [&quot;placement_idx__1d&quot;,     &quot;ohe_depth&quot;, &quot;ohe_vals&quot;], [&quot;placement_ohe__raw&quot;], axis=-1)<br>helper.make_node(&quot;Slice&quot;,   [&quot;placement_ohe__raw&quot;,    &quot;starts&quot;, &quot;ends&quot;, &quot;axes&quot;],[&quot;placement_ohe&quot;])</pre><p>‍<strong>Country code:</strong> Same LabelEncoder pattern, without the subsequent OneHot step.</p><p><strong>Cyclic hour features:</strong></p><pre>helper.make_node(&quot;Div&quot;, [&quot;hour&quot;,      &quot;const_24&quot;],   [&quot;hour_norm&quot;])<br>helper.make_node(&quot;Mul&quot;, [&quot;hour_norm&quot;, &quot;const_2pi&quot;],  [&quot;hour_angle&quot;])<br>helper.make_node(&quot;Sin&quot;, [&quot;hour_angle&quot;],              [&quot;hour_sin&quot;])<br>helper.make_node(&quot;Cos&quot;, [&quot;hour_angle&quot;],              [&quot;hour_cos&quot;])</pre><p>Constants (<em>24.0, 2π</em>) are stored as ONNX initializers — tensor constants baked into the graph.</p><p>Placement-dependent power transform for<strong> </strong>sentPrice:</p><p>This is the most interesting piece. Rather than a fixed exponent, we need <em>n</em> to vary by placement type. The solution is a lookup table (initializer) indexed by the placement index:</p><pre># n_table = [4.0, 3.0, 3.0, 3.0]  (banner, interstitial, rewarded, unknown)<br>helper.make_node(&quot;Cast&quot;,    [&quot;placement_idx&quot;],               [&quot;placement_idx__i64__pow&quot;],  to=TensorProto.INT64)<br>helper.make_node(&quot;Reshape&quot;, [&quot;placement_idx__i64__pow&quot;,      &quot;shape_minus1&quot;],              [&quot;placement_idx__1d__pow&quot;])<br>helper.make_node(&quot;Gather&quot;,  [&quot;placement_power_n_table&quot;,      &quot;placement_idx__1d__pow&quot;],    [&quot;placement_power_n__1d&quot;],  axis=0)<br>helper.make_node(&quot;Reshape&quot;, [&quot;placement_power_n__1d&quot;,        &quot;shape_n1&quot;],                  [&quot;placement_power_n__2d&quot;])<br># exponent = 1/n<br>helper.make_node(&quot;Div&quot;,     [&quot;const_one&quot;, &quot;placement_power_n__2d&quot;], [&quot;placement_power_inv_n&quot;])<br># sentPrice^(1/n)<br>helper.make_node(&quot;Max&quot;,     [&quot;sentPrice&quot;, &quot;const_zero&quot;],     [&quot;sentPrice__nonneg&quot;])<br>helper.make_node(&quot;Pow&quot;,     [&quot;sentPrice__nonneg&quot;,            &quot;placement_power_inv_n&quot;],      [&quot;sentPrice__pow&quot;])</pre><p><strong>Feature vector concatenation:</strong></p><p>Finally, all feature tensors are concatenated in the canonical order:</p><pre>helper.make_node(<br>    &quot;Concat&quot;,<br>    inputs=[&quot;placement_ohe&quot;, &quot;country_idx&quot;, &quot;hour_sin&quot;, &quot;hour_cos&quot;, &quot;sentPrice__pow&quot;],<br>    outputs=[&quot;features&quot;],<br>    axis=1,<br>)</pre><p>The feature order must exactly match the FEATURE_INPUT_COLS_ORDER used during Spark training. Any deviation is a silent correctness bug. Because both the Spark assembler and the ONNX builder read from the same config constant, this is enforced automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*98uEUqwtcOBKtfS-.png" /><figcaption>‍<em>Figure 5: preprocess.onnx. The four raw inputs (unitDisplayType, countryCode, hour, sentPrice) enter from the left. unitDisplayType passes through a LabelEncoder, then Cast → Reshape → OneHot → Slice (dropLast) for one-hot expansion. countryCode goes through a separate LabelEncoder directly to a float index. hour is divided by 24, multiplied by 2π, and split into Sin and Cos branches. sentPrice is clamped via Max and raised to a placement-dependent exponent looked up via Gather. All branches converge in the final Concat node, producing the features tensor.</em></figcaption></figure><h3>Stage 2: XGBoost ONNX Graph</h3><p>The trained XGBoost booster is converted via ONNXMLTools:</p><pre>from onnxmltools import convert_xgboost<br>from onnxmltools.convert.common.data_types import FloatTensorType<br><br>xgb_onnx = convert_xgboost(<br>    booster,<br>    initial_types=[(&quot;features&quot;, FloatTensorType([None, feature_dim]))],<br>)</pre><p>This produces a graph containing a TreeEnsembleRegressor node (from the ai.onnx.ml domain) that encodes all the tree structure and leaf values. The input is our features tensor of shape <em>[N, F]</em>; the output is the predicted value in normalized (power-transformed) space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*fUDQcGr5XNOu6xz4.png" /><figcaption>‍<em>Figure 6: xgboost.onnx. A single TreeEnsembleRegressor node encodes all 200 boosted trees. It takes the features tensor (</em>?×8<em>) as input and outputs variable (</em>?×1<em>) — the model’s prediction in power-transformed space.</em></figcaption></figure><h3>Stage 3: Merging the Graphs</h3><p>onnx.compose.merge_models stitches the two graphs together by wiring the preprocessing output (features) to the XGBoost input:</p><pre>xgb_input_name = xgb_onnx.graph.input[0].name   # e.g., &quot;features&quot; or &quot;input&quot;<br>merged = merge_models(<br>    preproc_onnx,<br>    xgb_onnx,<br>    io_map=[(&quot;features&quot;, xgb_input_name)],<br>)</pre><p>After merging, features is no longer a graph-level output — it becomes an internal tensor, invisible from outside. The merged graph exposes only the original raw inputs and the XGBoost prediction output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pt99UNxWQiOrnu7Q.png" /><figcaption>‍<em>Figure 7: merged.onnx. The preprocessing and XGBoost sub-graphs are stitched into a single end-to-end graph via merge_models. The features tensor that connected them is now an internal edge — callers only see the original raw inputs on the left and the normalized prediction (variable) on the right.</em></figcaption></figure><h3>Stage 4: Inverse Power Transform (Postprocessing)</h3><p>The model was trained on <em>winBid^(1/n)</em>, so its output is in transformed space. To get a bid price in the original scale we append the inverse: <em>max(0, pred)^n</em>, where <em>n</em> is again looked up per placement type using the same initializer table approach:</p><pre># Tensors with the __inv suffix belong to the inverse-transform block<br>g.node.extend([<br>    helper.make_node(&quot;Cast&quot;,    [&quot;placement_idx&quot;],              [&quot;placement_idx__i64__inv&quot;],  to=TensorProto.INT64),<br>    helper.make_node(&quot;Reshape&quot;, [&quot;placement_idx__i64__inv&quot;,     &quot;shape_minus1__inv&quot;],         [&quot;placement_idx__1d__inv&quot;]),<br>    helper.make_node(&quot;Gather&quot;,  [&quot;placement_power_n_table__inv&quot;,&quot;placement_idx__1d__inv&quot;],    [&quot;placement_power_n__1d__inv&quot;], axis=0),<br>    helper.make_node(&quot;Reshape&quot;, [&quot;placement_power_n__1d__inv&quot;,  &quot;shape_n1__inv&quot;],             [&quot;placement_power_n__2d__inv&quot;]),<br>])<br>g.node.append(helper.make_node(&quot;Reshape&quot;, [pred_name, &quot;shape_n1__inv&quot;], [pred_2d]))<br>g.node.append(helper.make_node(&quot;Max&quot;,     [pred_2d,   &quot;const_zero__inv&quot;],                    [pred_nonneg]))<br>g.node.append(helper.make_node(&quot;Pow&quot;,     [pred_nonneg, &quot;placement_power_n__2d__inv&quot;],       [&quot;prediction_final&quot;]))</pre><p>The final graph has two outputs: the raw normalized prediction (useful for debugging against the Spark evaluator’s RMSE), and prediction_final — the bid price in the original dollar-scale that the backend system actually uses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*50EBd64pKbGZv4n9.png" /><figcaption>‍<em>Figure 8: final_graph.onnx. The complete end-to-end pipeline. The preprocessing and XGBoost nodes from Stage 3 occupy the left and centre of the graph. The inverse power transform block (top-right) reads the placement index a second time via Cast → Reshape → Gather → Reshape to look up the inverse exponent, then applies Reshape → Max → Pow to the XGBoost output, producing prediction_final in the original bid-price scale alongside variable for diagnostics.</em></figcaption></figure><h3>Opset Versioning: A Critical Gotcha</h3><p>When you merge two ONNX graphs, they must agree on the opset version for each domain. A preprocessing graph built with onnx==1.15 defaults and an XGBoost graph produced by ONNXMLTools may carry different opset_import entries — ONNX will reject the merge with an explicit error rather than silently producing a broken graph.</p><p>The root cause is tooling lag: converter libraries like ONNXMLTools do not always keep pace with ONNX releases. This creates two practical problems. First, mismatched defaults between tools cause merge errors that must be resolved by normalizing opsets manually. Second, if a tool pins an older opset, you cannot use recently-added operators without bumping versions yourself — for example, StringConcat (added in main opset 20) or TreeEnsemble (added in ai.onnx.ml opset 5 in 2024, with significant improvements over the legacy TreeEnsembleRegressor and TreeEnsembleClassifier nodes, though not yet supported by ONNXMLTools).</p><p>We solve this with a single helper applied to every sub-graph before any merge:</p><pre># apply_onnx_model_config() in utils.py<br>model = set_domain_opset(model, &quot;&quot;,            cfg.ONNX_OPSET)        # 17<br>model = set_domain_opset(model, &quot;ai.onnx.ml&quot;,  cfg.ML_OPSET_VERSION)  # 3<br>model.ir_version = int(cfg.IR_VERSION)                                # 10</pre><p>All three values — ONNX_OPSET, ML_OPSET_VERSION, and IR_VERSION — are defined in config.py and applied to every graph: the hand-built preprocessing graph, the ONNXMLTools-converted XGBoost graph, and the merged/final graph. This guarantees that merge_models always operates on compatible graphs. A secondary benefit: explicit pinning prevents accidental version bumps when Python packages are upgraded — important because the inference server runtime is managed independently and may not yet support newer opsets or IR versions. ONNX Runtime maintains backward compatibility with older opsets and IR versions, so pinning conservatively is a safe default.</p><p><strong>Key takeaway:</strong> Always normalize the opset and IR version across all sub-graphs before merging. Define these as config constants and apply them centrally rather than relying on converter defaults.</p><h3>Debugging the ONNX Graph (or: How We Taught an AI Agent to Read Our Nodes)</h3><p>An ONNX graph can contain dozens or hundreds of nodes. Without good tooling, debugging one is painful. We use three tools.</p><h3>describe_graph: Human-Readable Graph Summary</h3><p>We wrote a describe_graph utility that prints a compact summary of any ONNX model, making it easy to quickly understand the graph structure and share it for debugging — including with AI assistants:</p><pre>describe_graph(final_graph, &quot;final&quot;, max_nodes=120, verbosity=1)</pre><p>Output:</p><pre>===== final =====<br>graph_name=blogpost_preprocess<br>nodes=47  initializers=18<br><br>inputs:<br>  - unitDisplayType<br>  - countryCode<br>  - hour<br>  - sentPrice<br><br>outputs:<br>  - variable       ← XGBoost raw output (normalized space)<br>  - prediction_final  ← bid price in original scale</pre><p>At verbosity=2 it lists every node with its op type, letting you trace the full computation graph as text. This is particularly useful when pasting the output into a chat with an AI assistant to diagnose a construction issue or an unexpected output shape.</p><p>We also provide print_preprocess_data_flow() which prints a detailed breakdown of each feature segment:</p><pre>PREPROCESS DATA FLOW (for debugging)<br>--- 5) FEATURE ORDER AND DIMENSIONS ---<br>  placement_ohe: dim = 3  (cumulative start index: 0)<br>  country_idx:   dim = 1  (cumulative start index: 3)<br>  hour_sin:      dim = 1  (cumulative start index: 4)<br>  hour_cos:      dim = 1  (cumulative start index: 5)<br>  sentPrice:     dim = 1  (cumulative start index: 6)<br>  Total feature dim F = 7</pre><p>This tells you exactly which index in the features vector corresponds to which input — critical for verifying that the tree model received the right data.</p><h3>Netron: Visual Graph Explorer</h3><p><a href="https://netron.app/">Netron</a> is a browser-based ONNX graph visualizer. Upload any .onnx file and get an interactive, clickable node graph.</p><p>It is invaluable for:</p><ul><li><strong>Tracing the preprocessing path:</strong> open preprocess.onnx, follow unitDisplayType through LabelEncoder → Cast → Reshape → OneHot → Slice → Concat</li><li><strong>Inspecting initializer values:</strong> click the placement_power_n_table initializer to verify the <em>n</em>-values match your config</li><li><strong>Finding the inverse transform block</strong>: in final_graph.onnx, search for the Pow node at the bottom of the graph and confirm it reads from the same placement index as the forward transform</li></ul><p>A recommended workflow is to export all four pipeline stages (Stage 1–4) as separate .onnx files, which we do with onnx.save(), and inspect them one by one in Netron to validate each layer of the pipeline before running inference.</p><h3>run_and_dump: Single-Row Trace</h3><p>The run_and_dump helper runs an ONNX Runtime session on a test input and prints dtype, shape, and a value preview for each output tensor:</p><pre>inputs_single = {<br>    &quot;unitDisplayType&quot;: np.array([[&quot;banner&quot;]]).astype(object),<br>    &quot;countryCode&quot;:     np.array([[&quot;US&quot;]]).astype(object),<br>    &quot;hour&quot;:            np.array([[12.0]], dtype=np.float32),<br>    &quot;sentPrice&quot;:       np.array([[0.7]],  dtype=np.float32),<br>}<br><br>preproc_results = run_and_dump(preproc_session, inputs_single, [&quot;features&quot;], &quot;preprocess&quot;)</pre><p>Output:</p><pre>===== RUN: preprocess =====<br>features: dtype=float32, shape=(1, 7), preview=[1. 0. 0. 47. 0. 1. 0.5916]</pre><p>Combined with print_preprocess_feature_vector, you get a named breakdown of exactly what values the model receives for each feature slot — making it easy to check that &quot;banner at hour 12 with sentPrice=0.7&quot; produces the expected one-hot encoding and transformed price.</p><p><strong>Key takeaway:</strong> describe_graph + Netron + run_and_dump give you three complementary views of the graph: structural, visual, and numerical. Use all three when debugging a new pipeline.</p><h3>Inference with ONNX Runtime</h3><p>Once the final_graph.onnx is exported, Python is no longer in the serving path. The graph runs in ONNX Runtime (ORT) — a lightweight, cross-platform inference engine optimized for low-latency prediction, with backends in Python, Java, C#, C++, and JavaScript.</p><h3>Creating a Session</h3><pre>import onnxruntime as ort<br><br>session = ort.InferenceSession(<br>    &quot;final_graph.onnx&quot;,<br>    providers=[&quot;CPUExecutionProvider&quot;]<br>)</pre><p>Note that the snippets in this section use Python for legibility — the same .onnx file is consumed via an identical session.run() call from any supported runtime. In production, latency-sensitive inference typically runs outside of Python; our backend uses Rust via the ort crate, which wraps the same C++ ONNX Runtime under the hood. As an added benefit, ONNX Runtime applies graph-level optimizations at session load time that can make even Python inference faster than calling the original model’s native predict() directly.</p><p>The session is stateless and thread-safe. In a production service, create it once at startup and call session.run() for each incoming auction request. However, in some language bindings, e.g., Rust’s ort crate, run() requires exclusive mutable access — the standard pattern there is a session pool with one instance per worker thread. ORT also supports hardware-accelerated providers — swap &quot;CPUExecutionProvider&quot; for &quot;CUDAExecutionProvider&quot;, &quot;TensorrtExecutionProvider&quot;, or &quot;CoreMLExecutionProvider&quot; depending on your inference hardware, with no changes to the graph or calling code.</p><h3>Single-Row Inference (Online Bidding)</h3><p>In a real-time auction, one request arrives at a time. The input tensors have shape <em>[1, 1]</em> for both string and float inputs:</p><pre>inputs_single = {<br>    &quot;unitDisplayType&quot;: np.array([[&quot;banner&quot;]]).astype(object),<br>    &quot;countryCode&quot;:     np.array([[&quot;US&quot;]]).astype(object),<br>    &quot;hour&quot;:            np.array([[12.0]], dtype=np.float32),<br>    &quot;sentPrice&quot;:       np.array([[0.7]],  dtype=np.float32),<br>}<br><br>outputs = session.run(None, inputs_single)<br>pred_normalized    = outputs[0]  # shape (1,)   - raw XGBoost score in transformed space, for diagnostics<br>prediction_final   = outputs[1]  # shape (1, 1) - bid price in original scale (note: 2D, unlike the 1D xgb output)<br>print(f&quot;Predicted bid: {prediction_final[0][0]:.4f}&quot;)</pre><h3>Batch Inference (Offline Scoring / Shadow Mode)</h3><p>The exact same session and exact same graph handle batches of arbitrary size <em>N</em>. No code changes, no graph modifications:</p><pre>inputs_batch = {<br>    &quot;unitDisplayType&quot;: np.array([[&quot;interstitial&quot;], [&quot;banner&quot;], [&quot;rewarded&quot;]]).astype(object),<br>    &quot;countryCode&quot;:     np.array([[&quot;US&quot;], [&quot;CA&quot;], [&quot;DE&quot;]]).astype(object),<br>    &quot;hour&quot;:            np.array([[12.0], [4.0], [20.0]], dtype=np.float32),<br>    &quot;sentPrice&quot;:       np.array([[0.7],  [0.3], [1.2]],  dtype=np.float32),<br>}<br><br>batch_outputs = session.run(None, inputs_batch)<br>predictions = batch_outputs[1]  # shape (3, 1)‍</pre><p>This batching is not just a convenience — ONNX Runtime executes all preprocessing nodes and the tree-ensemble regressor as vectorized <em>[N, …]</em> tensor operations, with no Python-level loop. For offline bulk re-scoring of historical logs, this makes ORT significantly faster than calling Python predict functions in a loop.</p><h3>Latency: The “Money Shot”</h3><p>The whole point of this architecture is speed. Running the complete pipeline — preprocessing, tree-ensemble inference, and inverse postprocessing — as a single ONNX Runtime session is fast. On a standard CPU, single-row inference for this pipeline runs in under 1 ms end-to-end. Batch inference (e.g., 1,000 rows for shadow-mode re-scoring) scales linearly and stays well below the 100 ms auction window even at significant batch sizes.</p><p>For context: the equivalent Python path — loading a Spark session, applying the pipeline transformers, calling model.predict(), and postprocessing in Python — adds latency in the range of hundreds of milliseconds to seconds, depending on JVM warm-up and Spark scheduling. ONNX Runtime eliminates that overhead entirely. The graph runs as a native, compiled computation with no Python interpreter in the hot path.</p><h3>What the Output Means</h3><p>The final graph has two outputs (when keep_original_output=True):</p><ul><li>variable; shape —[N]. Raw XGBoost score in power-transformed space. Useful for offline RMSE comparison against the Spark evaluator.</li><li>prediction_final; shape — [N, 1]. Bid price in original scale, after applying <em>max(0, pred)^n</em> per placement type. This is the value your backend system uses.</li></ul><p><strong>Key takeaway:</strong> The same InferenceSession handles both single-row (latency-critical) and batch (throughput-optimized) use cases without any code change. The graph itself is the serving logic.</p><h3>Conclusion</h3><p>ONNX is, at its core, a language for computation graphs. In the context of ad tech ML pipelines, it solves a real organizational problem: the chasm between how models are built (Python, Spark, Databricks) and how they need to be served (low-latency, non-Python backends).</p><p>By encoding not just the model weights but the full pipeline — label encoding, one-hot expansion, cyclic time features, placement-dependent power transforms, and the inverse postprocessing step — into a single .onnx artifact, we achieve:</p><ul><li><strong>Eliminated re-implementation skew:</strong> the ONNX graph <em>is</em> the Python preprocessing, not a re-implementation of it — removing the primary source of training-serving skew</li><li><strong>Clear ownership:</strong> data science owns the .onnx file; backend engineering calls session.run()</li><li><strong>Portability:</strong> the same file runs via ONNX Runtime bindings in Python, Java, C++, Rust, Go, and on hardware-accelerated inference engines</li><li><strong>Debuggability:</strong> describe_graph, Netron, and run_and_dump give you full visibility into the graph at every stage of construction</li></ul><p>In practice, this translates to concrete outcomes: sub-millisecond inference latency for single-row auction scoring (measured with Python ONNX Runtime), elimination of the “it worked on my machine” class of production bugs, and engineering autonomy — data science teams can update models, preprocessing logic, and postprocessing transforms, and ship a new .onnx file without touching a line of backend code.</p><p>The model in this post is a toy — a spark XGBoost regressor trained on a Kaggle dataset, with a handful of features. But the pattern scales to production: the same OnnxBlogpostGraphBuilder approach applies to deep learning models, complex multi-stage preprocessing, and ensembles. The operators available in the <a href="https://onnx.ai/onnx/operators/">ONNX operator catalog</a> are expressive enough to cover most preprocessing and postprocessing needs.</p><p>We invite you to explore the <a href="https://www.kaggle.com/competitions/digital-turbine-auction-bid-price-prediction/overview">Kaggle dataset</a>, run the notebook, and open the exported .onnx files in <a href="https://netron.app/">Netron</a> to see the full pipeline in action.</p><h4>This piece was written by:</h4><ul><li><strong>Gabriel Gurvich</strong> — Data Scientist at DT</li><li><strong>Julia Stachowiak</strong><em> — </em>Machine Learning Engineer at DT</li><li><strong>Ron Elias</strong> — Data Scientist at DT</li><li><strong>Yuval Eckstein </strong>— Senior Data Scientist at DT</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4280a425cf9c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dt-engineering-blog/onnx-at-auction-speed-part-2-building-the-pipeline-for-real-time-bidding-4280a425cf9c">ONNX at Auction Speed: Part 2 — Building the Pipeline for Real-Time Bidding</a> was originally published in <a href="https://medium.com/dt-engineering-blog">DT Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ONNX at Auction Speed: Part 1 — Why We Rebuilt Our Bidding Pipeline Around One File]]></title>
            <link>https://medium.com/dt-engineering-blog/onnx-at-auction-speed-part-1-why-we-rebuilt-our-bidding-pipeline-around-one-file-15381c39d7ca?source=rss-25681449fffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/15381c39d7ca</guid>
            <category><![CDATA[real-time-bidding]]></category>
            <category><![CDATA[ml-model-deployment]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[adtech]]></category>
            <category><![CDATA[onnx]]></category>
            <dc:creator><![CDATA[DT Engineering]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 13:03:09 GMT</pubDate>
            <atom:updated>2026-10-20T09:46:30.823Z</atom:updated>
            <content:encoded><![CDATA[<p><em>by Gabriel Gurvich, Julia Stachowiak, Ron Elias, Yuval Eckstein</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KA4-I-FxJdxMcbx4vl9tYg.png" /></figure><p>At <a href="https://www.digitalturbine.com/">DT</a>, we operate one of the world’s largest independent mobile advertising platforms. Through our App Growth Platform (AGP), we connect thousands of publishers and app developers with advertisers via programmatic, real-time bidding (RTB) auctions — processing hundreds of billions of ad requests every day across devices, carriers, and OEMs worldwide.</p><p>The engine behind this is a decision that has to happen in under 100 milliseconds: given an auction opportunity, what should we bid? That window encompasses network round-trips, fraud detection, audience matching, and — the subject of this post — a machine learning inference call that predicts the value of the impression.</p><p>The scale of modern RTB systems is staggering. Industry measurements put peak programmatic auction traffic above <a href="https://iabtechlab.com/standards/openrtb/">12 million bid requests per second</a>. Every millisecond of latency at that scale directly affects win rates and revenue. At this throughput, a model that adds even 5ms of overhead can cost millions of missed auction opportunities per day.</p><p>To build the models that power these decisions, our data science teams process weeks of auction logs at terabyte scale — billions of auction events per training window. At this volume, distributed training is not optional; it is the <a href="https://arxiv.org/abs/2003.05622">only viable path</a>. We train both classical ML models and deep learning models for different parts of the decision pipeline.</p><p>The challenge, however, is not training — it is serving.</p><h3>The Latency Wall</h3><p>Python is an extraordinary language for machine learning. Its ecosystem — NumPy, pandas, scikit-learn, XGBoost, PyTorch — makes experimentation fast and expressive. But Python is not fast at serving time. A production backend written in Java, Scala, C++, Rust, or Go cannot simply import standard ML libraries like scikit-learn or XGBoost, or call into a distributed training pipeline. Even if it could, loading such infrastructure in the hot path of an auction is not an option.</p><p>There is another, subtler problem: the gap between training and serving code. A production ML pipeline is rarely just a model. It includes:</p><ul><li><strong>Preprocessing:</strong> categorical feature encoding, string indexing, normalization, scaling</li><li><strong>Feature engineering:</strong> derived features, time-based features, ratio features</li><li><strong>Postprocessing:</strong> clamping outputs to valid ranges, inverting target transformations to return predictions in the original scale</li></ul><p>Traditionally, data scientists implement these steps in Python for training, then hand the model weights to a backend team who re-implements the same logic in another language. This creates a persistent risk: any discrepancy between the Python preprocessing and the production implementation causes training-serving skew — predictions that differ between offline evaluation and live traffic, silently, without an obvious error.</p><h3>ONNX to the Rescue</h3><p><a href="https://onnx.ai/">ONNX (Open Neural Network Exchange)</a> is an open standard for representing ML models and computational graphs. Originally designed to allow models to move between deep learning frameworks (PyTorch ↔ TensorFlow), it has matured into something much more powerful: a portable,<strong> </strong>runtime-agnostic computation graph that can encode not just model weights but arbitrary tensor operations — including all the preprocessing and postprocessing steps that surround the model.</p><p>The key insight is this: the entire pipeline — from raw request inputs to final bid-price prediction — can be compiled into a single<strong> </strong>.onnx<strong> </strong>file. The data science team owns and maintains this file. The backend team calls one function: session.run(raw_inputs). No re-implementation. No drift.</p><p>In this post, we walk through exactly how we do this, using a toy example built on data from our public <a href="https://www.kaggle.com/competitions/digital-turbine-auction-bid-price-prediction/overview">Kaggle auction bid-price prediction competition</a>. We will cover:</p><ol><li>Feature engineering in Spark and why certain transforms are necessary.</li><li>Training a distributed XGBoost model with SparkXGBRegressor.</li><li>Exporting a metadata contract that drives the ONNX graph.</li><li>Constructing a four-stage ONNX graph: preprocessing → XGBoost → merge → postprocessing.</li><li>Debugging tools: describe_graph, Netron, and ONNX Runtime traces.</li><li>Serving with ONNX Runtime in single-row and batch modes.</li></ol><p>The full code is available in our <a href="https://gitlab.com/digitalturbine.pub/blog/snippets/5971710">GitLab repository</a>.</p><h3>Why ONNX and Not Something Else?</h3><p>ONNX is not the only option for bridging the Python-to-production gap. So why not:</p><ul><li><strong>TorchScript / TorchServe?</strong> Excellent for PyTorch-native deep learning models, but they require LibTorch as a C++ dependency — significantly heavier than ONNX Runtime, and less standard in polyglot backend environments.</li><li><strong>Custom C++ implementations?</strong> Direct model C++ APIs exist and are fast, but backend engineers must manage the model format, versioning, and all preprocessing logic — recreating the exact ownership problem we are trying to solve.</li><li><strong>Model-specific export formats (e.g., XGBoost JSON)?</strong> Lightweight, but model weights only. Preprocessing still needs to be re-implemented on the backend side.</li><li><strong>NVIDIA Triton Inference Server?</strong> A more complete inference package with quality-of-life features like automatic micro-batching and model management, and native support for both ONNX and dedicated GBT backends (FIL — Forest Inference Library). We benchmarked ONNX Runtime directly against Triton and found the performance overhead and operational complexity weren’t justified for our use case: we only needed specific capabilities, so we built exactly that around it — lighter, purpose-fit, and nothing we don’t use.</li></ul><p>ONNX’s advantage is its generality: it is not tied to any single framework, runs in a lightweight runtime available in every major language, and can represent arbitrary preprocessing logic alongside the model. That combination — one file, one runtime call, full pipeline — is what makes it the right architectural choice here.</p><h3>The Ownership Problem</h3><p>Before diving into code, it is worth being precise about the problem we are solving.</p><p>A typical ML model deployment in an ad tech company looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*NewYbZyce6bcQVSK.png" /><figcaption>‍<em>Figure 1: Traditional ML deployment: data science exports model weights; backend re-implements preprocessing and postprocessing in a different language.</em></figcaption></figure><p>Re-implementing pre- and postprocessing is where bugs live. These steps are also where ownership is ambiguous: when a prediction looks wrong in production, is it the model, the preprocessing, or the postprocessing? Which team’s bug is it?</p><p>The ONNX approach consolidates ownership:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZqDtp4sMAhwW3t3N.png" /><figcaption>‍<em>Figure 2 (click to enlarge): ONNX ownership model: data science compiles the full pipeline into a single .onnx file; backend engineering calls one function.</em></figcaption></figure><p>The backend team receives a versioned, validated artifact. There is nothing to re-implement. The data science team can update transformations, retrain the model, and ship a new .onnx file — all without changing a line of backend code.</p><p>A further benefit: because the preprocessing is encoded as ONNX operators rather than re-implemented in a separate language, it directly represents the same computation — the same graph structure, the same initializer values, the same operator semantics. This eliminates the primary source of training-serving skew: divergent re-implementations. Minor numerical differences from floating-point precision (ONNX uses 32-bit floats) or platform variations remain possible, but are far easier to detect and test when a single team owns the full pipeline.</p><p><strong>Key takeaway:</strong> ONNX moves the preprocessing/postprocessing ownership boundary from “split between two teams” to “fully owned by data science, consumed as a black box by backend.”</p><p>In Part 2, we walk through building one of these pipelines end to end — from a Spark feature pipeline through a distributed XGBoost model to a single merged ONNX graph running sub-millisecond inference in production. If you’re evaluating whether this pattern fits your stack, the mechanics matter as much as the motivation — read <a href="https://medium.com/dt-engineering-blog/onnx-at-auction-speed-part-2-building-the-pipeline-for-real-time-bidding-4280a425cf9c">Part 2</a>.</p><h4>This piece was written by:</h4><ul><li><strong>Gabriel Gurvich</strong> — Data Scientist at DT</li><li><strong>Julia Stachowiak</strong><em>—</em>Machine Learning Engineer at DT</li><li><strong>Ron Elias</strong> — Data Scientist at DT</li><li><strong>Yuval Eckstein </strong>— Senior Data Scientist at DT</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15381c39d7ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dt-engineering-blog/onnx-at-auction-speed-part-1-why-we-rebuilt-our-bidding-pipeline-around-one-file-15381c39d7ca">ONNX at Auction Speed: Part 1 — Why We Rebuilt Our Bidding Pipeline Around One File</a> was originally published in <a href="https://medium.com/dt-engineering-blog">DT Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Kafka with Tiered Storage]]></title>
            <link>https://medium.com/dt-engineering-blog/optimizing-kafka-with-tiered-storage-4363c004bfe9?source=rss-25681449fffc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4363c004bfe9</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <dc:creator><![CDATA[DT Engineering]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 13:00:23 GMT</pubDate>
            <atom:updated>2026-10-20T09:39:13.324Z</atom:updated>
            <content:encoded><![CDATA[<p><em>by Shachar Avuka</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sq8VV9LaUWUA7w__sWiISA.png" /></figure><p>At DT, Apache Kafka is the backbone of our architecture. It ingests every event flowing through our system — ad requests, billing events, win notifications, and more — totaling around five million writes per second. It serves as the entry point for all data pipelines, where consumers read data, write it to Google Cloud Storage (GCS), or trigger downstream workflows.</p><p>Because Kafka is mission-critical, we require large, efficient, and robust clusters. However, this reliability comes with significant infrastructure costs. Disk storage drives our primary costs. To ensure consumers have time to catch up and to provide a buffer for downstream incident recovery, we retain the most recent six hours of data. Across our clusters, the total retention is over 160 TB of high-performance disk storage.</p><h3>Kafka Setup</h3><p>In this post, we focus on two of our critical clusters, each managing data related to a different product line.</p><p>We manage all our Kafka clusters ourselves on VMs using the open-source version of Apache Kafka 3.9.1. Combined, these two clusters operate with 10s of brokers, 100s of disks, and 100s of TB of storage. Each cluster handles approximately two million messages per second.</p><p>We utilize multiple disks per broker to bypass the bandwidth cap of individual physical disks. While a global throughput limit per VM (which scales with CPU count) still exists, this multi-disk strategy ensures we maximize the available I/O. However, this approach introduces a complexity: unless you use RAID to abstract the storage, the disk count must align with the number and size of partitions. Misalignment can lead to uneven disk usage and hot disks that throttle the entire broker.</p><h3>Tiered Storage (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a>)</h3><p>To reduce storage costs without sacrificing stability, we adopted Tiered Storage (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage">KIP-405</a>). This feature allows us to offload older data to cost-effective object storage (GCS), keeping only the latest hot data on expensive local disks.</p><p>Kafka Tiered Storage functions by loading a RemoteStorageManager (RSM) plugin. We decided to use the <a href="https://github.com/Aiven-Open/tiered-storage-for-apache-kafka#tiered-storage-for-apache-kafka">Aiven-Open RSM plugin</a> to bridge Kafka with GCS. At the time, this was the most mature and battle-tested plugin and the obvious choice for our needs.</p><p>Producing to Kafka remains the same with Tiered Storage; incoming messages append to the active segment. When a Kafka log file rolls over (after reaching the log.segment.size or log.segment.ms configuration), the RSM splits the file into smaller chunks (configurable with rsm.config.chunk.size) and uploads them in parallel into your remote storage backend.<br>Once the upload is complete, the broker updates its internal RemoteLogMetadataManager (RLMM) and uploads the index and manifest files.<br>When a log segment reaches its local retention, it is deleted from disk, and after reaching its global retention, it is also deleted from GCS.</p><p>When consuming recent messages from Kafka, messages from the local segments are served normally. However, when reading from older offsets, the RSM fetches chunks from remote storage into a local cache (configured with rsm.config.fetch.chunk.cache.size/path/retention) and serves them from there. This allows more efficient reads for multiple consumers. You can also configure a prefetch size (rsm.config.fetch.chunk.cache.prefetch.max.size). Prefetch predicts sequential consumer reads and proactively fetches the next chunk while the current chunk is consumed, reducing latency for sequential access patterns.</p><p>It is important to ensure that the chunks cache is located on a fast, adequate disk and has sufficient size; if it’s too small or too slow, it can start deleting its own files before reading them, potentially causing various errors in various seemingly unrelated parts of the system — files not being found, timeouts exceeded, and more.</p><p>Also note that the broker keeps a cache of the remote log indexes. The broker is always located inside the first log directory that you configured, and it cannot be moved (it is <a href="https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerServer.scala#L755">hardcoded in the source code</a>). This may be important if you are using multiple disks without RAID.</p><p>Kafka stores Tiered Storage metadata (which segments are located remotely) in an internal topic called __remote_log_metadata.</p><p>We configured the cluster to keep one hour of data on local disks and the remaining five hours on GCS. Since most consumers process data within minutes of ingestion, they rarely experience the higher latency of reading from object storage. However, when an incident occurs, the deep history remains seamlessly available.</p><pre># On the brokers and controllers<br>remote.log.storage.enable = true<br><br># Per Topic<br>remote.storage.enable = true<br>local.retention.ms = 3600000 # Keep 1 hour on high-speed disk<br>retention.ms = 21600000 # Keep 6 hours total<br>cleanup.policy = delete Per Topic<br>remote.storage.enable = true<br>local.retention.ms = 3600000 # Keep 1 hour on high-speed disk<br>retention.ms = 21600000 # Keep 6 hours total<br>cleanup.policy = delete</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*OdTsI68Rr8DKx0P1.png" /></figure><h3>Switching to Local SSDs</h3><p>Tiered Storage enabled our second optimization: local SSDs.</p><p>Local SSDs attach physically to the server, offering superior IOPS and throughput at a lower cost compared to network-attached persistent disks (PDs). The catch is durability — data on a local SSD is lost if the VM stops.</p><p>Previously, losing a broker meant recovering six hours of data across the network, which was risky. With Tiered Storage, we only need to recover one hour of local data — the rest remains safely persisted in GCS.</p><p>Switching to local SSD allowed us to save on storage costs and gain performance through higher disk throughput, while maintaining the cluster’s reliability and fault tolerance.</p><p>To mitigate the risk of simultaneous hardware failures, employ a <a href="https://docs.cloud.google.com/compute/docs/instances/use-spread-placement-policies">Spread Placement Policy</a> to ensure each broker resides in a distinct availability domain.</p><h3>The Gotcha: Tuning Segment Sizes</h3><p>When you calculate disk requirements for one hour of local retention, simple division leads to a trap.</p><p>Tiered Storage only offloads a log file (segment) once it is closed. It cannot offload the active segment currently being written to.</p><p>With our default log.segment.bytes of 1 GiB, a topic with 400 partitions and three replicas could have up to 1.2 TiB of active data sitting on disk, regardless of the retention time policy.</p><p>To reduce wasted disk space, we gradually reduced the segment size until we reached a point where the increased number of files hurt system performance. We reached a good balance by reducing the segment size for big topics from 1 GB to 200 MB, and for smaller, less busy topics to 50 MB.</p><p>This tuning significantly reduced our active disk footprint without exhausting OS file descriptor limits. These numbers are specific to our use case and depend on the traffic volume per partition replica. We recommend gradually reducing the segment size to find the optimal size for your cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*CoQj20EBExN3xVsL.png" /></figure><h3>Results</h3><p>By combining Tiered Storage with local SSDs and properly tuning segment sizes, we successfully accomplished the following:</p><ul><li>Maintained our six-hour retention safety net</li><li>Increased write-throughput capabilities</li><li>Added the ability to increase retention independently of disk size</li><li>Reduced our Kafka infrastructure costs by over 70%</li></ul><h3>What’s Next? Use Cases for the Future</h3><p>This architectural shift opens the door to new paradigms in Kafka management:</p><ol><li>Direct Read (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1248:+Broker+support+for+remote+tiered+storage+fetch+from+consumer">KIP-1248</a> &amp; <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1254:+Kafka+Consumer+Support+for+Remote+Tiered+Storage+Fetch">KIP-1254</a>): Currently, reading historical data requires the broker to fetch data from GCS and serve it to the consumer. Future improvements may allow consumers to read directly from object storage, bypassing the broker entirely.</li><li>Diskless Topics (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics">KIP-1150</a>): If speed isn’t the primary constraint, could we store everything in object storage? This approach would eliminate the need for replication, making cluster scaling nearly instantaneous and operations significantly simpler.</li></ol><p>Combining these ideas could fundamentally change how we operate streaming data — moving toward stateless, ultra-scalable brokers with infinite storage.</p><h4>This piece was written by:</h4><ul><li><strong>Shachar Avuka</strong> — DevOps Engineer at DT</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4363c004bfe9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dt-engineering-blog/optimizing-kafka-with-tiered-storage-4363c004bfe9">Optimizing Kafka with Tiered Storage</a> was originally published in <a href="https://medium.com/dt-engineering-blog">DT Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>