<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DV Engineering on Medium]]></title>
        <description><![CDATA[Stories by DV Engineering on Medium]]></description>
        <link>https://medium.com/@dv-engineering?source=rss-15c3378cf733------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TnPau7XQiJx1MFMZNDBp3w.png</url>
            <title>Stories by DV Engineering on Medium</title>
            <link>https://medium.com/@dv-engineering?source=rss-15c3378cf733------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:48:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dv-engineering/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating the Optimization of Dockerfiles for Faster and More Efficient CI/CD Pipelines]]></title>
            <link>https://medium.com/doubleverify-engineering/automating-the-optimization-of-dockerfiles-for-faster-and-more-efficient-ci-cd-pipelines-8b1e7e211225?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b1e7e211225</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 11:41:25 GMT</pubDate>
            <atom:updated>2026-11-09T11:41:41.070Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em><a href="https://www.linkedin.com/in/briter-gonzalez/">Bríter González</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*c1HOJ2aRIpZ9teyRBd-weA.png" /></figure><p>As modern applications grow increasingly complex, orchestrating their many components can become tedious and time-consuming. This is precisely why Docker remains such a powerful tool, offering simpler portability and deployment workflows while ensuring correct software behavior.</p><p>However, building a Docker image can be resource- and time-intensive. Therefore, multiple optimization techniques have been introduced, each with a different purpose. For example, tools such as Trivy allow you to inspect images and find potential improvement opportunities, while automated systems like Slim Toolkit inspect an image and suggest ways to reduce its size.</p><h3>Docker Layers and Caching</h3><p>There are also techniques to optimize the Dockerfile itself, which is the actual source of an image. Under the hood, Docker stacks layers of other images to create new ones. For example, a Dockerfile can be based on a simple Alpine Linux base image. But once Docker executes any instruction (e.g., COPY), it creates a new temporary image, called a layer, with the changes.</p><p>Thanks to this system, Docker introduces a core optimization mechanism: cached layers. When a Docker image is built, Docker is smart enough to detect that it has already been built in the past and will cache as many layers as possible. A layer is cached if neither the instruction syntax nor the associated files have changed. Once the layer’s cache is invalidated, due to the nondeterministic nature of the build process, all subsequent layers must be executed again. However, the cache system’s effectiveness depends entirely on how the Dockerfile is written.</p><p><strong>Figure 1</strong> shows a Dockerfile with several optimization issues. The first is related to the <strong>ARG</strong> instruction. Whenever the value of the argument changes, Docker invalidates the cache at that line, forcing it to re-run every instruction from that point on. Additionally, by using a general path for the <strong>COPY</strong> instruction, whenever the source code changes, the installation of NMP dependencies must be repeated. This unoptimized Dockerfile can significantly delay deployment pipelines and waste resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AdXyKRA7vkJrRmYaFMHucw.png" /><figcaption><strong>Figure 1:</strong> Unoptimized Dockerfile for Node.js application</figcaption></figure><p>With a few simple changes to the Dockerfile, as shown in <strong>Figure</strong> <strong>2</strong>, we can optimize it to significantly improve image build performance. The environment argument is only required for the instruction that builds the project, and placing it at the top invalidates many instructions that could otherwise be cached. Additionally, by first copying the dependency configuration files and immediately installing the <strong>NPM</strong> dependencies, the cache is invalidated only when the dependency configuration changes, rather than when the source code itself changes. Both of these optimizations can be combined through a simple analysis of the instruction order so that the cache is invalidated only when strictly necessary, saving significant time in incremental builds and across environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKRSTJCO432LwPDwOHUroQ.png" /><figcaption><strong>Figure 2: </strong>Optimized Dockerfile for Node.js application</figcaption></figure><h3>Automated Dockerfile Rewriting</h3><p>As you can probably infer, small Dockerfile changes can significantly improve cache usage. However, identifying these optimization opportunities is not always easy or obvious. Additionally, these optimizations tend to be low priority in organizations that must prioritize other roadmap projects. To help automate this process, some tools, such as basic linters and more advanced ones like Drive or DR Miner, attempt to detect these types of smells and fix them automatically.</p><p>However, they are either very limited, relying solely on static information from the Dockerfile, or they depend on large language models (LLMs) to find a solution, which is not deterministic. Based on this gap, in collaboration with the Vrije Universiteit Brussel (VUB) and under the guidance of Professor Dr. Coen De Roover, Dr. Ruben Opdebeeck and Gilles Coremans, I implemented a platform to automatically rewrite the Dockerfile based on both static (syntax of the code) and dynamic (runtime behavior) data.</p><p>This tool is focused on generating a dependency graph. It uses static analysis to detect syntax-based dependencies, for example, which instructions depend on an argument defined with an <strong>ARG</strong> instruction. This is combined with the power of dynamic analysis, which is capable of inspecting the image as it is being built, understanding how each instruction in the Dockerfile interacts with the image’s file system, and generating a dependency graph. A dynamic dependency between two instructions is created if an instruction reads or deletes any file that the other one added or modified. <strong>Figure 3</strong> shows the generated dependency graph for the previously explained Dockerfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0W8Z6ZQrMtnHxaMZllwd9g.png" /><figcaption><strong>Figure 3: </strong>Generated dependency graph</figcaption></figure><p>Once this dependency graph has been generated, an algorithm finds optimization opportunities by applying two strategies: Splitting <strong>COPY</strong> instructions and Reordering instructions. For splitting COPY instructions, the dependency graph indicates not only that an instruction depends on a <strong>COPY</strong> instruction, but also which specific files it depends on. Based on this, the algorithm can split a <strong>COPY</strong> instruction into multiple instructions, moving the dependency to a new one that copies fewer files, reducing the likelihood of cache invalidation.</p><p>Next, the second strategy applies two heuristics: the number of copied files and the instruction execution times. It first attempts to move instructions that copy the most files as close to the bottom as possible, following the idea that the more files a <strong>COPY</strong> instruction includes, the more likely its cache will be invalidated. In this example, it would automatically detect and extract the dependency configuration files (package.json and package-lock.json) into its own instruction, because those are the files the <strong>npm install</strong> command reads.</p><p>Then, it orders the instructions by execution time, moving the longest ones as close to the top as possible because they offer the highest reward when cached. In the example, it would automatically detect that the instructions for installing Graphviz, copying the source code, and installing the NPM dependencies take much longer to execute than the <strong>ARG</strong> instruction, and it would reorder them without breaking the dependencies defined by the graph.</p><h3>Unlocking the Optimizer as a Platform</h3><p>This system was designed from the start as a platform. The Dockerfile optimizer was built as a collection of <strong>NPM</strong> packages that are easy to install and use. The first package is a parser that not only produces a clean, complete Abstract Syntax Tree (AST), but also provides an expressive query system for traversing and modifying it.</p><p>Now, <strong>Figure 4</strong> provides an overview of the main optimizer’s architecture. The optimizer relies on the parser to generate the AST of the Dockerfile, which is then modified as optimizations are identified. To build the dependency graph (DAG), the platform clearly separates the subsystem that analyzes syntax to detect static dependencies from the one that inspects filesystem interactions during the build process, combining the results through a separate algorithm that generates the DAG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtM9W-DjGvVFVfn-wqmRmA.png" /><figcaption><strong>Figure 4:</strong> Architectural overview of the optimizer</figcaption></figure><p>The DAG and the AST together serve as the inputs to the optimizer. At its core, the optimizer orchestrates strategies: each strategy takes the DAG and AST and applies potential optimizations. Currently, we have implemented and evaluated only the two strategies described earlier. Even so, the platform was designed to be easily extended, whether by adding new strategies or refining existing ones. For instance, new strategies could target auxiliary elements that influence the build process, such as the<em> .dockerignore</em> file and other Docker-specific configuration files. As a future milestone, we also plan to incorporate evolutionary analysis by examining <strong>Git</strong> history to identify frequently changing files. This would enable the system to apply additional heuristics for splitting and relocating <strong>COPY</strong> instructions, placing those involving frequently modified files as close to the bottom of the Dockerfile as possible.</p><p>The optimizer can be used as a standard <strong>NPM</strong> library, exposing classes and functions that allow consumers to trigger the optimization process or extend its capabilities programmatically. It is also available as a <strong>CLI</strong>, which can be executed with a simple command within the Docker build context to optimize the image-creation workflow.</p><h3>Evaluation</h3><p>Now, this had to be tested in real-life scenarios. First, we gathered multiple publicly available GitHub repositories with Dockerfiles. Then, we used this tool to optimize them. We created an initial image for each Dockerfile. Once the image is created, through static analysis, the system identifies the source code used in the Docker image and adds a new line to simulate a change in the source code, as if it were a real change. After that, the system rebuilds the image and gathers information about the process.</p><p>We compared the build times of both images: the one created with the old Dockerfile and the one created with the new one. In total, 339 repositories were used, and the results were quite surprising. On average, the build time decreased by <strong>16.5%</strong>. However, <strong>60 significant</strong> cases showed an average build-time decrease of <strong>57.3%</strong>. The largest improvement reduced a build <strong>from 14 minutes to just 12 seconds</strong> by preventing the reinstallation of heavy dependencies on every build. <strong>Figure 5</strong> shows the distribution of relative build-time improvement for the 339 images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9PdjQQ09wyLIGjZfshwGtw.png" /><figcaption><strong>Figure 5:</strong> Distribution of relative improvement in build time</figcaption></figure><p>The results also show that the cache hit rate for these Dockerfiles increased from <strong>41.7%</strong> to <strong>51.8%</strong>, representing an absolute increase of <strong>10 percentage points</strong> and a relative increase of <strong>24.2%</strong>. This indicates that a much larger portion of the build process can now use cached layers instead of rerunning the same instructions from scratch.</p><p>We also found that the tool is language-agnostic. Among the 60 cases with significant improvements, we identified projects written in a range of programming languages, including Python, JavaScript, TypeScript, Shell, Go, PHP and C#. Other cases involved minor optimizations for languages such as Ruby, Rust, Scala, Java and Erlang. In addition, the dynamic analysis accounts for native libraries and external tools, so these components may also present further optimization opportunities.</p><p>Based on these results, the improvements should reduce compute time and resource consumption for future builds. For example, if we rebuild all 339 images weekly due to source code changes, optimizing the Dockerfiles would save an estimated <strong>40 hours</strong> of compute time per year.</p><p>We also confirmed that the resulting Docker image based on the optimized Dockerfile was semantically identical to the original. We extracted and compared snapshots of each image’s file system bit by bit. For all optimized Dockerfiles, the tool reported identical file systems (excluding non-functional dynamic files, such as timestamps or generated certificates), confirming they are semantically identical and preserving the image’s behavior.</p><p>These results show that there are still many opportunities to improve and optimize Dockerfiles, even in real production images. By reducing build time, we can lower CI/CD pipeline costs and energy consumption. It can also significantly increase the productivity and efficiency of entire teams or departments.</p><p>Compared with existing tools, this approach goes further, automatically rewriting Dockerfiles based on dependency analysis derived from the image’s runtime build behavior. However, this automation tool is not perfect and has a few limitations. For example, it only works for UNIX-based Docker images with POSIX Shell as the command-line interface.</p><p>This automation tool is still a work in progress. Beyond fixing smaller limitations like this one and adding support for multi-source <strong>COPY</strong> instructions, we also plan to expand its core capabilities with additional optimization strategies. Another key enhancement will be support for multi-stage builds, which could further improve both build time and image size.</p><p>Even though Docker is a powerful tool for simplifying the deployment of complex applications, effectively using its full potential still requires considerable knowledge and expertise. Furthermore, optimization opportunities are often not immediately apparent, and the use of both deterministic and AI-powered automation systems should be encouraged across a wide range of application domains. These systems should be viewed as collaborative tools that complement, rather than replace, human expertise. By combining human judgment with automation systems, it is possible to achieve levels of productivity and efficiency that would otherwise be difficult to attain.</p><h3>References</h3><p>González Daza, Bríter Andrés. (2025). <em>Automating the Optimization of Dockerfiles for Incremental Docker Builds</em>. Master’s thesis, Faculty of Science and Bio-Engineering Sciences, Vrije Universiteit Brussel.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b1e7e211225" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/automating-the-optimization-of-dockerfiles-for-faster-and-more-efficient-ci-cd-pipelines-8b1e7e211225">Automating the Optimization of Dockerfiles for Faster and More Efficient CI/CD Pipelines</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigQuery Compute Costs: Fix the Configuration Before You Fix the Queries]]></title>
            <link>https://medium.com/doubleverify-engineering/bigquery-compute-costs-fix-the-configuration-before-you-fix-the-queries-2009450262e4?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/2009450262e4</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[google-cloud]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 14:09:27 GMT</pubDate>
            <atom:updated>2026-11-05T14:13:18.945Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em><a href="https://www.linkedin.com/in/sajad-bhat-aa3962275/">Sajad Bhat</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*rkWw2CfMMVk5EwzPTnegdg.png" /></figure><p>BigQuery compute and storage together were consuming 65–70% of our entire GCP bill. Our instinct was to look at this problem where it was most visible: the queries themselves.</p><p>So we built observability first. We created dashboards that track slot consumption and average slot usage per hour and per day, broken down by project and across the overall workload. Alongside that, we built a query cost dashboard pulling from job history metadata, showing the total cost incurred by each query and the average cost per run.</p><p>The turning point came when we overlaid slot utilization against our reservation configuration. The chart told a clear story: our baseline was sized for an earlier workload, so routine, predictable demand was regularly spilling into autoscale every day. We weren’t overspending because of bad queries. We were overspending because the infrastructure underneath was configured for a workload pattern we didn’t have.</p><p>When we dug further, we found more configuration drift. Slot reservations still reflected our initial setup and hadn’t yet been revisited as the workload grew. Materialized views were refreshing hundreds of times a day against tables that were rarely, if ever, queried. The real money was hiding in configuration, not code.</p><p>The biggest wins came from settings that made sense at the start but hadn’t been revisited as our workload changed. We started with the change that had the biggest impact and required zero code, even though it meant proposing a higher commitment to reduce the bill.</p><h3>Right-Sizing Slot Reservations</h3><p>This was the change that felt the most counterintuitive when we proposed it: increasing committed capacity to reduce cost.</p><p>To see why spending more saved us money, the pricing mechanics matter. We had configured a baseline (the minimum number of slots always allocated to the reservation). We had configured a baseline, which is the minimum number of slots always allocated to the reservation. We pay for these slots whether they’re in use or not, but commitments on baseline slots can usually be purchased at a discounted rate. Pairing a baseline with a commitment lets you lock in a lower rate for the capacity you know you’ll consume.</p><p>On top of the baseline, we set an autoscale ceiling. BigQuery provisions additional slots on demand when the workload exceeds the baseline. Autoscale slots are always billed at the PAYG rate, with a one-minute minimum and are billed for allocated slots, not used slots.</p><p>The original configuration reflected a conservative, cost-conscious setup: a low baseline of 1,800 slots and a generous autoscale ceiling of 10,000. The logic was “only pay for what we use.” In practice, this configuration cost more than needed for our workload profile because our demand was predictable, not spiky. The “base load” (the slot demand present almost every hour) was well above the 1,800-slot baseline. So every day, predictable, routine work was spilling into the autoscale tier.</p><p>The internal conversation wasn’t easy. Proposing a 122% increase in committed capacity to reduce the bill looks wrong on a slide: the commitment line goes up, and you’re asking leadership to trust that the autoscale line comes down by more. We had to lead with the utilization data, not the proposal: show the pattern first, let the math make the case, and only then present the recommendation.</p><p>According to the official <a href="https://cloud.google.com/bigquery/pricing#capacity-compute-pricing">Google pricing</a>, the one-year commitment rate is roughly 20% cheaper per slot-hour than the autoscale PAYG. We were paying a flexibility premium for capacity we were going to use anyway.</p><p>We analyzed nearly six months of hourly consumption data across percentiles. The 90th percentile sat around 4,000 slots. We increased the baseline from 1,800 to 4,000, a 122% increase in committed capacity, to reduce total compute cost. Those 2,200 additional slots shifted from the PAYG autoscale rate to the commitment rate, resulting in a 20% per-slot-hour reduction in the capacity we were consuming regardless.</p><p>Simultaneously, we lowered the autoscale max from 10,000 to 9,000. A high ceiling doesn’t just mean capacity to burst; it means permission to burst. Without backpressure, queries happily consumed whatever was available. Lowering the ceiling forced us to be intentional about which workloads genuinely needed burst capacity and which could queue. We brought it down in phases, monitoring for queuing or increased query latency at each step.</p><p>The result: roughly 15% annual savings on slot reservations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2sKmnKXf_-U6wH1EZLCNdQ.png" /></figure><p>The “feels wrong” moment was proposing a higher level of commitment to leadership as a cost-reduction measure. For workloads with predictable baselines, on-demand isn’t flexibility; it’s a premium we pay for capacity we were going to use anyway.</p><p>Right-sizing isn’t a one-time exercise. Workloads change as pipelines get deployed and retired. We set up a weekly check to compare actual slot utilization against the configuration.</p><p>With reservations accounting for the largest single reduction, two other compute optimizations stacked on top — each meaningful on its own, but deliberate in sequencing.</p><h3>Other Compute Wins That Stacked on Top</h3><h4>1. Rethinking Materialized View Refresh Intervals</h4><p>Materialized views are meant to save money by precomputing expensive aggregations, so downstream queries hit a cached result instead of reprocessing raw data every time. In practice, the MVs were costing more in slot consumption than the query savings they delivered. This fix accounted for 7% of the total BQ spend reduction.</p><p>BigQuery’s default MV refresh interval is five minutes. That interval suits near-real-time data needs. It is very costly when the base table is queried only a handful of times a day. Each refresh consumes slot-seconds and rewrites data, and the cost scales with both the number of MVs and the size of their base tables.</p><p>We audited every MV in the project and found dozens refreshing more than 200 times per day against tables that were queried once a day, or not at all.</p><p>One of the most extreme cases was an MV that aggregated about 1 TB of data. It was built on a 19.5 TB base table with 6.4 billion rows, and it used the default five-minute refresh interval. That meant 288 refreshes per day. We checked the last 90 days of job metadata: three queries per day on average, with some days as low as one. The view was refreshed 288 times to serve three reads.</p><p>We moved the view from a five-minute to a six-hour refresh interval with a matching max_staleness of six hours, dropping from 288 to four refreshes per day. That’s 284 daily query refreshes against a 19.5 TB base table, each consuming slots, scanning partitions and rewriting results that no downstream consumer would read before the next refresh overwrote it. Over a month, that’s over 8,500 wasted refreshes eliminated from a single view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3JwdYa0ZpKUlwfR29026jQ.png" /></figure><p>Now multiply that across the dozens of other MVs we found with similar profiles. The fix wasn’t to turn off MVs; it was to align refresh frequency to actual consumption patterns. We cross-referenced MV refresh logs with downstream query frequency and bucketed every MV into three categories:</p><p><strong>High-frequency consumers</strong> (operational dashboards, live monitoring): kept at aggressive refresh intervals, typically one to three hours.</p><p><strong>Low-frequency consumers</strong> (daily batch reads, periodic analyst queries): moved to three- to twelve-hour refresh intervals.</p><p><strong>Zero consumers dropped</strong> these MVs entirely. They had been built for pipelines that were later retired, so their consumers were gone while the views kept running as configured. They continued to refresh on schedule, using slots to produce results that no one read.</p><p>Tuning refresh intervals, aligning staleness tolerance with actual downstream query frequency and dropping orphaned MVs resulted in up to a ~90% reduction in compute cost for MV-related slot consumption. The effort was entirely configuration: no SQL rewrites, no pipeline changes. Just ALTER MATERIALIZED VIEW … SET OPTIONS (max_staleness = INTERVAL ‘6’ HOUR, refresh_interval_minutes = 360) and a few DROP MATERIALIZED VIEW statements for the defunct ones. Only after these configuration-level wins were in place did we turn to the work that most teams start with– the SQL itself.</p><h4>2. Reducing Slot Waste in Existing Queries</h4><p>We made SQL changes last, after the infrastructure and configuration changes described above had already lowered the baseline. SQL optimization was the second-largest category, at 10% of total BQ spend savings. It was also the highest-effort category, because every SQL fix requires engineering time, testing, and deployment.</p><p>We pulled the top queries by total slot-hours consumed over 90 days. Those top queries, out of hundreds running daily, accounted for roughly 90% of total slot consumption. The patterns were well-documented BigQuery anti-patterns.</p><p>Google’s own <a href="https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute">query optimization documentation</a> guide covers query-level best practices, and we applied them to these queries. The specific fixes are not the main point; profiling by slot-hours told us which queries mattered, and we deliberately sequenced this highest-effort work after the configuration changes that needed no SQL or pipeline changes.</p><h3>Takeaways</h3><p>Three changes: reservation right-sizing (~15% on slot costs), MV refresh tuning (7% of BQ spend) and targeted SQL fixes (10% of BQ spend), together accounted for the largest share of the compute cost reduction. But the order matters more than the individual fixes.</p><p>We deliberately worked lowest-effort-first. The reservation restructuring and MV tuning were configuration changes: no code, no tests, no deployments, no risk of breaking a pipeline. SQL optimization came last because it required engineering time for every improvement. By the time we got to query rewrites, the infrastructure wins had already lowered the baseline and the SQL fixes stacked on top, compounding the savings.</p><p>If your BigQuery compute bill is growing, and your first instinct is to start rewriting queries — pause. Check the reservation configuration. Check whether the baseline covers steady-state demand or whether you’re paying autoscale rates for a predictable load. Check whether materialized views are refreshed hundreds of times a day for tables nobody is querying. The most expensive line in the BigQuery bill might not be a bad query. It might be a default you never revisited. We didn’t discover a better way to optimize SQL. We discovered we’d been solving the wrong problem first. The queries were never the biggest cost driver. The infrastructure they ran on was.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2009450262e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/bigquery-compute-costs-fix-the-configuration-before-you-fix-the-queries-2009450262e4">BigQuery Compute Costs: Fix the Configuration Before You Fix the Queries</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Expanding Our Pre-Bid Dataset Using Aerospike and Kafka]]></title>
            <link>https://medium.com/doubleverify-engineering/expanding-our-pre-bid-dataset-using-aerospike-and-kafka-8756895c7dbc?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/8756895c7dbc</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 14:30:17 GMT</pubDate>
            <atom:updated>2026-11-01T14:30:42.354Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em><a href="https://www.linkedin.com/in/jason-reed-541a467b/?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base%3BU2%2BSPIIbRfiInqME96Q8Tg%3D%3D">Jason Reed</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*oZW7Be61g17n9BT51_HRfA.png" /></figure><p>At DV, our pre-bid service is called during real-time bidding — the high-speed auctions used to sell ad impressions across web and app environments. The service handles hundreds of billions of requests per day and must respond in milliseconds. It performs complex business logic and consults dozens of datasets to determine whether the auctioned ad placement meets our client advertisers’ standards.</p><p>Historically, we met these requirements by pre-loading copies of each dataset into each service instance’s memory. When a new version of one of our largest datasets grew from 30 million to over 200 million records, we decided to explore a new approach, moving beyond local in-memory replication to a more centralized, highly scalable datastore.</p><p>This post covers how we stored that dataset in Aerospike, scaled ingestion with Apache Kafka, tuned the Aerospike client and replicated globally, while meeting strict freshness and latency targets.</p><h3>Expanding Beyond Local Memory</h3><p>With the arrival of this new, large dataset, storing the data in a high-performance datastore like Aerospike promised several benefits. Aerospike is designed for sharding, distributed replication and persistence on SSD, providing a highly available, durable and scalable solution. Aerospike clusters are built to support a large volume of lookups at low latency; other teams at DV had experienced consistent sub-millisecond performance on high-volume workloads.</p><p>Using a centralized datastore provided some simplicity and efficiency benefits as well. Instead of keeping hundreds of local dataset copies up to date, we could focus on one shared Aerospike cluster per region. New service instances would no longer have to wait the additional minutes to load the full dataset before serving requests. We worked with DV’s platform team to set up and start loading our data into some test Aerospike clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xLjOwgs4jJDtpKvwPWfW0A.png" /><figcaption>Aerospike Cache Data Flow</figcaption></figure><h3>Streaming Data Into Aerospike</h3><p>The new 200 million record dataset was produced to a Kafka topic by an upstream team. Around 50 million Kafka messages were written to the topic per day, in batches. We targeted data availability within minutes, so we wanted a solution that could process large message volumes quickly.</p><p>To load from Kafka to Aerospike, we used the Aerospike Connect inbound Kafka connector, which is built on the Kafka Connect framework. In addition to data loading, the connector allows users to configure simple functions (single-message transforms, or SMTs) to transform data before writing it to Aerospike. We used a combination of built-in and custom SMTs to format our data efficiently, filter it, and set record-level TTLs.</p><p>We set up the connector and began to test it on the full production-scale dataset. During extended testing of the data loading component, we saw unpredictable spikes in the incoming message rate that caused the connector to build up a Kafka lag that wasn’t cleared fast enough to meet our data freshness targets. Fortunately, the source topic was sufficiently partitioned, and Aerospike Connect can scale out seamlessly, distributing tasks and Kafka partitions across multiple instances. We scaled out and quickly had a solution capable of handling the peak workload.</p><h3>Optimizing Aerospike Lookup Performance</h3><p>Next, our service had to start retrieving data from Aerospike. Consistent low latency was critical to meeting our performance targets. We used Aerospike’s Java client library to do key-value lookups. In our first pass, we tried using asynchronous lookups. We ran load tests to simulate real request volume and noticed that under high load, P99 Aerospike client latency was surprisingly high, reaching several milliseconds. Aerospike’s internal latency was very fast, suggesting the additional latency originated outside the database. We decided to dig deeper to optimize performance to a level we could release in production.</p><p>We tested multiple Aerospike client approaches and configurations. We tried both synchronous and asynchronous Aerospike clients, using varied workloads to simulate handling our real-world request load of tens of thousands of requests per second per service instance. We adjusted client settings, such as the maximum number of concurrent in-flight requests and the event loop configurations used by the asynchronous client.</p><p>After several rounds of testing, we decided to go with synchronous lookups. Under our tested workloads, synchronous lookups showed lower CPU usage and had more consistent tail latency than the asynchronous configurations we evaluated. Synchronous lookups were also simpler to integrate into the service code.</p><p>Once we had a performant client configuration, we began using Aerospike in production. We gradually rolled out the new data to partner integrations in our first region. After verifying performance, we were ready to launch our Aerospike dataset globally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eZwXbWn-z3IcL0NBXS8DdA.png" /><figcaption>Cache System with Cross-datacenter Replication</figcaption></figure><h3>Replicating Aerospike Across Regions</h3><p>DV operates in a number of regions. We host our pre-bid service in each of these regions near our partners to provide quick responses. With the new cache system, we needed to set up Aerospike locally in each region to stay fast.</p><p>Our platform team provisioned Aerospike clusters in each region. We ingested the data into one regional Aerospike cluster, then replicated it to other regions using Aerospike’s cross-datacenter replication (XDR). We enabled Aerospike in each region one by one, with our service instances using nearby Aerospike clusters. Replication worked smoothly, and we were able to roll out Aerospike usage worldwide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Icpgob9hKrFr3TahphGxA.png" /></figure><h3>Monitoring and Observability</h3><p>For this first wide rollout of Aerospike usage, monitoring was crucial to catch issues rapidly and keep our service performing reliably. We took advantage of Aerospike’s built-in monitoring capabilities and implemented additional client-side monitoring. We monitor the data-loading process, the Aerospike DB clusters and the Aerospike lookups in our service. We have separate dashboards and sets of alerts for each of these three major components.</p><p>Some key metrics we monitor from the data-loading process include:</p><ul><li>Kafka message arrival rate</li><li>Kafka lag</li><li>Connector read/write rate</li><li>Connector timeout/exception rate</li><li>Connector task status</li><li>Number of records in Aerospike</li><li>Time from Kafka message creation to Aerospike write</li><li>Rate, lag, and errors in cross-datacenter replication</li></ul><p>To track the performance of lookups from our service, we observe:</p><ul><li>Hit/Miss rate</li><li>Client timeouts/exceptions</li><li>Aerospike DB lookup latency</li><li>Client round-trip lookup latency</li></ul><p>Our platform team owns the monitoring of the Aerospike DB itself. They provide views into key DB-internal metrics, including available storage capacity, hardware resource usage, replication and partitioning.</p><h3>Production Performance</h3><p>Over several months, we’ve rolled out Aerospike and our expanded dataset across regions and integrations. Right now, the pre-bid service performs over 20 billion Aerospike lookups per day with consistent sub-millisecond round-trip latency. We write around 40 million records per day to our new cache and meet our latency and data-freshness needs.</p><p>Operationally, Aerospike has been remarkably smooth to work with. It helps to have a battle-tested platform team watching over the infrastructure! Our service no longer has to load this large dataset, and we’ve seen significant improvements in startup time. The large, frequently updating dataset we migrated is now more scalable and easy to manage.</p><h3>Building on Our Aerospike Foundation</h3><p>Following the success of our first large Aerospike dataset migration, we moved additional datasets to Aerospike, including several of the pre-bid service’s largest. We introduced near caching for datasets that change less frequently, and we refined our data preprocessing. Together, these improvements simplified operations, reduced startup times, and improved the elasticity and scalability of our pre-bid service.</p><blockquote><em>Apache, Apache Kafka, and Kafka are trademarks of the Apache Software Foundation (ASF).</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8756895c7dbc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/expanding-our-pre-bid-dataset-using-aerospike-and-kafka-8756895c7dbc">Expanding Our Pre-Bid Dataset Using Aerospike and Kafka</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Case for a Boring Agent Graph]]></title>
            <link>https://medium.com/doubleverify-engineering/the-case-for-a-boring-agent-graph-71add8dae049?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/71add8dae049</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 13:31:00 GMT</pubDate>
            <atom:updated>2026-10-05T13:31:13.949Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em><a href="https://www.linkedin.com/in/goldbergmaurice/">Maurice Goldberg</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*CrEaPBNhIOahhoV8W91a8Q.png" /></figure><p>Let’s say you’re on call for your agent platform, working through a support ticket about your agent — let’s call it <em>research-agent-v2</em> — and the agent is returning a 500. There are calls to two different teams’ plugins in the trace, along with a subagent that you don’t remember seeing a week prior. You rerun the request to see if it fails again, but this time <em>research-agent-v2</em> builds a different graph. You’re now in the unenviable position of asking two teams to debug a failure that you can’t reliably reproduce.</p><p>At DV, we built our Agent OS platform to prevent this scenario. The problem above is that the structure of <em>research-agent-v2</em>’s graph was decided at request time instead of release time. If the graph changes between runs, it takes far more work to compare traces and figure out which team should investigate a failure. Furthermore, rolling back the release doesn’t necessarily bring back the graph from the earlier run. For DV’s agent platform, we found that the simplest way to avoid that extra work is <em>not to let the graph rearrange itself in the first place</em>.</p><h3>One Graph, Many Assistants</h3><p>DV’s Agent OS is an internal agent platform shared by the company’s product teams. Each team embeds an assistant in its own product, but every assistant runs on the same graph. At its core, the platform is a thin API layer in front of one shared runtime graph. Every assistant on the platform runs this same graph, which has two nodes, a conditional edge, and a set of middleware hooks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkcrYQSt9QKUVr68_DuL9A.png" /></figure><p>The simplicity of this basic ReAct loop means that when a new product team wants to use our platform and ship their own assistant (a versioned, pinned configuration of model, prompts, plugins, and policies that runs on the shared graph), they don’t have to write an entirely new graph. Their new assistant amounts to a small repo that breaks down roughly to something like this:</p><pre>my-assistant/<br>├── assistant.json    # display name + model<br>├── system.md         # assistant-level system prompt<br>├── pyproject.toml    # pinned plugin versions<br>└── evals/            # eval cases that gate every release<br><br>// assistant.json<br>{<br>  &quot;name&quot;: &quot;My Assistant&quot;,<br>  &quot;model&quot;: &quot;provider:model-name&quot;<br>}<br><br># pyproject.toml<br>dependencies = [<br>  &quot;plugin-domain-a==1.2.0&quot;,<br>  &quot;plugin-domain-b==0.4.1&quot;,<br>]</pre><p>This is the assistant’s entire surface area. The graph, loop, streaming, persistence, and gateway in front of MCP servers all stay exactly the same. The CI literally just takes that repo, deploys it as an immutable version, runs the evals against it, and only then lets you promote the assistant to be usable on the platform.</p><p>This way, plugin work can be as in-depth as the author needs, but the agent’s surface area is limited to skills, MCP tools, and evals.</p><h3>Why We Don’t Build the Graph at Runtime</h3><p>Many dynamic agent patterns are getting attention in 2026, patterns like supervisor-router agents, spawn-on-demand subagents, dynamic tool loading from registries, and others. For the most part, these patterns are designed for the generic but complex use case where one team owns the whole loop and the domain is open.</p><p>For these situations, dynamic design patterns are the right fit. A generic research agent, for instance, doesn’t know what you’ll ask, and likewise, a generic coding agent can’t know ahead of time which subagents should be dispatched to solve a problem. For open-ended research and coding tasks, that flexibility can be useful, but in shipping our Agent OS system at DV, we had a different set of priorities:</p><h4>Provide agents as a platform</h4><p>Multiple product teams ship customer features on our runtime. When the graph reshapes itself per request, a bug in one team’s plugin can trip a path neither team owns. That blast radius is built into the design.</p><h4>Enable complete vertical capability in the ad analytics space</h4><p>DV’s domain is digital media measurement and optimization: viewability, fraud, brand suitability, attribution, and using that trusted measurement data to maximize campaign performance. The domain is complex, but we know which tasks our assistants need to support. A user of an agent platform can pre-declare its agents’ toolset because they know what the agent needs to do.</p><h4>Make it productionizable</h4><p>Support a clear ability to audit and reproduce bugs, eval gates, and a need to roll something back when it breaks. None of these properties stay stable in the same way with a graph that re-composes itself from run to run.</p><p>By definition, agent systems are already nondeterministic, but the question we kept coming back to is <em>where to allow that nondeterminism</em>. Our answer was inside skills, not in the graph itself. The graph is fixed when an assistant is released. At runtime, the model can choose tools, arguments, and responses within that structure. A plugin can still encode sequences and branching behavior, but it cannot recompose the runtime around itself.</p><h3>Debugging and Rollback</h3><p>Say we picked a run from our agent system three months ago. What do we need to reconstruct this run’s starting conditions?</p><p>In a dynamic agent system, the honest answer is the input, version label, model weights as they were that day and a recording of every structural decision the LLM made: which subagent it spawned, tools it pulled and the plan it generated. Without that recording, you can’t reproduce it.</p><p>In our system at DV, the answer is intentionally simple. We just need:</p><ol><li>Input</li><li>Assistant version</li><li>Thread state captured before the run</li></ol><p>The assistant version identifies the graph and configuration used for the run, while the input and captured thread state provide the starting context. A rerun may still produce different tool calls, arguments, or responses, and external services may return different data. We’re still running the same graph with the same prompts and plugin versions.</p><p>For an on-call engineer, this removes one source of uncertainty: we know which graph, prompts, and plugin versions were in use. We still need the original trace to investigate what happened, but we don’t also have to reconstruct a graph assembled at request time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-9z6wJxsirc_xk8H8kvZ4A.png" /></figure><p>Every deployed commit creates a new, immutable assistant version. When our host creates a thread, it records the selected assistant ID and uses that ID for subsequent requests. Changing the default version changes what the host selects for new threads; for existing threads, it keeps using the recorded ID.</p><p>This gives rollback a precise meaning. Rolling back to v2 means updating the release channel so new threads select v2. Our host continues using the assistant version recorded when each existing thread was created. We don’t currently support moving those threads between versions.</p><p>Fixes follow the same rule. We publish v3, run it through the release gates, and make it the default for new threads. Threads already on v2 remain on v2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDL3O3CLBJDJy8HRY32C4w.png" /></figure><p>Each assistant version records exactly what shipped: the Git commit, the lockfile digest, and the resolved version of every plugin. Traces tell us what happened during a run. Release metadata tells us what was running.</p><p>Gradual rollout is then straightforward. When a thread is created, the rollout policy selects a version based on a cohort or percentage bucket. A thread in the 5% test group pins to v3; everyone else pins to v2. Expanding the rollout changes the choice for future threads, rather than threads that already exist.</p><h3>Trade-Offs</h3><p>As you might expect, we give up some runtime flexibility. If a new use case does not fit the existing configuration surface, the platform team has to extend it. That puts the work on our schedule rather than the product team’s. We accept that extra work because it makes the platform easier to operate.</p><p>Agent OS also doesn’t currently provide an immediate cutoff for a vulnerable version. We can deprovision it, but cached configurations and runs already in progress may continue using it.</p><p>Furthermore, the small configuration surface, while meaningfully simplifying our operations, does not make onboarding effortless. Teams still need to understand skills, MCP tools, and evals well enough to ship effectively. To address this, we have repo templates, scaffolding skills, and SDLC documentation to help teams navigate their release process.</p><p>At DV, we chose a fixed graph because, today, it gives us a precise answer to a basic production question: what exactly was running? If improved tooling for dynamic graphs makes debugging and replay reliable enough, we’ll reconsider. The bet we’re making right now is that fixed graphs are the right answer for a vertical agent platform that needs to be traceable, auditable, and productionizable in 2026.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71add8dae049" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/the-case-for-a-boring-agent-graph-71add8dae049">The Case for a Boring Agent Graph</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Security Culture That Runs Without You: A Case Study on Democratizing Application…]]></title>
            <link>https://medium.com/doubleverify-engineering/building-a-security-culture-that-runs-without-you-a-case-study-on-democratizing-application-2ba0ddcf57d5?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ba0ddcf57d5</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Thu, 24 Sep 2026 08:53:17 GMT</pubDate>
            <atom:updated>2026-09-24T09:37:33.289Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building a Security Culture That Runs Without You: </strong>A Case Study on Democratizing Application Security and Autonomous Lifecycle Integration</h3><p><em>Written By: </em><a href="https://www.linkedin.com/in/naveen-rajamannar/?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base%3BwAtpKdbCTXuktntUkz4VZQ%3D%3D">Naveen Rajamannar</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*CWsoTNETQgJIn4RmVCgqdQ.png" /></figure><p>At DV, application security spans hundreds of developers and thousands of repositories. Product teams ship data products, UI and API service platforms, agentic workflows, plugins and chatbots weekly, with only a few AppSec team members to support the workload. Security knowledge travels by word of mouth, with each team having to repeat the findings.</p><p>While mature DevSecOps principles and robust automation tools could cover this, what if security engineers stopped being the only people who do security testing and instead became the ones who build the systems that empower everyone else?</p><p>Over the past year, that is exactly what our team has designed and built. This guide details how we democratized application security, transferring true ownership across engineering by embedding automated security checks directly into pre-existing developer and QA workflows, and learned from each other to make security a shared, cross-functional mindset.</p><h3>The Challenge</h3><p>The biggest bottleneck in security was never tools; it was access and confidence. Developers and QA engineers already ran thousands of test cycles. What they needed was a security vocabulary and tooling they could use in their workflows without a security engineer in the room. Below is a representation of the main building blocks of security democratization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PKdNu_SijtEMh_s1Tbogg.png" /></figure><h3>Establishing a Common Language and Training: OWASP</h3><p>DevSecOps is exceptionally easy to put on a slide deck, but difficult to execute in reality. The first thing we did was not write a pipeline or configure a new scanner; instead, we established a common technical vocabulary that every engineer could reference and plan a security championship program around. The OWASP standards provided that open framework. We anchored our platform on a comprehensive matrix of industry standards, mapping our baseline to OWASP ASVS, backend pipelines to the API Top 10, UI scenarios to the WSTG, and AI-specific risks to the LLM Top 10.</p><p>But raw compliance standards alone do not change day-to-day engineering behavior. Partnering with L&amp;D, we launched a Confluence program that stripped away complex jargon to focus on short, practical engineering examples. By leveraging video-generation tools DV makes available to us, like Google Vids and Gemini Canvas, we built an asynchronous library of bite-sized videos and high-impact instructional material engineers could easily absorb between meetings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QSS-3RbrNI783LPxx0HLuA.png" /></figure><h4>Transferring Capability: Prompt Files</h4><p>Knowledge that lives in a separate place, even when well-designed, only gets referenced when people remember to reference it. This insight came from observing how QA teams worked: they started adopting prompt files, structured instructions and reusable templates used directly in the IDE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vv5HLPwynD85C-nAV_AM1g.png" /><figcaption>Security testing workflow: quick question-based code generation</figcaption></figure><p>Security should be built into where the work is already happening, not in a different place as a separate task. Instead of asking QA engineers to consult a knowledge base before writing tests, we built security directly into the prompt files they were already using, which were also designed to use the same tools that QA already uses: Postman scripts, Newman execution and Playwright code with pre-built security assertions. <strong>No new tool required mastery, because security was never meant for learning something new and complex.</strong></p><h4>From Prompt Files to IDE commands: The MCP Step</h4><p>Prompt files lowered the barrier. But they still required a developer to find the file, set it up, and choose how to run it. Through Model Context Protocol (MCP), the security testing capability was embedded directly into the IDE as a native command. A developer working in Cursor, Copilot, or Claude already had the security skills loaded. Running a full OWASP API security scan meant typing one command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kkULyeZCn9kg2AHL32t48Q.png" /><figcaption>Test output in the IDE: each OWASP category surfaces a Watch Video link alongside the evidence and scenario count</figcaption></figure><p>Prompt files transferred the knowledge. MCP embedded the capability into the places where people already worked. Each layer made security more automatic and accessible, with humans in the loop and in the right places.</p><p>Taking this further, we built an interactive MCP-based system with <strong>PromptFlow MCP</strong> (in-house MCP, not Microsoft Azure PromptFlow) that has custom skills, resources and tools to be built and plugged in. We engineered the security-specific skills and tools on top of this framework. Answering a few questions activates the right flow for the requirement at hand, handling everything from simple username-and-password logins to complex auth flows. This scaled our intended design across all projects without requiring manual execution in each one. Since it takes contextual input from the tester, each team’s product knowledge also gets directly ingrained into the security process.</p><p>Because engineers never had to leave their existing workflows, security never felt like a separate step. This question-driven, context-aware model also became the basis for what was coming next: autonomous agents that could carry that same intelligence and operate end-to-end within the Quality Engineering agentic workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P1NlWwkxALfm-1zblzX32w.png" /></figure><h3>Automation: Security Scripts Inside the QA Workflow</h3><p>Automation is powerful, but it’s most effective when it is woven directly into the daily habits of individual contributors. The defining power of our decentralized framework is that security scans are no longer separate, isolated events scheduled by an outside AppSec team on a disparate cadence. Instead, they run natively inside the exact same GitLab CI/CD pipelines as functional QA suites, triggered automatically by the exact same merge request events. A developer merges code, the functional test suite runs, and security runs right alongside it.</p><p>This pipeline alignment was entirely deliberate. Because the underlying tests are structured in formats that QA and development teams already understand, there was no new complex tooling to learn. The output looks and feels exactly like any standard unit or integration run. To prevent engineering fatigue, we rolled out OWASP coverage in systematic phases. This gave product delivery teams ample time to ingest initial findings, ask architectural questions, and build natural habits before the next set of test categories and automated assertions arrived.</p><h3>The Final Milestone of Our Vision: Autonomous AI Agents</h3><p>The next milestone we have been building toward is the rollout of fully <strong>Autonomous AI Agents</strong>. Operating within a highly extensible AI framework engineered by our QA Automation Center of Excellence (CoE) team, we deployed our first two autonomous security agents: api-security-gen and web-security-gen (with many more currently in the pipeline).</p><p>These agents independently discover live application endpoints, generate relevant test code, map assertions to explicit OWASP categories, build extensive test scenarios, execute them dynamically, and deliver comprehensive, triage-ready reports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYiF7RJV7sg6JQanUhWepA.png" /></figure><p>But the more interesting question is not who can call the agents, but what else can call them? For example, the <strong>orchestrator agent</strong> integrates with our project management lifecycle, where code modifications automatically trigger the orchestrator to generate targeted performance suites in the background. Security suites can be triggered the same way if configured similarly.</p><p>Second, the orchestrator agent leverages a crawler network that runs on demand as new crawl requests come in, mapping live application changes and feeding fresh feature surfaces into our test pipelines with minimal manual configuration.</p><h3>Dashboards Live Where the Audience Lives</h3><p>While standard analytics monitoring platforms like Grafana worked beautifully for our initial baseline metrics, we intentionally routed this program’s telemetry straight into the environments our teams live in every day: Confluence and GitLab. Given the complex cross-team collaboration required for this rollout, utilizing rich HTML allowed us to break free from standard dashboard constraints, giving us the design freedom and future flexibility we needed to serve different audiences.</p><p>We engineered a dedicated background pipeline that systematically parses the artifacts created during every QA pipeline run. It aggregates this data into two distinct visibility layers tailored for separate audiences:</p><ol><li><strong>The Management Confluence View:</strong> A high-level pane tracking roadmaps, adoption rates, and portfolio volumes so leadership instantly understands their security posture without sync meetings.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FBYTB2bE0sYDzcHqG-qusg.png" /></figure><p>2.<strong> The Engineering GitLab Pages View:</strong> A granular, squad-level dashboard mapping per-project timelines, code sources, endpoint coverage, and explicit execution results over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftS1rXQa4HRvSdKhJsqO4w.png" /></figure><h3>The Heavy Lifting Behind the Seamless Automation</h3><p>Flawless automation is the visible outcome that stakeholders see, but <strong>relentless manual groundwork is the engineering grit</strong> that knits separate tools together and enables the entire process to move forward as one cohesive unit, making an advanced AI ecosystem look effortless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mNXCe-jgk_EVYOVgG6hC9w.png" /></figure><p>To bridge the gap between cold automation pipelines and fluid engineering workflows, we focused our deepest engineering energy on an intensive, human-centric rollout playbook:</p><ul><li><strong>Culture and Enablement: </strong>Shared OWASP vocabulary, video libraries, Security Champions Program, embedded initial security QA engineer with QA teams, and security built directly into existing QA frameworks with no new tools.</li><li><strong>Workflow Integration: </strong>Security prompts in existing QA workflows, MCP as native IDE commands in Cursor, Claude and Copilot, functional QA suites and Security suites side-by-side.</li><li><strong>Framework Architecture: </strong>PromptFlow MCP, autonomous api-security-gen and web-security-gen agents, passive crawler network for live endpoint discovery, and automated data pipelines publishing to Confluence and GitLab Pages.</li><li><strong>Closing the Loop: </strong>Embedded feedback loops feeding every human-identified gap back into the next automation cycle.</li></ul><p>The first instinct was to provide the tools to the team and step back. Alas, it did not work; access is not the same as ownership. Handing a developer a security scanner without the vocabulary to read what it returns, or having the expectation that fixing it is their job, just produces unread reports.</p><p>As a result, we slowed down. We recruited a dedicated security QA engineer to sit directly with the QA teams and do the groundwork by hand first: running the early tests with them, translating findings into language that made sense in their context, and learning where people actually got stuck. That manual phase is what made the self-serve model possible later. You cannot hand off ownership of something you have not first walked through together.</p><p>The manual groundwork paid off.</p><h3>The Payoff: Measurable Real-World Impact</h3><p>By treating security as an integrated, shared engineering responsibility rather than a separate compliance checkbox, the platform has achieved verifiable scale across its portfolio:</p><ul><li><strong>76.47% Overall Adoption:</strong> Across 17 active API portfolio projects, 13 have completely adopted automated security testing, achieving 100% endpoint coverage.</li><li><strong>20,454 Security Tests Executed per Cycle:</strong> Over twenty thousand distinct security assertions are run autonomously every cycle across active delivery pipelines, maintaining a remarkably stable <strong>success rate</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xAk91hTvHixO-s91b-83lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSTkoddQdKgaEntdI2VjNg.png" /></figure><p>Historically, executing a comprehensive DevSecOps transformation required massive, specialized security teams and immense effort. By leveraging native tools that engineering squads already possessed, using context-aware autonomous AI agents, and prioritizing meticulous human scaffolding, this entire <strong>orchestration groundwork</strong> platform was successfully planned, built, and shipped on roughly a <strong>20% to 30% time allocation</strong> alongside standard daily AppSec workloads.</p><p>Let’s see how it stacks up against the DevSecOps Principles:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgvVr7RLRfTyZXRg_EFBgg.png" /></figure><p>The security backlog is no longer AppSec’s exclusive queue to manage. It belongs directly to the people who write and test the code, a self-learning asset completely adapted to the way our engineering teams actually work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ba0ddcf57d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/building-a-security-culture-that-runs-without-you-a-case-study-on-democratizing-application-2ba0ddcf57d5">Building a Security Culture That Runs Without You: A Case Study on Democratizing Application…</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How and Why We Split One Managed ELT Box into Three Components We Control]]></title>
            <link>https://medium.com/doubleverify-engineering/how-and-why-we-split-one-managed-elt-box-into-three-components-we-control-eace7f8e8e74?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/eace7f8e8e74</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 12:18:39 GMT</pubDate>
            <atom:updated>2026-09-22T12:18:56.231Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em><a href="https://www.linkedin.com/in/aminziaei/">Amin Ziaei</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*oAOU3UwwOR4Nn4DD2khJ2Q.png" /></figure><p>We recently launched a project that I led to build our own self-hosted Extract, Load, Transform (ELT) solution for our data pipelines. The reasons were threefold: to gain more control over the scalability and availability of our data pipelines; to give us more technical flexibility in the tools we can use for those pipelines; and to find a cheaper alternative to the tool we had been using (a SaaS solution).</p><p>What follows is an overview of this project and a guide for others to do the same, including how to set up the stack, what each tool is responsible for and how to make the full ELT solution production-ready for critical data pipelines within your system. The stack consists of just three tools, Airbyte, DBT and Airflow, each of which performs a distinct role, as follows:</p><h3>Baseline: A Self-Hosted Three-Tool ELT Stack for Critical Data Components</h3><p>Here’s what each component performs as part of a full ELT solution:</p><ul><li>Airbyte: Extract and load data from a source database to a destination database</li><li>DBT: Transformation of the data loaded into the destination database</li><li>Airflow: Orchestration between DBT and Airbyte, such as scheduling and monitoring tools</li></ul><p>Here’s a bit more on how each tool functions within the full ELT solution and why it was the best option for our purposes.</p><h4>Airbyte</h4><p>Airbyte is a self-hosted cluster that spins up a number of pods within your Kubernetes cluster, with a dedicated task to successfully extract data from a source database and transfer it to a destination database.</p><p>We chose this tool because its concepts are simple to grasp, with no code or complex logic needed to move data between databases, thanks to its easy-to-use UI and admin APIs. That said, Airbyte is a heavy tool; it requires many components to set up, which makes it difficult to run on a local machine. Airbyte also has some pitfalls we learned over time while using it in production, which I’ll elaborate on in the Production Readiness section below.</p><p>Airbyte includes the following components:</p><ul><li><strong>Sources:</strong> Airbyte has a massive selection of options. Each database source needs to be installed on your Airbyte instance through its admin UI. When defining a source, you are presented with configuration options specific to the database you’re connecting to.</li><li><strong>Destinations</strong>: Airbyte also offers a comprehensive list of database types to choose from, with configuration options specific to each type.</li><li><strong>Connections:</strong> Airbyte connects a source to a destination and also holds configurations for the schema, the type of data sync that works best with your data (incremental, full refresh, etc.), the source database and other general configurations, such as which schema to push to if not defined in the destination configuration.</li></ul><p>The type of data sync options you are presented with depends on:</p><ul><li><strong>Source Data:</strong> Without a cursor column in your source data, Airbyte cannot track the last synced entry and cannot perform an incremental sync.</li><li><strong>Database Type</strong>: Enabling CDC (Change Data Capture) may require trade-offs like storage and performance. For example, PostgreSQL CDC requires creating a replication slot and a publication to capture WAL (Write-Ahead Log) within your database. This can significantly increase storage usage if not consumed frequently enough. So you might want to consider a full refresh of the underlying data instead.</li></ul><h4>DBT</h4><p>Airbyte lands raw data; something else has to shape it. That’s DBT. It operates on a single database per model: it reads the raw data copied into the destination table, writes the transformed data back to the same destination database, and ideally provides a final schema that your application can consume.</p><p>It has become one of my favorite tools to work with due to its simplicity and capabilities. It’s lightweight and has proven reliable for our projects, since a model is just a SQL query with no framework to fight.</p><p>Note that DBT is a CLI tool, not a cluster like Airbyte, which matters because it has to run on a server (in this case, Airflow worker pods).</p><p>What follows are examples of how DBT models are structured and how easy it is to write tests to achieve a solid, production-ready data transformation layer.</p><p>For context, DBT models are .sql files, each holding a single query whose result becomes the final table. The filename will become the name of the final table. Here’s an example DBT query:</p><pre>select u.id as user_id,<br>      u.first_name,<br>      u.last_name,<br>      p.country as country_of_origin<br>from user_entity u<br>        join profile p on u.id = p.user_id<br>where p.domain = &#39;SOME_DOMAIN&#39;;</pre><p>This model will generate a table with the columns: user_id, first_name, last_name, and country_of_origin. The data type for each column depends on the source data type unless you cast it in the SELECT statement.</p><p>In DBT, it’s possible to have dependencies between models. So if a table depends on another transformed table, you can reference the first model as shown in the following example:</p><pre>select u.id as user_id,<br>      u.first_name,<br>      u.last_name,<br>      p.country as country_of_origin<br>from user_entity u<br>        join {{ ref(&quot;profile&quot;) }} p on u.id = p.user_id<br>where p.domain = &#39;SOME_DOMAIN&#39;;</pre><p>In this example, another model file named `profile.sql` is referenced. So DBT would know to build the table profile first, before building this model.</p><p>This is all you need for DBT to know that the second model depends on the first; it then handles the order of execution for you.</p><p>Additionally, in DBT you can add tests to your models. The simplest form of such a test involves writing another query; if it returns any results, the test is considered a failure. This gives you flexibility, as you can run your test cases within the database before deployment, which holds true for the transformation models as well.</p><p>Here’s an example of a test in DBT:</p><pre>select *<br>from {{ ref(&quot;user_info&quot;) }}<br>where user_id is null</pre><p>This query should not return any values, as we expect user_id not to be null.</p><h4>Airflow</h4><p>Now that I’ve explained these first two components, I’ll move on to how they can interact. Airbyte and DBT have no integration, so we need a third tool to handle the most important tasks in any ELT solution: orchestrating between tools, providing visibility into each pipeline’s runs and scheduling runs. If you’ve worked on a centralized scheduler before, you know how complex it can get. That’s where Airflow comes in. It exposes per-task visibility and scheduling that a hand-rolled scheduler rarely matches.</p><p>What’s more, Airflow integrates with many tools, as its plugin ecosystem includes purpose-built providers for most sources. Additional packages must be installed for these integrations; however, they are specifically designed for Airflow and work seamlessly with the Airflow UI and Airflow servers.</p><p>Airflow has components that we use heavily in our system:</p><ul><li><strong>DAGs:</strong> Runnable Python scripts that contain one or more tasks. Each DAG run usually performs a full data pipeline and ELT on your data. DAGs can be scheduled, or you can run them on demand.</li><li><strong>Tasks: </strong>Each task has a distinct use; tasks can be linked by having one task depend on another’s success.</li><li><strong>Variables: </strong>You can set variables or read them within your DAG Python script. This makes development easier. I’ll show some examples below.</li><li><strong>Connections:</strong> You can connect directly to a database system outside of Airbyte. This is useful if you need to query or write data in a centralized database for your DAG Python script business logic.</li></ul><p>We don’t use Airflow only for data transformation. It’s an excellent scheduling tool for any need, including cron jobs and webhooks.</p><h4>Integration between the three tools</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lw88VDVwtUTMB27BpEfNJQ.png" /></figure><p>Airflow is integrated with Airbyte and DBT as follows:</p><ul><li><strong>DBT:</strong> Since it runs within Airflow worker pods, Airflow has full visibility into each component of DBT. So you can see each model being run for transformation and its tests’ success/failure, logs, etc.</li><li><strong>Airbyte: </strong>Airflow connects to Airbyte via Airbyte’s admin APIs. It uses a polling mechanism to check the status of a sync, but it’s more limited than what you see in a DBT task in Airflow. For example, you won’t have access to the Airbyte logs within the Airflow UI. But if an Airbyte task fails, you will see the API response Airbyte returns when polling the sync run’s status.</li></ul><h3>Failure Modes: Pipeline Visibility Gaps and Concurrent Runs</h3><p>Such visibility gaps are exactly where things quietly break. Airflow, as the core orchestrator, provides extensive metrics that you can use for alerting and monitoring your DAG runs and even tasks within each DAG. Here are some example alerts we use for our data pipelines:</p><ul><li><strong>DAG and task failures</strong></li><li><strong>DAG runs skipped:</strong> This is lower priority, but helpful to know if a DAG is taking significantly longer than expected, which can cause the next run to be skipped. With Airflow, you can investigate why a single DAG run is taking longer than expected.</li><li><strong>DAGs removed:</strong> Unfortunately, when you have a syntax error within your DAG Python script, Airflow won’t load the DAG at all. An unintentional change can cause the DAG to disappear after deployment without you noticing. This alert helps prevent that.</li></ul><h3>Implementation: Orchestration-Driven Observability and Production-Readiness Practices</h3><p>Knowing what breaks is one thing; designing so it doesn’t is another. To design a destination for your Airbyte, you’ll need a table that’s separate from the final table DBT produces. This is because Airbyte and DBT cannot integrate. We solve this by having these tables in different schemas. We create what we like to call a Landing Zone schema, as well as a final schema that the application consumes.</p><p>DBT recreates the transformed table (though you can configure it to be incremental) on each run. Initially, we granted the application’s user access to the tables DBT produced; however, we noticed that access was lost after each DBT run. So, make sure you update the default schema permissions for your application’s access at the schema level. You can automatically assign permissions to a database user with a query like the one below (this is for PostgreSQL, but a similar concept should be available for other databases):</p><pre>ALTER DEFAULT PRIVILEGES IN SCHEMA some_schema<br>  GRANT SELECT ON TABLES TO &quot;some-user&quot;;</pre><p>DBT performs transformations atomically, by swapping the table in a single transaction, so your application never reads a half-built table.</p><p>DBT profiles hold your database connections: target schema, credentials, and so on. They also define an important variable: the number of concurrent threads.</p><p>DBT is smart enough to figure out how many transformations it can run concurrently based on dependencies you define between your models.</p><h4>Some other things to keep in mind</h4><p>If you have two models where one depends sequentially on the other, you cannot go higher than one thread. But if you have four models, two of which depend on the other two, you can set the number of threads to two. This will transform two models at the same time and then process the next two that depend on the first two.</p><p>When using the Airbyte source for PostgreSQL, I noticed our data syncs can take more than 20 minutes when there is not much data to sync (especially in lower environments). After some research, I found a configuration called “Initial Waiting Time in Seconds (Advanced)”. This configuration defaults to 1200 seconds, so the Airbyte connection will wait that long for a change to appear before closing the sync. This can cause false-positive alerts for skipped DAG runs when your system doesn’t have many writes to the PostgreSQL database. To avoid this issue, reduce the setting, as a lower wait lets no-op syncs finish fast instead of tripping skipped-run alerts.</p><h3>Conclusion</h3><p>Business requirements, team needs, and available technology tools can all change over time. Building your own self-hosted ELT (Extract, Load, Transform) solution gives you more control, more technical flexibility and more cost-effectiveness to deal with those changes than a SaaS solution. Airflow’s extensive integrations let you swap out any part of the ELT tools you’re using for another, without affecting the final result and with minimal changes to your data pipeline.</p><p>For example, we’re running into scalability issues with Airbyte, and this architecture lets us swap it out without a rewrite. Note that if you do choose this full ELT solution, you’ll need to get more involved in the nitty-gritty details of implementing it yourself. The trade-off is more control over your tools and their performance, but you lose the simplicity of a SaaS solution that provides a single-pane-of-glass ELT. I hope this article has clearly laid out how to weigh the two options. For us, the choice was clear.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eace7f8e8e74" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/how-and-why-we-split-one-managed-elt-box-into-three-components-we-control-eace7f8e8e74">How and Why We Split One Managed ELT Box into Three Components We Control</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Unified Explore in Semantic Layer Is a Performance Liability, Not a Design Virtue]]></title>
            <link>https://medium.com/doubleverify-engineering/one-unified-explore-in-semantic-layer-is-a-performance-liability-not-a-design-virtue-57a6a3567063?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/57a6a3567063</guid>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 11:38:43 GMT</pubDate>
            <atom:updated>2026-09-16T12:46:24.434Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em><a href="https://www.linkedin.com/in/marutha-sangeetha-shanmugam-28b5a5101/">Marutha Sangeetha Shanmugam</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*zlf1D5aWW0EMuw7eql5CJA.png" /></figure><p>If you’re a Data Engineer, Data Architect, BI Developer, or someone who’s ever stared at a slow dashboard and thought, “Why is this so painful?”, this one’s for you. Just a real problem, a relatable analogy, and the fix that actually worked.</p><p>I’ll reference Google Looker throughout this post. It’s what we run at DV. But swap in the dbt Semantic Layer, Cube.dev, Power BI, Tableau, or even a raw PostgreSQL view layer, the anti-pattern is identical. The tool doesn’t create the problem — the design does.</p><p>Alright, imagine this.</p><p>You’re cooking dinner for three different friends. One is vegan, one is keto, and one eats everything. So you cook three separate meals; it totally makes sense, right? Each one is perfectly tailored.</p><p>But then your sous chef walks in and says, <em>“</em>Hey, can you just pick a few things from the keto meal, a few from the vegan one, and a little from everything else, mix it all up so it’s a better-balanced diet for everyone?”</p><p>That’s basically what we were doing with our data pipelines.</p><h3>Here’s the Actual Situation — Semantic Layer Join</h3><p>We have three separate data pipelines at work: <strong><em>Orders, Billing</em>, and <em>Shipments</em></strong><em>.</em> Each one does its own thing. Different business logic, different data shapes. And as pipelines, they were really well designed. We’ve never faced any issues with reports or dashboards that run against an individual dataset.</p><p>But then came the reporting pressure. Stakeholders wanted<strong> a single dashboard with multiple tiles and slide-outs </strong>that required metrics from 3 different datasets. Makes sense from a business perspective. But here’s where things went sideways.</p><p>Instead of thinking hard about <em>how</em> to combine them properly, we just said, “Eh, let’s stitch them together in Looker.” Just create one big <em>Explore</em>.</p><p>It felt clean. It felt like good architecture. <em>It was not.</em> 🙃</p><h3>Why It Blew Up</h3><p>Here’s what actually happens under the hood when you do that.</p><p>Every time someone loads a dashboard tile, even if they only care about <em>Shipments</em> data, the query engine touches all three datasets.</p><p>It’s like asking the waiter for your bill, but before handing it over, they go back and manually recalculate every single order from every table in the restaurant, just to figure out what <em>you</em> owe.</p><p>On top of that, the three datasets don’t speak the same “language.” Their grain sizes (the level of detail at which they store data) differ. So Snowflake has to do a lot of expensive on-the-fly math to make them agree. And our persistent derived tables (PDT- the pre-built summary tables) rebuild costs? They scale based on the biggest table in the join, not what you actually queried. So you’re paying for a full mansion renovation every time someone wants to repaint one wall.</p><p>We’re ignoring that <strong>storing a pre-aggregated result costs</strong> <strong>a fraction of what recomputing it on every query costs.</strong> Storage is cheap. Computation, especially at scale, is not cheap.</p><p>Caching doesn’t fix this. You can’t cache your way out of a bad design.</p><p>Caching optimizes execution. It does nothing to fix the design. If your schema forces every query to do unnecessary work, caching just means you’re doing that unnecessary work less often.</p><h3>So What’s the Fix?</h3><p>Here’s the thing that is actually the right call:</p><p><strong>You decide <em>how</em> the data is combined at the pipeline layer before a single Looker (or semantic or reporting) layer view is ever written.</strong></p><p>Specifically:</p><ul><li>Figure out what grain each dashboard/report actually needs. The grain each surface needs is determined by its finest dimension combination.</li><li>Build a purpose-scoped, pre-aggregated table for each reporting surface, whether that’s a visual dashboard, a self-serve export tool, an API, or an external platform integration<em>.</em></li><li>If you absolutely need data from all three pipelines combined, integrate them in Snowflake at the pipeline layer. Own it there, instead of in the semantic layer.</li><li>Then let Looker just do Looker things, such as metric definitions, access control, and calculated fields, on top of data that’s <strong>already clean and aggregated.</strong></li></ul><p>Basically, <strong>Looker is the finishing touch, not the construction site.</strong></p><h3>Okay, But Does It Actually Work? (Answer Is: Yes )</h3><p>We ran a POC. Same data, same dashboard, two different approaches:</p><h4>Before: The “Just Stitch It in Looker” World</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_qvckj4BZiWqvkdKnoYiMA.png" /></figure><p>Every time someone opened the dashboard, Snowflake was waking up, joining <em>Orders </em>+ <em>Billing</em> + <em>Shipments</em>, resolving grain mismatches, and running CASE logic on the fly. Even for a simple “How many orders today?” tile.</p><h4>After: Purpose-Scoped Pre-Aggregated Tables</h4><p>We built three lightweight pre-aggregated tables, one per pipeline, each pre-joined and rolled up to the reporting grain. Cross-dataset metrics got their own dedicated Snowflake aggregate, built at the pipeline layer. Looker’s job is just to read and display.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iy1X7H0NmbOE5ACsFvOm4w.png" /></figure><h4>The Real-World Feel</h4><p>Before the fix, analysts would open the dashboard, wait long enough, and <em>still</em> be loading. After? Users don’t see the architecture change. They just notice that suddenly everything feels fast. And that’s exactly the point.</p><h3>Takeaway</h3><p>One big unified Explore <em>feels</em> like good design. It’s actually a performance liability. Design the aggregation contract upstream, and everything downstream gets cheaper, faster, and way easier to maintain.</p><p>Looker gave me the vocabulary to explain this problem clearly: Explores, PDTs, and derived tables. But the lesson isn’t about Looker. It’s about the contract between your pipeline and your reporting layer.</p><p>Whatever tool sits between your warehouse and your end user <strong>should receive clean, pre-aggregated, grain-correct data. It should not be the place where that work happens for the first time.</strong></p><p>If your semantic layer is doing heavy lifting, that’s not a sign of a powerful semantic layer. That’s a sign of an upstream design gap.</p><p>That’s it. That’s the whole story. 🎉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57a6a3567063" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/one-unified-explore-in-semantic-layer-is-a-performance-liability-not-a-design-virtue-57a6a3567063">One Unified Explore in Semantic Layer Is a Performance Liability, Not a Design Virtue</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Database Is the Source of Truth. Your Agent Only Reads the Repo]]></title>
            <link>https://medium.com/doubleverify-engineering/the-database-is-the-source-of-truth-your-agent-only-reads-the-repo-644b1cff0051?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/644b1cff0051</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 10:24:58 GMT</pubDate>
            <atom:updated>2026-09-03T10:26:46.955Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em>Matthew Weiss</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*zlf1D5aWW0EMuw7eql5CJA.png" /></figure><p>In AI-aided development, even with a well-defined agents.md file and a detailed prompt, your agent can still make a fatal assumption: it sees a relationship in your ORM files and assumes a foreign key constraint enforces it, then builds accordingly when no such constraint actually exists in the database. That wrong assumption becomes the foundation for the next 15 minutes of its work, wasting your time and your token budget.</p><p>Why does this keep happening as the models get smarter? Because your agent only reads the repo, and the repo is only a piece of the puzzle; the database contains the other missing pieces. Some of that context never appears in the code at all: DB-side validations and triggers won’t show up in your ORM, and the test data developers rely on is inaccessible to agents. A developer missing that information would ask for it or check first; an agent starts guessing.</p><p>The fix is about 15 minutes of setup. Give your agent read-only access to the database itself, and it stops guessing and starts knowing, testing its own queries against real data and even end-to-end testing the feature it just wrote.</p><h3>Giving Your Agent Its Missing Context</h3><p>By the end of this section, you’ll have:</p><ul><li>An MCP server that your agent can query directly</li><li>Credentials that prevent your agents from wreaking havoc in your data layer</li><li>A generated schema that agents without this MCP can use to get started in your codebase, write away.</li></ul><h4>MCP Installation</h4><p>To keep this article sharp, I’ll focus on Postgres and MongoDB as database technologies, since this covers both NoSQL and SQL examples and aligns with the tech stack I use on my team. With the targets chosen, the first move is to get the agents wired in. This is a prerequisite for giving your agents crucial DB context.</p><h4>Make Writes Impossible</h4><p>For credentials, I’ve configured both the MCP tool itself and the credentials I provided to be read-only. Remember that even though you may instruct your agent to “never write to the database”, these are not hard rules, and the agent can simply ignore them, since prompt instructions are guidance, not enforcement.</p><p>To ensure data integrity, we need to make it impossible for DB write commands to go through, even if they attempt to. Additionally, queries run through the MCP may be heavy, and data still gets processed by the agent, so I’d stick to your non-production environments for performance and data governance reasons.</p><p>With the guardrails in place, the question is whether your agent can actually see the MCPs yet. In Claude, you can verify this with /mcp to see which are listed. If you can’t see them, restarting your instance of Claude or Cursor will usually pick them up.</p><p>Now the agent can <em>check </em>the schema rather than infer it from your ORM.</p><h4>Give the Agent Durable Memory</h4><p>At this point, you should have a working MCP and be ready to let your agent start writing queries. The downside to this is that, just like an agent onboarding to a repo without an agents.md file, every new instance will treat your codebase as if it’s seeing it for the first time.</p><p>Nothing persists between sessions aside from your agents.md, so it’ll need to learn your database each time. We can speed up subsequent sessions by storing durable knowledge for the agent that can be read into context without a ton of prodding around in the database MCP <em>every</em> session.</p><p>Let’s have our agent do this work for us this session. The tokens and time we spend now will save us both in the future when working in this repo. The only caveat is that these files will need to be updated whenever there is a significant change in data or architecture, or they risk becoming stale. You can regenerate them with the same prompt or make adjustments that match your setup.</p><p>For our first-time setup, this is the prompt I would use for Mongo:</p><blockquote>“Using the `&lt;MCP name&gt;` MCP , generate `docs/mongodb-&lt;schemaName&gt;-reference.md` so</blockquote><blockquote>future work needn’t re-introspect the DB. Note the env + date.</blockquote><blockquote>For each collection in the &lt;DB Name&gt; DB find via:</blockquote><blockquote>schema, indexes, count, and 1–2 sample docs. Per collection, document its purpose, a field table, indexes (flag unindexed fields), 2–3 copy-pasteable example queries, and gotchas.</blockquote><blockquote>Add an overview table and Mermaid ER diagram up front. Document only what you observe. Mention this file in your agents.md and claude.md”</blockquote><p>This is the prompt I would use for Postgres:</p><blockquote>“With read-only access to the Postgres DB (&lt;table or database here&gt;), generate `docs/postgres-reference.md` so future work needn’t re-introspect it. Note the env + date.</blockquote><blockquote>For each table (enumerated via `information_schema`): columns + types, primary/foreign keys, indexes, row count, and 1–2 sample rows. Per table, document its purpose, a column table, indexes + FKs (flag unindexed filter columns), 2–3 copy-pasteable example queries, and gotchas. Add an overview table and Mermaid ER diagram up front. Document only what you observe.</blockquote><blockquote>Mention this file in your agents.md and claude.md files.”</blockquote><p>The resulting artifacts of this are a Markdown file and a reference to it in your claude.md. The complexity of this can be quite large and uses a fair amount of context. Since it’s not being added directly to your claude.md, it should only be referenced when needed. You can share this with other developers on your team to get immediate benefits even without them setting up their MCPs.</p><p>Every future session starts with verified facts about your database, not assumptions reconstructed from code.</p><h3>From Guessing to Knowing</h3><p>Agents are aware of what’s in the database, which changes how you can test. Unit tests with fixtures still have their place, since they run in CI, independent of any database. But fixtures drift when they aren’t maintained, and with DB access, an agent can validate them against real data and update them when they’ve fallen out of sync. It can also go a step further and end-to-end test the endpoint it just changed. Instead of you hunting down the right test data, the agent already knows what exists and can use it directly. The setup and maintenance steps you’d normally do yourself just disappear.</p><p>That’s testing against data. The bigger win shows up while you’re still writing the code. Database access lets agents test the queries they write. For complicated query languages like MongoDB’s, that’s a tremendous timesaver, since untested aggregation pipelines are error-prone for agents and humans alike. The same goes for implementing APIs: rather than manually checking that data persisted correctly, you can defer to the agent and let it self-correct. It saves the back-and-forth and gives your agents the tools they need to validate their own work.</p><p>Additionally, it allows for conversational data access. You can ask “How many of x entity, of type y, exist with z parameters?” This may require information across multiple collections and deep knowledge of the relationships between the data.</p><p>Rather than banging your head against the wall trying to write this one-off lookup, you can now defer to AI, freeing up your time for other tasks. While I only connect to our development environment or locally, the query it generates is usable in other environments when debugging something in a higher-level environment. So its benefits are not limited to local development work.</p><h3>The Payoff</h3><p>Your database is as fundamental to your projects as the code in your repository, so you can’t expect your agents to perform well without this crucial missing context. Spend these 15 minutes once in this session, and you’ve turned that blind spot into something every future session can read on its own.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=644b1cff0051" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/the-database-is-the-source-of-truth-your-agent-only-reads-the-repo-644b1cff0051">The Database Is the Source of Truth. Your Agent Only Reads the Repo</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Built an Evaluation Harness That Catches Agent Drift Before It Reaches Production]]></title>
            <link>https://medium.com/doubleverify-engineering/how-we-built-an-evaluation-harness-that-catches-agent-drift-before-it-reaches-production-5469ab099a6c?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/5469ab099a6c</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 14:52:36 GMT</pubDate>
            <atom:updated>2026-08-10T14:54:55.118Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By: </em><a href="https://www.linkedin.com/in/rohitgawli91/">Rohit Gawli</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*uIw2jGyEFTru05En6bUiAQ.png" /></figure><h3><em>The demo worked until we swapped models</em></h3><p>Earlier this year, my team started building AI agents that could answer data questions, the kind business users ask, posed in the plain language they use to ask them.</p><p>Think of it like giving a smart junior analyst access to your entire data warehouse and asking: “<em>What’s our coverage rate for this campaign last month?</em>” That’s exactly the question we asked, and the answers come back grounded in real data, no SQL, no dashboard queue. The demo worked. We were excited.</p><p>Then we swapped model versions and asked the new model the same question. The old model had answered 84.1%. The new one said 87.3%. Neither errored. Neither flagged uncertainty. The new model had simply chosen a different denominator, monitored impressions instead of measured impressions, and presented the result with the same confidence.</p><p>No errors, no alerts. Just drift, quietly baked into outputs that were already being copy-pasted into decks.</p><h3>The Issue Wasn’t the Model</h3><p>But the failure wasn’t where we expected it to be. The issue wasn’t the model. What actually breaks a data agent in production? It’s not the model. It’s the question. Data questions are inherently underspecified.</p><p>That matters because a general-purpose LLM won’t tell you it doesn’t know which definition you meant. It guesses, confidently, with no caveats.</p><p><em>“What’s our coverage rate?</em>” could mean five different things depending on the platform, date grain and filter assumptions. Five different SQL queries before you’ve even opened an editor.</p><p>So before writing a single line of harness code, we had to answer a harder question first: <strong>What is a valid data question, and what does an acceptable answer actually look like?</strong></p><p>Without that contract, you’re not evaluating an agent. You’re just vibing. Most teams skip this when the demo is working. That’s a mistake, because the harness is only as good as the definition you start with.</p><h3>What We Built</h3><p>Once we understood the failure mode, we built the harness around three components, keeping in mind that the harness sits between the agent and production. It’s not a unit test. It’s closer to a smoke test plus regression suite for natural language behavior, and it runs on every model update, prompt change and schema modification.</p><h4>Three components make it work</h4><p><strong>A curated question bank</strong>. Boundary cases, not easy ones. Ambiguous metrics, multi-hop reasoning, questions that look answerable but aren’t. This took the most time to build, and it’s the most valuable artifact we have, because you can’t catch failure modes you haven’t defined.</p><p><strong>Expected output schemas</strong>, not exact strings. We check if the output conforms to the right metric, grain and filters, not whether it matches a specific phrase. Exact-match scoring fails here because two correct answers can be structured completely differently.</p><p><strong>A semantic scoring layer</strong>. It flags divergence from expected schemas and gives partial credit for mostly correct answers. In plain terms: if the agent picks the right metric but the wrong date grain, say, weekly coverage when the question implies daily, it scores partial, not zero. That granularity matters when you’re tracking degradation over time, not just pass or fail.</p><h3>What the Harness Taught Us</h3><p>We expected the build to be hard. We didn’t expect the lessons that came after the harness was running.</p><p><strong>Prompt changes are riskier than model upgrades</strong>. A 10-word tweak in our system prompt shifted answer distributions more than a full version bump, because the model is sensitive to framing in ways that aren’t obvious until you measure them. We’d had it completely backward.</p><p><strong>Exact-match scoring is a trap</strong>. “Coverage was 87.3%” and “87.3% of impressions were classified” are the same answer. Scoring them as different drowns you in false positives while missing real failures.</p><p><strong>The harness surfaced bugs we didn’t know we had</strong>. Not because the agent changed, but because we’d never formally defined what “right” looked like. Writing the schemas forced us to have conversations we’d been avoiding. Different parts of the team had different definitions for the same metric. Our own “coverage rate” was the best example: One team computed it against measured impressions, another against monitored impressions. Both were “right” by their own docs. The harness forced us to pick one canonical definition, write it into the schema and make the other an explicitly named variant.</p><p><strong>Latency and cost belong in eval</strong>. An answer that’s 10% more accurate but costs 3x more and takes 4x longer is not a production win, so both belong in the go/no-go gate alongside fidelity.</p><h3>The Four Signals of Production-Ready</h3><p>“Production-ready” means something different depending on what your agent does. For a data agent, it’s not “it answers correctly in staging.” It’s “it answers consistently, within defined tolerance, across versions, at acceptable cost.” We track four signals to know where we stand:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8IT0XANcaIDGcZEE7-Iivg.png" /></figure><h3>Silent Failures Are the Scariest</h3><p>The agent returns something plausible: a number, a trend, a percentage. But it hallucinated a metric or confused two similar fields. No error fires. It just gets used. Monitoring these signals across every release is what gives us the confidence to ship. Without them, “it seemed fine in testing” is the best we can say.</p><p>In general, this means you can’t shortcut eval in analytics the way you might in a general-purpose chatbot. The reason is simple: The blast radius is different.</p><p>A wrong restaurant recommendation gets you a mediocre dinner. A wrong coverage number gets copy-pasted into an executive review.</p><p><strong>Bottom line:</strong> If your agent is touching data that influences decisions, the harness isn’t optional. It’s the thing that earns you the right to deploy, a circuit breaker you can’t skip just because the demo looked good. The agents are getting better fast. The teams that deploy them reliably will be the ones that have figured out when they’re not.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5469ab099a6c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/how-we-built-an-evaluation-harness-that-catches-agent-drift-before-it-reaches-production-5469ab099a6c">How We Built an Evaluation Harness That Catches Agent Drift Before It Reaches Production</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A New Ad Fraud Trend: “AfterCall” Ads]]></title>
            <link>https://medium.com/doubleverify-engineering/a-new-ad-fraud-trend-aftercall-ads-ec44279843b2?source=rss-15c3378cf733------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec44279843b2</guid>
            <category><![CDATA[android-development]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[DV Engineering]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 13:25:34 GMT</pubDate>
            <atom:updated>2026-07-13T13:29:06.886Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Written By:</em> <a href="https://www.linkedin.com/in/nir-danon-333b171a5/">Nir Danon</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*Dz-qoTHwbncy1MoCBZ8mfw.png" /></figure><p>Picture this: you visit your mom, and she asks you to help her with her phone. She tells you, “I think I have a virus.” Every time she hangs up a call, an ad pops up. Sometimes it’s for a game, a shopping service or a random app. Thinking she may have downloaded a suspicious app, you take a look, check her recent apps, scan for shady-looking ones and review her browsing and download history. Nothing suspicious, but the ads keep coming. It looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*rWDwy-_B2yLHCc2MciG-hQ.gif" /><figcaption>[Notice the large ad at the bottom once the phone call ends.]</figcaption></figure><p>We call such ads “AfterCall” ads. They appear at the end of a phone call, outside the app’s normal context. App platforms have always been fighting these out-of-context apps and have implemented different and increasingly effective protection mechanisms. They scan apps before submitting them to the app store and prompt users to uninstall apps that have already been classified as malicious, with apps displaying out-of-context ads at the top of their priority list.</p><p>Yet fraudsters continue to adapt their methods to evade platforms’ efforts and keep pushing these low-quality inventory apps. The DV Fraud Lab has witnessed a suspicious and significant spike in out-of-context apps that use a rather simple mechanism: displaying ads after a user places a phone call on their mobile device, and so dubbed “AfterCall ads”.</p><h3>How It Works</h3><p>All AfterCall apps we have analyzed follow a fairly similar flow, consisting primarily of three parts: requesting special <strong>permissions</strong> from the user under false pretense, a <strong>Broadcast Receiver</strong> to handle end-of-call events and an <strong>Activity</strong> to display ads.</p><h4>The permission that makes it all possible</h4><p>At this stage, you might think, “Wait, how is this allowed?” Can any random app just take over my entire phone and show ads whenever it wants? The answer is, of course, not. But the fraudsters rely on the oldest tricks in the book — deception, and the user’s lack of knowledge — to obtain the permission that allows this behavior.</p><p>Android strictly controls the <strong>“Display over other apps”</strong> permission (SYSTEM_ALERT_WINDOW). Unlike normal permissions, it cannot be granted through the usual in-app pop-up dialog. Instead, the developer redirects the user to the Settings app to make the decision more deliberate.</p><p>Yet malicious developers still manage to work around this safeguard by manipulating the user journey. On the app’s first launch, it requests the user’s permissions, a standard procedure in most apps. However, AfterCall apps usually attempt to deceive the user into granting the “Display over other apps” permission by claiming they need it for legitimate reasons, when in reality it’s used to show AfterCall ads. Most non-technical users simply tap through, unaware of the consequences.</p><p>Below are two examples:</p><ol><li>On the left, we see that during the onboarding stage, it prompts the user with a plausible reason for requesting the permission (which may be true), but the permission is being abused.</li><li>On the right, we see that the fraudulent app blocks the UI until the user grants overlay permissions, prompting most users to agree to continue using the app.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*zI-jAO5ioE0iItJex1bIgg.gif" /></figure><h4>The Call-State Receiver</h4><p>To understand how the fraudsters “hook” the end-of-call event, we first need to understand what Receivers are in Android.</p><p>Android has a distinctive approach to cross-application communication built around <strong>Intents </strong>and <strong>Broadcast Receivers</strong>. The operating system and user apps can send <strong>Intents</strong>, small messages that describe an event. Apps can implement Receivers to listen for these Intents and act accordingly. It resembles a publish–subscribe model, where any app can opt in to receive the broadcast.</p><p>For instance, when you receive a phone call, the TelephonyManager component, which handles calls on the device, broadcasts an ACTION_PHONE_STATE_CHANGED Intent with the state set to RINGING. For example, a typical music streaming app will listen for this Intent and pause the music when it receives it so that you can hear the ringtone. When the call ends, TelephonyManager broadcasts another ACTION_PHONE_STATE_CHANGED Intent with the state set to IDLE, signaling that the call is over. The music app can then resume.</p><p>This kind of communication is part of what makes our phones feel so fluent and responsive. However, this can also be exploited by apps with ulterior motives. Fraudsters often use it to trigger the out-of-context behavior in their apps.</p><p>All apps exhibiting the AfterCall behavior implement their own BroadcastReceiver, which listens for the same telephony intents described above. They wait for the phone call to finish, then fire the ACTION_PHONE_STATE_CHANGED intent with the state set to IDLE, and use that signal to display an ad to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K16lML6jlcZPG1BN1FmgVA.png" /><figcaption>[declaration of the receiver in AndroidManifest.xml]</figcaption></figure><p>In the receiver declaration above, we can notice a few things: first, the receiver sets its priority to an unusually high value (998), likely to run first in case multiple receivers are listening to the same broadcast. We can also see the PHONE_STATE intent (highlighted), which it specifically requests. In addition, it registers for events such as device startup, app updates and power connection, all of which allow the app to persist and stay in memory, increasing its chances of displaying AfterCall ads. Note that the developer appears to have mistakenly placed a permissions declaration inside an intent-filter tag. Such errors reflect how low-quality these apps usually are, which is also visible in their app activity design.</p><h4>The Ad Display Activity</h4><p>While the receiver handles the event logic, the <strong>Activity</strong> handles what the user actually sees. When it opens, it displays an ad like the one shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YghDzncfa3-BYv3NeTLLSA.png" /><figcaption>[Ad display activity from different AfterCall apps]</figcaption></figure><p>There are a few things worth noting here. First, these apps’ activities are remarkably similar, even though the actual apps are very different — they all have the same design: call details, a facade about how it relates to the app, a “profile picture,” and always an ad placed underneath.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kix2b5H1iOSmb-S-nWStwg.png" /><figcaption>[Declaration of the activity in AndroidManifest.xml]</figcaption></figure><p>We can also see from the GIF in the introduction and the activity declaration above that the developers employ evasion techniques. When the AfterCall ad activity appears, it removes itself from the recent apps list. This means that when a user is trying to identify which app is taking over their phone after each phone call, they won’t be able to trace it back to the malicious app, reducing the rate of uninstalls and increasing revenue.</p><p>At first glance, the app icons in the top-right corner may seem like a clue. Still, the developers deliberately select icons that resemble everyday utilities, such as a clock, notes or calendar. For the average non-technical user, these icons appear completely harmless, making it unlikely they’d connect the behavior to the fraudulent app.</p><h3>Detection Difficulties and Impact</h3><p>Detecting the AfterCall fraud trend is not an easy task. A static signature, such as matching a package name or class name, is ineffective since each developer implements their own variation. A dynamic signature of behavioral patterns is also insufficient, as legitimate apps (like caller ID apps) use the same after-call events. As a result, they often slip through the cracks in the platforms’ pre-installation scans. Because they use evasion mechanisms, user reporting alone is often insufficient — which is why collaboration between independent verification providers and platforms is essential. DV’s AI-powered detection methods successfully detect this scheme in all variations we observed.</p><p>The impact is also significant: DV FraudLab uncovers dozens of these apps every month, collectively responsible for hundreds of millions of impressions. Not only is the ad budget wasted, but the user experience is also frustrating, and in turn, it severely damages the brand’s reputation as users associate advertisers with malicious behavior.</p><p>The user experience is the most significant pain point of this tactic. When advertisers invest in ad campaigns, their goal is to build brand awareness and create positive associations with their brand among potential customers. But a user who is repeatedly bombarded with intrusive AfterCall ads, unable to understand where they come from or why they appear, ends up experiencing the opposite effect. Instead of developing trust, the user associates the brand shown in the ad with malicious, disruptive behavior on their device. This is yet another reason advertisers should always use DV’s fraud detection and brand awareness; it is not enough to put in more spend, they must also ensure the ads are served fairly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec44279843b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/doubleverify-engineering/a-new-ad-fraud-trend-aftercall-ads-ec44279843b2">A New Ad Fraud Trend: “AfterCall” Ads</a> was originally published in <a href="https://medium.com/doubleverify-engineering">DoubleVerify Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>