<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Cyrille DUBARRY on Medium]]></title>
        <description><![CDATA[Stories by Cyrille DUBARRY on Medium]]></description>
        <link>https://medium.com/@dubarry?source=rss-d5400d44a383------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TCzSvLr7HtfCnglKelqPZg.png</url>
            <title>Stories by Cyrille DUBARRY on Medium</title>
            <link>https://medium.com/@dubarry?source=rss-d5400d44a383------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:38:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dubarry/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Machine Learning @ Teads — Part II]]></title>
            <link>https://medium.com/teads-engineering/teads-machine-learning-stack-tooling-and-community-of-practice-9a20a980fdf?source=rss-d5400d44a383------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a20a980fdf</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[breeze]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Cyrille DUBARRY]]></dc:creator>
            <pubDate>Tue, 19 Dec 2017 07:01:02 GMT</pubDate>
            <atom:updated>2019-05-07T08:28:51.379Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning @ Teads — Part 2</h3><h4><strong>Stack, production workflow and practice</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*71ySgOI4e_iSE5gEEwSHug.png" /></figure><p><em>In the </em><a href="https://medium.com/teads-engineering/ml1-d84f645132c9"><em>previous post</em></a><em> we talked about why we use Machine Learning at Teads and which particular use cases we work on. In this article we will be covering which technologies we use, why we had to build new solutions and our ML production workflow. We will end with how we actually enhance our ML practice.</em></p><h4><strong>ML Stack and why we do not use MLlib</strong></h4><p>Our stack leverages existing technologies, with Apache <a href="https://spark.apache.org/">Spark</a> sitting at the center. Our training jobs are scheduled by <a href="https://jenkins.io/">Jenkins</a>, their dependencies are handled by <a href="https://github.com/coursier/coursier">Coursier</a>, and they run on AWS <a href="https://aws.amazon.com/fr/emr/">Elastic Map Reduce</a> with training logs and resulting models stored on <a href="https://aws.amazon.com/fr/s3/">S3</a>. We also use <a href="http://jupyter.org/">Jupyter</a> notebook for analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*snzuNtlAXoXtL3f1Ip5iuw.png" /></figure><p>Our specific use cases require to be able to work with large streams of ad delivery events such as bid requests, impressions or complete views and continuously update our models. In this context, using <a href="https://spark.apache.org/docs/latest/ml-guide.html">MLlib</a> seemed to be the right solution.</p><p>MLlib is a native component of Spark dedicated to Machine Learning applications. It provides tools to build ML pipelines as well as common learning algorithms. That being said, <strong>MLlib had major limitations for us</strong>:</p><ul><li>The logistic regression and clustering implementations in Spark use DenseVectors which is incompatible with high-dimensional sparse data like ours.</li><li>In order to avoid any discrepancy, we needed to use the same library for offline trainings and online predictions.</li></ul><h4><strong>Building our own prediction library</strong></h4><p>To get over MLlib’s limitations we decided to build our own library. The runtime is still based on Spark and our library acts as an abstraction layer between Spark and underlying implementations from Breeze and embeds our custom algorithms.</p><p>This library is part of a more general prediction framework that enables to test new experimental approaches and guarantees that the same code is used both online and offline.</p><p>Here is our Machine Learning workflow, we will get into offline/online details later on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*zfcTlIsDchRc_6ivv5eoZQ.png" /><figcaption><em>Machine Learning Workflow — Credit </em><a href="https://github.com/darkjh"><em>JU Han</em></a></figcaption></figure><p>The first step [1] is the actual service making predictions. The generated application logs are then used, together with other sources of data (DMPs, etc.) to build training sets [2].</p><p>For the training jobs [3], these data sets are randomly split into several partitions that are balanced to avoid hotspots. Following this, we manipulate each partition to clean and transform the features before vectorizing them using the <a href="https://en.wikipedia.org/wiki/Feature_hashing">Hashing Trick</a>. For a model with 20 features, the resulting sparse vector can activate up to one million different indices.</p><p>Results are then stored in local matrices used during iterations of the optimization algorithms: <a href="https://en.wikipedia.org/wiki/Limited-memory_BFGS">L-BFGS</a> for logistic regressions and <a href="https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm">Expectation-Maximization</a> for clustering algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*RASzyL2fABLIntsNUmVYEg.png" /><figcaption><em>Focus on step [3] Training job — Credit </em><a href="https://github.com/darkjh"><em>JU Han</em></a></figcaption></figure><p>Training jobs are scheduled using Jenkins and generate a prediction model embedding various weights stored on S3. All these steps are done offline. Once ready, models are used by different online services (back to step [1]).</p><h4><strong>Online model testing using A/B tests</strong></h4><p>Evaluating complex web systems and their impact on user behavior is a challenge of growing importance. A/B tests aim at deciding which algorithm, which home page, which user interface, etc., provides the best results in terms of relevant criteria such as traffic and revenue.</p><p>Temptation is high to conjecture that a system will perform better than another. In fact, <strong>we are wrong most of the time</strong> when it comes to guess what users want or what they are the most reactive to. The only way to evaluate a new system is to test it in a statistically valid framework.</p><p>The use of A/B testing is now widespread in the industry. It compares two versions of a system, A and B, by splitting users randomly into two independent populations to which systems A and B are respectively applied. We use this method to evaluate the impact of our prediction models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*ITg9Av8fpiYhtuoI4J2h_g.png" /><figcaption>A/B Testing — icons from <a href="https://icons8.com/">icons8</a></figcaption></figure><p>We created a small web app to setup and manage the A/B tests and keep track of each experiment. A simple web UI makes it easy to define parameters for different populations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p7OuUfJvpT7V2bOe." /><figcaption><em>A/B Testing setup UI</em></figcaption></figure><p>We usually split the population into three and define one small population to be able to identify if there is a major issue with the experiment.</p><h4><strong><em>A/B test analysis</em></strong></h4><p>We automatically perform general analysis of the results and use notebooks for custom reviews. Special attention is paid to statistical significance, thanks to confidence intervals computation using the <a href="https://projecteuclid.org/download/pdf_1/euclid.aos/1176344552">bootstrap</a> method.</p><p>An important aspect of the analysis is that we observe business metrics like: margin, revenue, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xknTS5Yqs0acsvLkI9hepA.png" /><figcaption><em>Example of completion rate increase for each day of an AB-test, with 80% confidence intervals in dotted lines</em></figcaption></figure><p><strong>A/B tests are great because they are precise and realistic</strong>. However, performing an A/B test takes time (up to several weeks) and is hence costly.</p><p>Moreover, <strong>A/B tests have an impact on production</strong>, we cannot rely on them to select the best model among tens/hundreds of combinations.</p><h4><strong>Offline model testing</strong></h4><p>Since we cannot test all of our models online we also perform offline testing. An offline testing implies several steps, the first one being model training.</p><p>The second step is the model validation. In this step, we want to measure the predictive power of our models by computing validation metrics on a validation set.</p><p>This validation is made by iterations to be as close as possible to the production cycle, as illustrated in the following chart (3 iterations).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*MnL1wLUmQ-uJ0bbNyp5j7g.png" /></figure><p>In particular, the validation metrics we use include the <strong>weighted Mean Square Error</strong> (MSE), using specific weights for each use cases.</p><p>For example, when we want to predict billable events, we weigh the results according to the amount to be paid when the event occurs. Thus, we focus on the MSE of the expected revenue (Cost per View or CPV) and not the MSE of a billable probability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/171/0*qm4gG-s8Fg7MP-bR." /></figure><p>We created a tool, called Datakinator, to facilitate the creation of homogeneous experiments that all respect the same testing protocol. This tool also archive them to keep track of the results.</p><p>Using Datakinator, we are able to efficiently create tens of experiments without having to worry about weighing the MSE or defining the protocol. It also simplifies later comparisons between models.</p><h4>How we practice Machine Learning</h4><p>At Teads we are organized in <a href="http://featureteamprimer.org/feature_team_primer12.pdf">feature teams</a> and several of them are becoming autonomous in using Machine Learning, like Buying Engine (campaign delivery optimization) and SSP (yield optimization).</p><p>As this dynamic is growing, we decided to set up a <strong>dedicated Community of Practice</strong> (CoP).</p><p>This community brings together ML practitioners but is also open to people interested in this field. Our Machine Learning community usually meets once a month to:</p><ul><li>Foster this discipline across all feature teams</li><li>Share knowledge and stay up-to-date with the latest research (reading groups)</li><li>Homogenize technologies, tools, code and practices</li><li>Pick interesting conferences to attend to</li><li>Define a common recruiting and onboarding process for newcomers</li><li>Promote the community through presentations, documentation … and Medium posts like this one</li></ul><p>Community of practices are <strong>great to discuss, brainstorm on cross feature team challenges and help each other</strong>. Finding the right format and pace is crucial to make it work. We found important to define a CoP leader and give him latitude to embrace this pivotal role.</p><p>We are constantly looking for great teammates in both our offices in Paris and Montpellier. If you are interested in <a href="https://teads.tv/teads-jobs/">joining</a>, feel free to <strong>contact us </strong>on Medium or give us a <a href="/service/mailto:mathilde.barthe@teads.tv">shout</a>.</p><p><em>By </em><a href="https://medium.com/u/ec3cbdc8778"><em>Benjamin DAVY</em></a><em> and </em><a href="https://medium.com/u/d5400d44a383"><em>Cyrille DUBARRY</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a20a980fdf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/teads-engineering/teads-machine-learning-stack-tooling-and-community-of-practice-9a20a980fdf">Machine Learning @ Teads — Part II</a> was originally published in <a href="https://medium.com/teads-engineering">Teads Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning @ Teads]]></title>
            <link>https://medium.com/teads-engineering/ml1-d84f645132c9?source=rss-d5400d44a383------2</link>
            <guid isPermaLink="false">https://medium.com/p/d84f645132c9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programmatic]]></category>
            <category><![CDATA[adtech]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Cyrille DUBARRY]]></dc:creator>
            <pubDate>Tue, 24 Oct 2017 06:01:00 GMT</pubDate>
            <atom:updated>2018-01-02T13:15:20.386Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning @ Teads (Part 1)</h3><h4><strong>4 use cases from the AdTech industry</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vs-MG7raKHewnSqqxo2GDg.png" /></figure><p><strong>Digital Advertising is an astonishing Machine Learning playground</strong>, it combines data rich activities, scaling challenges and a lot of automation, especially since the rise of Programmatic buying and selling of ads in real-time.</p><p>With <strong>20 billion page views</strong> and more than <strong>3 billion unique viewer IDs each month</strong> we are now reaching interesting volumes for our algorithms.</p><p>In this first post we will describe some of the Machine Learning use cases that we have been working on:</p><ul><li>View-through rate prediction</li><li>Broken creative detection</li><li>Bid-request relevancy prediction</li><li>Look-alike modeling</li></ul><h4><strong><em>View-through rate prediction (VTR)</em></strong></h4><p>When we started experimenting machine learning two years ago, we wanted to <strong>predict the probability for a video to be watched for more than <em>x</em> seconds</strong>, according to the advertiser requirement. This prediction aims at only showing the most interesting ads for the users. Considering that at Teads we charge buyers only when an Ad is viewed, another benefit of using this model is that it avoids the waste of inventory. When the predicted view-through rate for a given advertiser is too low, the display opportunity is free for someone else to take it.</p><p>At that time, <strong>different possible options were available</strong> to build a Machine Learning system. Ranging from open source initiatives like Spark MLlib, Scikit-Learn, etc. to the recently introduced managed Machine Learning service from Amazon. As we were already using Spark to compute analytics jobs we initially studied MLlib.</p><p>Unfortunately, <strong>MLlib had (and still has) major limitations</strong> for our use cases. One of them is that Spark uses <em>DenseVectors</em> in its logistic regression implementation, which is incompatible with high-dimensional sparse data like ours. We also wanted to be able to use the same code for offline training and online predictions to avoid discrepancies.</p><p>This led us to <strong>develop our own prediction library</strong>, acting as an abstraction layer between Spark and underlying implementations from Breeze and embeds our custom algorithms. This library is part of a more general prediction framework that enables us to test new experimental approaches and guarantees that the same code is used both online and offline.</p><p>Coming back to the VTR prediction, we used our framework to design an efficient model by tuning its hyper-parameters (training duration, regularization, optimization, …) and selecting an appropriate set of features.</p><p>These features include information from the advertiser, the publisher, the user and various interactions amongst them. For example, some ads have a better VTR on specific websites or on specific users.</p><p>To be effective, the resulting model <strong>learns 10^5+ parameters from 10^7+ examples</strong> and is updated every few hours. It takes 1ms to perform this prediction.</p><h4><strong><em>Broken creative detection</em></strong></h4><p>In another attempt to avoid inventory waste, we adopted a machine learning approach to <strong>detect broken video ads</strong>, in the sense that they cannot be played properly. The fact is, we cannot simply assess if a creative is « broken » as it can change over time and could be due to many reasons:</p><ul><li>It can be directly related to the availability and quality of the creative’s files,</li><li>Also, it could be due to the creative’s behavior, depending on the context of execution. This case is impossible to test as it would require to assess all the combinations of publishers (webpages) and user contexts (OS and Browser type &amp; version).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*FWZrNv5OgvhUE0Ci-DnvHg.png" /><figcaption><em>An ad can be considered broken even when it can be displayed properly in some contexts</em></figcaption></figure><p>Thus, we needed to make a prediction as to when a video is likely to be broken so that we no longer try to display it.</p><p>This gives the opportunity for other advertisers to display their ads, and for the publishers to increase their fill rate.</p><p>This algorithm <strong>predicts if a creative is unlikely to start</strong> and is able to distinguish between many different contexts.</p><h4><strong><em>Bid-request relevancy prediction</em></strong></h4><p>Alongside the previous studies we embraced Programmatic, an automated and non-managed way of delivering ads. With Programmatic, Teads opened its platform to external demand (DSPs). Prior to this work, whenever a slot was available to display an ad to a user, our SSP used to systematically send a bid request to all the connected DSPs.</p><p>This was rather <strong>inefficient</strong> for both sides, it caused a great waste of network resources and a useless load for our connected DSPs. The challenge was here to learn which requests were interesting to the buyers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*QJLM3NtirKTHyBe6SeMW4A.png" /><figcaption><em>Simplified Real Time Bidding process (RTB) — icons from </em><a href="https://icons8.com/">icons8</a></figcaption></figure><p>To tackle this issue, we developed a model that computes the <strong>probability that a given bid request will trigger a response from a given DSP</strong>. Using this model, only the most relevant users/contexts are sent. On the buyer side, only a high quality inventory will be seen, this will simplify its filtering process and improve the overall performance.</p><p>We used a logistic regression model to classify the different requests depending on their probability to get a response. Then we defined a threshold above which requests are sent. Several feature combinations have been studied to adjust the model. The tuning of the classifier’s parameters was done during an offline experiment protocol.</p><p>As a result, we achieved a significant reduction of calls that were previously considered pointless. On average <strong>we are able to cut 60%</strong> <strong>of the Bid Requests</strong> without impacting the delivery for both DSPs and Publishers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*uup_ClBcuEM4DDTzYz9Ufg.png" /><figcaption><em>Throttling effect on the Network Out volumes generated by Bid Requests, last uptrend is specific to that day.</em></figcaption></figure><p>However, traffic reduction will never be optimal, our models need to stay reactive when the market evolves. Hence, <strong>we need to keep a high exploration rate</strong> and continue sending calls that have a low response probability to detect any behavior changes from the market.</p><h4><strong><em>Look-alike Modeling (on-going)</em></strong></h4><p>Some of our advertisers know the particular users who are highly interested in their brand or products and wish to promote to a similar audience. We plan to leverage our first party data and our machine learning tools to compute similarities between any user and a target audience.</p><p><strong>User clustering based on browsing history</strong> is a promising way of addressing this challenge. This will allow the advertisers to deliver their ads on the most interested and interesting users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*oLA7umgrom-KnujS6OOQFg.png" /><figcaption><em>How to identify users with similar interests ?</em></figcaption></figure><h4><strong><em>What’s next</em></strong></h4><p>As one can imagine, there is an <strong>infinite source of applications</strong> for Machine Learning in the fast paced AdTech environment. In particular, other research topics address industry challenges like:</p><ul><li>Campaign conversion,</li><li>Cross-device user identification,</li><li>Inventory forecasting,</li><li>Dynamic Creative Optimization,</li><li>And <strong>many other cool things</strong> …</li></ul><p>Machine Learning gives us the possibility to improve the advertising experience on all sides and most importantly scale our business. There is still a lot to do and <strong>we are growing our team of Data Scientists &amp; ML Engineers</strong> in both our offices in Paris and Montpellier to tackle these exciting challenges.</p><p>If you are interested in <a href="https://teads.tv/teads-jobs/">joining</a>, you are more than welcome, feel free to <strong>contact us </strong>on Medium or give us a <a href="/service/mailto:mathilde.barthe@teads.tv">shout</a>.</p><p>On the <a href="https://medium.com/teads-engineering/teads-machine-learning-stack-tooling-and-community-of-practice-9a20a980fdf">next article</a> we <strong>talk about our tech stack</strong> and the tooling we built to efficiently train and A/B test our prediction models.</p><p><a href="https://medium.com/teads-engineering/teads-machine-learning-stack-tooling-and-community-of-practice-9a20a980fdf">Machine Learning @ Teads (part 2)</a></p><p>Do not hesitate to ask any question, or comment if you wish a specific topic to be discussed!</p><p><em>By </em><a href="https://medium.com/u/ec3cbdc8778"><em>Benjamin DAVY</em></a><em> and </em><a href="https://medium.com/u/d5400d44a383"><em>Cyrille DUBARRY</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d84f645132c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/teads-engineering/ml1-d84f645132c9">Machine Learning @ Teads</a> was originally published in <a href="https://medium.com/teads-engineering">Teads Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>