<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Moloco Tech Blog on Medium]]></title>
        <description><![CDATA[Stories by Moloco Tech Blog on Medium]]></description>
        <link>https://medium.com/@molocotechblog?source=rss-890e66df8bdd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*oeswDah0z25yj9R43Jq1UA.jpeg</url>
            <title>Stories by Moloco Tech Blog on Medium</title>
            <link>https://medium.com/@molocotechblog?source=rss-890e66df8bdd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 03:19:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@molocotechblog/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Quality Muscle]]></title>
            <link>https://medium.com/@molocotechblog/the-quality-muscle-scaling-with-uncertainty-e221562b08b8?source=rss-890e66df8bdd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e221562b08b8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[moloco]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Moloco Tech Blog]]></dc:creator>
            <pubDate>Sat, 14 Dec 2024 23:09:39 GMT</pubDate>
            <atom:updated>2024-12-16T19:43:43.907Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qyzp8NJgK9YBd5vzkkzTIg.jpeg" /></figure><p>By <a href="https://www.linkedin.com/in/harpreet-singh-chadha/">Harpreet Chadha</a>, Director, Core ML Products and Programs</p><p>Building large software systems traditionally relied on a level of determinism in the output from the system. Scalable machine learning systems have now enabled a richer set of use cases by dealing with uncertainty. Similar to traditional software systems, ML systems still rely on deterministic pipelines to process, validate, store, analyze, and annotate large volumes of log data. What’s newer is the added quality muscle that enables ML based products and teams to be comfortable with uncertainty in the output from subsystems. In this blog, we dive into how to successfully scale with uncertainty, which includes culture, metrics, A/B test frameworks, and decision-making.</p><p>A year back, we were looking for opportunities to significantly improve campaign performance for Moloco’s customers. A month into Moloco myself, it was clear that there was no silver bullet. Instead, the teams had lots of ideas with uncertain performance impact, which was a great opportunity to exercise the quality muscle. As an ML-first company, most of the pieces were already there. With a bit more focus around a shared performance improvement objective for the organization, the pieces came together to transform the trajectory of experiments, launches, and performance impact.</p><ul><li>First and foremost, we already have an ML engineering team <strong>culture </strong>that inherently rewards experimentation with new models, new algorithms, new data, and other system improvements. Most importantly, engineers who use ML to develop models are inherently comfortable with failures of individual experiments, and with the fact that gains from each change are less predictable, and that those gains may be much smaller or much larger in the real world as compared to offline estimates. So are data scientists adept at assessing statistical significance in the results from offline analysis and online experiments. Probability is their friend. They have an appetite for uncertainty, humility that we may not have the perfect solution yet, and a hunger to continually iterate to a better model and higher impact.</li><li>Second is alignment on <strong>metrics </strong>to evaluate the impact of experiments. At the senior leadership level, that alignment is easier if there is a single global metric across launches that is correlated with a key business goal. For performance marketing, that global metric typically is some form of ROAS — short for Return on Advertiser Spend — that leadership can evaluate with each launch, and that customers can observe in performance metrics for their campaigns. At the team level, that also requires alignment on a series of metrics that the ML, Data Science, and Engineering teams highlight in evaluating the impact of each product change. The additional metrics typically serve as leading indicators for the key metric, or informative measures of ecosystem impact.</li><li>Third is the availability of an <strong>A/B test</strong> framework for the engineers to run lots of experiments in parallel, and have automated results with confidence intervals against each metric easily accessible in a central dashboard. That engenders a culture of experimentation to rapidly test the impact of new hypotheses, new algorithms, parameter changes, and much more.</li><li>Fourth is agreeing on a <strong>decision making</strong> process to quickly approve requests to ramp traffic when test results show significant and positive impact on the key metrics. A best practice is a lightweight email based approval process with a clear set of decision makers to approve a change, along with an agreed window to approve the changes. The result is more data and zero meetings, unless live discussion is useful to assess impact of competing metrics, e.g., high cost to integrate a new data source that shows a significant performance upside.</li></ul><p>Together, a healthy experimentation culture, aligned set of metrics, reliable test frameworks, and a streamlined decision-making process enable amazing scale for an org. Even junior ML engineers new to the organization can test, evaluate, and safely ramp traffic on changes that have uncertain impact.</p><p>In addition to all the above, you have to be in a business that can afford ML at scale. You also need a reliable infrastructure to process a very high volume of real time data and address data quality issues quickly: it is much easier to evaluate impact of a new model when you have clean data, versus puzzling over test results if the model learns from bad data.</p><p>Instead of focusing on a single idea, the quality muscle enables a continual focus to explore a range of options to drive improvements across install, action, revenue, pricing, impression frequency, and other models. A pleasant surprise over the last year is the impact from hardware accelerators. From an infrastructure perspective, we expected higher computation speed and hence lower costs. What became evident was that the move to accelerators also enabled a faster feedback loop from idea to offline evaluation and live experimentation, resulting in testing deeper models and obtaining large performance gains from algorithm changes. A similar surprise came from experiments with the frequency of ad impressions that a user sees. Experiments to test performance improvements for a handful of customers with multiple campaigns led to a launch with outsized performance gain across many customers. A healthy quality muscle enables rapidly graduating scores of such ideas to experiments and decisions to launch or pause, based on expected impact. There is a healthy appreciation that quality systems are hard, yet also super exciting to work on.</p><p>For an engineer or product manager or program manager or data scientist, that means working at the cutting edge of ML and computer science. A good ads quality muscle brings high energy with bottoms up ideas, clarity on impact through good metrics, regular opportunities for success with A/B tests, fast progress with streamlined decision making, and so much more.</p><p>If interested in career opportunities at Moloco, visit <a href="http://moloco.com/careers">Moloco.com/careers</a> for more information. You can also subscribe to the MolocoLife <a href="https://www.youtube.com/@molocolife">Youtube</a> channel and follow us on <a href="https://www.instagram.com/molocolife">Instagram</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e221562b08b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>