<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andrew Yates on Medium]]></title>
        <description><![CDATA[Stories by Andrew Yates on Medium]]></description>
        <link>https://medium.com/@promoted?source=rss-48973c0b7bfa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Q0NxFBfVGwZgZavyP2Z2YQ.png</url>
            <title>Stories by Andrew Yates on Medium</title>
            <link>https://medium.com/@promoted?source=rss-48973c0b7bfa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:46:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@promoted/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Promoted Joins Dropbox Dash!]]></title>
            <link>https://medium.com/promoted/were-joining-dropbox-dash-f8394969d2af?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8394969d2af</guid>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 21:58:53 GMT</pubDate>
            <atom:updated>2025-04-29T01:23:58.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYozLt3A2MLW7VB8u1KPGw.jpeg" /></figure><p>From the beginning, our mission at Promoted.ai has been to help people find what they’re looking for through smarter, more relevant, and more effective experiences. Today, we’re thrilled to share the next step in that journey: <strong>Dropbox has acquired Promoted.ai</strong>, and we’ll be joining the team to build <a href="https://dash.dropbox.com/"><strong>Dropbox Dash</strong></a>, an AI-powered search experience for business collaboration and productivity.</p><p>We’ve always believed that great technology should feel seamless, human, and empowering — and Dropbox shares that vision. Their mission to <em>design a more enlightened way of working</em> deeply resonates with us. But what truly drew us to Dropbox goes beyond the mission — it’s the people, the culture, and the values. Dropbox’s core values — <em>be worthy of trust, they win, we win, keep it simple, own it, make work human</em> — reflect the same principles that have guided us at Promoted: trust, clarity, and shared success.</p><p>We’re incredibly grateful to our customers, partners, and team members who’ve supported us along the way. We’re excited to bring our technology and experience to a broader audience, and to shape the future of work — together with Dropbox.</p><p>Let’s go!</p><p><em>Andrew Yates, ex-CEO<br>Promoted.ai — Dropbox</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8394969d2af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/were-joining-dropbox-dash-f8394969d2af">Promoted Joins Dropbox Dash!</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leading High-Impact Teams from ML to AI: A CEO’s Perspective]]></title>
            <link>https://promoted.medium.com/leading-high-impact-teams-from-ml-to-ai-a-ceos-perspective-280e8d12052e?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/280e8d12052e</guid>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 21:14:54 GMT</pubDate>
            <atom:updated>2025-01-07T21:14:54.635Z</atom:updated>
            <content:encoded><![CDATA[<p>Guiding organizations through the transition from traditional machine learning to generative AI requires vision, strategy, and trust in the engineering leaders who turn ideas into reality. Below are the critical areas I focus on as a CEO hiring and managing top engineering talent to drive this evolution.</p><h3>1. Recognize the New AI Landscape</h3><p>Generative AI changes what’s possible with data, software, and products. Where ML once delivered incremental predictions, AI now can create text, images, and even code. Leaders overseeing high-stakes transitions must think beyond incremental improvements and empower teams to explore untapped potential in product design, personalization, and user engagement.</p><h3>2. Hire and Mentor Engineering Leaders with Range</h3><p>Experienced leaders with a background in traditional ML and robust data processing are essential. I look for individuals who can also pivot toward the nuanced demands of generative AI. Beyond knowledge of algorithms, these leaders must excel at architectural design, GPU optimization, and large-scale deployment — all while guiding cross-functional teams to adopt new capabilities.</p><h3>3. Build on Proven Strengths in Data and Architecture</h3><p>Generative AI hinges on data quality. The efficient pipelines and governance frameworks that worked in ML remain key for AI. I ensure my engineering leaders reinforce these fundamentals, while also scaling infrastructure to handle compute-intensive tasks like training and GPU inference. A reliable backbone keeps models performing consistently under production loads.</p><h3>4. Rethink AI’s Potential and Constraints</h3><p>Success demands openness to new possibilities and acceptance of increased complexity. Generative models can be less predictable than classic ML approaches, requiring sophisticated guardrails and testing protocols. I task my engineering directors with balancing creativity and risk management to deliver AI-driven features that users trust and value.</p><h3>5. Support Experimentation and Ongoing Learning</h3><p>Creating a culture of learning is non-negotiable. Generative AI is evolving fast, so I encourage leaders to stay curious and regularly refine their techniques. This involves dedicating resources to R&amp;D, supporting continuous upskilling, and celebrating discoveries — even if they come from unexpected places within the organization.</p><h3>Final Thoughts</h3><p>Leading the shift from ML to AI begins with strong leadership: a blend of technical prowess, strategic thinking, and people skills. By hiring exceptional engineering managers, nurturing a learning culture, and recognizing both the power and complexity of generative AI, CEOs and executive product leaders can position their organizations at the forefront of this transformative wave.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=280e8d12052e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Do You Mean by “Relevance”]]></title>
            <link>https://medium.com/promoted/what-do-you-mean-by-relevance-c7c21bcc0cd7?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7c21bcc0cd7</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Sat, 16 Nov 2024 03:28:15 GMT</pubDate>
            <atom:updated>2024-11-16T03:28:15.816Z</atom:updated>
            <content:encoded><![CDATA[<p>Consider that the AI boom comes from “solving relevancy hard enough” when “just predicting clicks” wasn’t good enough at Google and Meta.</p><p><strong>Predicting clicks isn’t enough to solve human-like relevancy</strong>. Five years ago, the state of the art was that if you could predict clicks well enough — if you collected enough data and with enough features, then you could solve relevancy. Not quite. As ML more correctly predicts clicks over traditional BM25 keyword search, search, recommendations, and ads can feel <em>less</em> relevant. This is true even when adding semantic content embeddings or click-trained text embeddings as features in click models, which may not improve click predictions much. Predicting clicks better still helps but doesn’t always solve relevancy. Adding personalization, conversion optimization, and especially profitability adjustments can make relevancy appear even worse, even though engagement metrics can still be better in A/B testing.</p><p>Relevance is ultimately a human judgment and, therefore, subjective. Predicting better clicks wasn’t enough. The breakthrough was large language models (LLMs). These models do not predict clicks but produce human-like judgments about relationships in text and images. LLMs are related to modern “content embeddings,” representing text and media for ML applications. Content embeddings are not new, but the newest embeddings are so powerful that they are effectively used as new technologies. LLMs excel at modeling subtle human judgments about meaning but can’t compute the probability of clicks, sales, and revenue.</p><p>At Promoted, we make search and ads more relevant by combining the best “old ML” to predict clicks and conversions using statistical learning and “new AI” for subjective human judgment with LLMs. Two years ago, we thought we could get perfect search and ads if we built a big enough click model. Many companies still believe this. We help our customers evolve past this thinking by modeling all different types of relevance, isolating each definition into a subsystem that dedicated teams can optimize independently. Then, we combine all relevance models flexibly to achieve the best results.</p><p><strong>Ranking by semantic relevance will probably <em>not</em> maximize engagement or profitability. </strong>You still want old-style ML for predicting clicks and conversions in commercial medial search. This is especially true for ads, which work by showing somewhat less relevant results but for higher profitability. However, you also cannot show irrelevant results, and if you show less relevant results, you want to make more profit. Promoted typically uses semantic relevance as a <em>constraint</em> for different positions: top positions require the best relevance, and lower positions can be less relevant.</p><p>Subject to all results exceeding some semantic relevancy threshold, we maximize other objectives like profitability, ad revenue, or marketplace dynamic objectives like diversity and exploration. That way, no matter what other systems do, the result will always “appear relevant” to the end user.</p><p><strong>Semantic Relevance</strong></p><p>“Semantic Relevance” is how a human reviewer would rate relevance. When people complain that “ads are bad” or “search doesn’t work,” they typically mean this type of relevance. Any rubric could work here; we use a 1 to 5 rubric by default, though we can customize this for different domains as needed.</p><p>When we work with engineering teams, to focus this discussion on the common sense use of the product and away from technical details of AI engineering, we use the “mother test:” <strong>“If you show this to your mother, what would she say?” </strong>The answer has to exclusively use information that’s obvious to a non-expert human using the product <em>and cannot use numerical tabular data like click rates</em> (because your mother wouldn’t know what that means). The goal is to stop discussing technical details like embeddings and GPUs and to<em> open the product</em> and <em>look at it</em>. If search and ads look wrong, they are <em>wrong</em>. We focus on fixing that before launching engagement and profitability maximization using ML.</p><p>Promoted uses a <a href="https://dash.readme.com/project/promoted/v1.0/docs/default-semantic-relevance-rubric">1-to-5 human rating rubric template</a> that we customize for customers, domains, and different product experiences, like search, related products, or recommendations. Iteratively customizing this rubric is important — semantic relevance is frequently best solved on a “when you see it, you can recognize it” iteratively method of refinement and experience. LLMs are fantastic at this type of scoring and can be better than human reviewers because they can be calibrated to the judgment of your best product and merchandising leaders.</p><p>Downstream consumers include Semantic Relevance include Blender for trimming and allocation rules and other ML systems like click prediction as a feature.</p><p><strong>Other Considerations for using Semantic Relevance in Search and Ads</strong></p><p><strong>Mean versus Variance</strong>: Different people can reasonably give different judgements of relevance, and this is true for repeated runs of the same LLM. We observe that some query and item pairs have more stable scores, while others can reasonably have more variance. We expect a variance of about +/- 1 rating, and this is the maximum precision, which limits the lowest achievable loss from fitting a statistical model before overfitting. We use the mean rating after repeated ratings, and we also record the label variance. When we observe a higher variance in labeling for a query &lt;&gt; item pair than expected, we send this to manual human review.</p><p><strong>Undefined Relevance</strong>: Relevance is not always defined. We allow a “relevance undefined” rating for when the query is malformed or has content safety issue or when the item rated is malformed. For some domains, queries may be search functions outside of semantic search, like seller names or exact IDs, which may use a different search system. These are also scored as “undefined.” Excluding undefined relevance labels from statistical relevance model training and evaluation improves the model result and reduces noise. When the query has undefined relevance, we can make blender rules to not use relevance as a consideration for allocation.</p><p><strong>Tokens versus Numbers</strong>: LLMs return token IDs which are converted to strings. They do not return numbers. We treat the output of LLMs like user reviewer input in that it could be unreliable or malformed and needs to be converted from strings to numerical types.</p><p><strong>Multiple Relevance Rubrics and Scores</strong>: We presented a single rubric of 1 to 5. However, some domains may have multiple definitions of semantic relevance for different experiences or contexts.</p><p><strong>Manual Fungibility</strong>: Unlike click predictions or “goodness” scores, Semantic Relevance inferences are interchangeable with manual human annotation. If I, the product leader, say that Listing X is a “5: Perfect Match” for Query Y, that’s the “real answer” for the purpose of delivery. If there is a big difference between what our semantic relevance models infer and what the expert reviewer rates, then the next regular relevance model training will automatically account for this to improve inference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7c21bcc0cd7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/what-do-you-mean-by-relevance-c7c21bcc0cd7">What Do You Mean by “Relevance”</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CEO Andrew on About Promoted.ai]]></title>
            <link>https://medium.com/promoted/ceo-andrew-on-about-promoted-ai-5140090f266c?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/5140090f266c</guid>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Thu, 08 Aug 2024 00:00:56 GMT</pubDate>
            <atom:updated>2024-08-08T00:00:56.866Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FyRegmSFA68s%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DyRegmSFA68s&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FyRegmSFA68s%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/eccdde0aaddbeef9674dfc335f8cb53a/href">https://medium.com/media/eccdde0aaddbeef9674dfc335f8cb53a/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5140090f266c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/ceo-andrew-on-about-promoted-ai-5140090f266c">CEO Andrew on About Promoted.ai</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Unusually Comprehensive Review of Position Bias Correction Methods in Search and Ads Ranking]]></title>
            <link>https://medium.com/promoted/an-unusually-comprehensive-review-of-position-bias-correction-in-search-and-ads-ranking-d1fe0ff69904?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1fe0ff69904</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Thu, 30 May 2024 23:59:13 GMT</pubDate>
            <atom:updated>2024-05-31T00:07:25.167Z</atom:updated>
            <content:encoded><![CDATA[<p>“Position Bias” in search, feed, and ads ranking is when the probability of engagement, like a click, is confounded by a bias from where the item was shown in the past. Generally, the top ranks are more engaging. Items allocated to the top get more engagement regardless of how good that item was for that allocation. Because engagement models learn from past engagement models, this can result in a bias where past winners always remain winners, and the model can never learn otherwise.</p><figure><img alt="Some positions generate more user engagement than others, irrespective of the item allocated in that position. This position bias can depend on placement rank, media, device, surface, context, relative position, rendering, and other product and user experience details which may evolve over time." src="https://cdn-images-1.medium.com/max/710/1*8u_8AYgRzt5s5tedb0ZXeQ.png" /><figcaption>Figure 1: An overall description of Position Bias. Positions are enumerated in rough order of decreasing prominence. Some positions (typically, the first few) have a much higher ambient engagement probability.</figcaption></figure><p>In ads ranking, position bias also impacts ad pricing when auction prices depend on the bids for multiple positions in an allocation. In general, for GSP and VCG ad auctions, uncorrected position bias tends to inflate prices for top positions because the value of lower, less valuable positions is relatively overestimated.</p><p>This is a survey of method for correcting position bias, including:</p><ul><li>Eventing</li><li>L2 Machine Learning Feature Design</li><li>L2 Model Architecture</li><li>Offline Evaluation of L2 Models with Bias Correction</li><li>Metrics and Counters</li><li>L3 Model Design.</li></ul><p><a href="http://promoted.ai/">Promoted.ai</a> uses these methods to provide the best possible search and ads ranking performance for enterprise customers. In some cases, we provide experimental evidence on real search ranking data to demonstrate the impact of position bias correction.</p><figure><img alt="Without correction, A will always be ranked first in position 1 because past history always shows A with the highest p(click). However, B and C are equally good (and maybe even better!) The model never learns this without position bias correction." src="https://cdn-images-1.medium.com/max/848/1*Xg2tbBw9VMuFKKFF060DQA.png" /><figcaption>Figure 2: An illustration of an uncorrected position bias feedback loop that causes poor results.</figcaption></figure><p><strong>Survey of Methods for Correcting Position Bias</strong></p><h3>User Eventing</h3><h4>Visibility Eventing in p(click) modeling</h4><p>The most common and extreme position bias results when an allocation position is likely never presented to a user. If the item allocated to a position is never presented to a user, it can never generate engagement. An engagement model is typically used to estimate “quality,” which assumes presentation to the user. An engagement model where some negative examples are never presented to users, especially when the likelihood of presentation to users is influenced by past engagement modeling, generates a spurious negative bias.</p><p>Promoted <a href="https://www.promoted.ai/client-metrics-libraries/">logs impressions using IAB visibility standards</a> and uses impressions as negative examples in our p(click) models, not server-side insertions.</p><h4>Post-Click Modeling</h4><p>A second way Promoted uses eventing to address position bias is post-click modeling, where all training examples are clicks, and the labels are some user engagement after a click. Once a user has clicked or navigated to a list item, the allocation position’s impact on additional, more highly considered user behavior like “purchase” is largely mitigated because the user has already decided to select that item. This “Markov model” of user engagement isolates the impact of position on the impression-to-click model. Because most of the position bias is mitigated by the design post-click models, the post-click model fits using other non-position-bias features that are more likely to model the actual quality of the item for this user in this context after sufficient user consideration. We can observe this effect by including allocation position in post-click models and confirming that the allocation position is not an important feature for these models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9d3HVYuy0SJ2brDEjriKsg.png" /><figcaption>Figure 3: Promoted uses a <a href="https://docs.promoted.ai/docs/delivery-system-diagrams#user-state-transition-inference-graph">user state transition model</a> to design engagement model training examples and labels.</figcaption></figure><h3>L2 Machine Learning Feature Design</h3><h4>Allocation Position Features</h4><p>Some features, like “position,” are used in training but set to 0 at inference time. This is because the feature value depends on the allocation, which isn’t known at the time of L2 inference. (In multi-stage search and ads ranking, L2 is a reranking of the top K results from a retrieval system where K is typically about ≤1,000). There may be multiple “allocation position” features, like overall position, carousel position, relative page position, horizontal position, ads rank position, etc. All such features are handled similarly: set for training to model position bias but set to 0 at L2 inference time.</p><p>In effect, this models the impact of the position feature in training and subtracts its bias at inference. This is similar in concept to “batch effect correction” techniques used in other sciences. Furthermore, the top position tends to be the most commonly sampled in training data, so engagement at the top position is the most well-modeled compared to any other position.</p><h4>Context Features</h4><p>The allocation position alone may not provide enough information to model the impact of position bias if different devices, surfaces, and displays are combined in the training set. To add this context, we include context features like device type, page size, and product surface and model the impact of the position feature in interaction with these context features.</p><h4>Exclusion of Retrieval Features During Installation</h4><p>Promoted sometimes uses the retrieval score or rank as a ranking feature in L2 second-stage ranking. However, when adding a new L2 second-stage model on top of a traditional L1 search ranking system, the retrieval rank and score are proxies for the allocation position in the existing system. That is, the retrieval rank and score exactly or nearly exactly represent the allocation position. Using retrieval score and rank in this case can circumvent other special treatments we use to correct position bias.</p><p>When training on data generated by the existing system, particularly when bootstrapping a new L2 ranking system, we exclude retrieval score and rank as features in L2 ranking models. Later, after deploying the L2 and decoupling the direct relationship between retrieval score and rank and allocation position, we sometimes add retrieval ranks and scores back as L2 ranking features if that significantly improves L2 model fit. Sometimes, this is because our customers do custom modeling to generate retrieval scores on data Promoted cannot easily access, like domain-level computation of term-frequency as used in computing BM25 scores commonly used as an L1 ranking score.</p><h3>L2 Model Architecture</h3><p>When including allocation position as a feature in ranking models, we often observe that the model overfits on the allocation position, particularly for p(click|impression) models, at the exclusion of other features. This is partly because of allocation position bias, but it is also because of a confounding bias that high engagement probability items are likely allocated in first positions, and past delivery generates future training data. The result is a model that overfits to a signal of “goodness,” the position as decided by a past model, that isn’t available at inference time. This wastes model complexity and results in poor performance.</p><p>To mitigate this impact, we fit a combination of model architectures that include and exclude the allocation features in different layers. For example, we remove the position feature from GBDT component or dense connected NN component. Instead, we concatenate position feature along with other numerical features (optional) at different neural network layers.</p><p>Generally, while we use autoML to find ideal model architectures depending on the domain, customer, and data volume, we find that <strong>Neural Network (NN) architectures outperform GBDT alone when accounting for position bias, even at medium data volumes</strong> where GBDTs tend to perform better than NN on position biased data. This may be because the discreet “branching” of GBDT causes all splits on the position feature to be discarded at inference when position is set to 0, thereby wasting considerable model complexity never used at inference time. NN can model a continuous relationship with position and does not have this issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*0ebgfrkHOiXO6yMAs9CvYw.png" /><figcaption>Figure 4: a simplified model architecture from Promoted.ai 2022 showing how position features are only used in some of the neural network to predict engagement. Position features are included with numeric features but excluded from the GBDT that generates categorical features.</figcaption></figure><h3>Offline Evaluation of L2 Models with Bias Correction</h3><p>Ideally, the impact of position bias correction is measured using live A/B testing data where users can react differently to different allocation positions. However, this may be infeasible, and an offline model evaluation metric is still desirable to demonstrate a mechanistic improvement in model performance before spending resources to collect A/B testing results. However, evaluation data generated from past delivery data will have biased labels from the position bias, making it challenging to measure improvements in correcting the position bias with that data. We used offline evaluation techniques to estimate improvements from correcting position bias.</p><p><strong>Biased data set</strong></p><p><strong>[</strong>V1<strong>]</strong> click/impression data with allocation position as feature in both training and evaluation sets. This is the standard hold-out validation data used in most L2 model training without correction.</p><ul><li>Pros: Align with training data, where position is used as features</li><li>Cons: Does not align with inference distribution, where position = 0. Uncorrected position bias performs best on V1, so using V1 to estimate the positive impact of position correction is challenging.</li></ul><p><strong>Unbiased data set [Estimation of the true data set]</strong></p><p>[V2] full click/impression data, set position = 0 in evaluation</p><ul><li>Pros: Consistent with the inference distribution where the position is set to 0. No distortion for position = 0 in evaluation.</li><li>Cons: User responses and therefore evaluation labels are still biased by position</li></ul><p>[V3] Subset of impression with attributed conversions only, set position = 0</p><ul><li>Pros: Consistent with the inference distribution where the position is set to 0. No distortion for position = 0 in evaluation. An attributed conversion could indicate true matching intent regardless of allocation position.</li><li>Cons: Very small subset evaluation data makes measurement noisy, attributed conversion bias correction assumption may not be correct.</li></ul><p>[V4] click/impression data with real position = 0, other positions are discarded</p><ul><li>Pros: Consistent with the inference distribution where the position is set to 0. No evaluation data are distorted.</li><li>Cons: Small subset of evaluation data. Does not directly test the impact of position on inference.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E10l0QG49Pi-OvLzcC_oEQ.png" /><figcaption>Figure 5: A table of different L2 model architectures, evaluation sets, and evaluation metrics to select a position bias correction. In this case, we selected “model architecture 8.” Note that Neural Networks (NN) outperform GBDT when accounting for position bias for this dataset.</figcaption></figure><h3>Metrics and Counters</h3><p>Average engagement rates are potent predictors of future engagement. However, the same bias in engagement labels applies to engagement features, too. Counter features and related measurements like average click-through rate (CTR) are also affected by position bias. This bias reduces the reliability of counter features, resulting in worse model fits and lower production impact.</p><p>For example, suppose an item is consistently delivered in the first position, and there is a strong position bias that items in the first position are likelier to be clicked regardless of the items allocated to the first position. In that case, CTR counter features computed using clicks for that item will be inflated compared to those calculated for items typically delivered in lower positions.</p><p>We correct this bias by computing the average engagement rate per position in a context (e.g., iOS position 1, 2, 3… web position 1, 2, 3… etc.) and generating “position bias-corrected” engagement rate counter features and metrics. Generally, positions with lower engagement rates “count more,” while positions with higher engagement rates “count less” in aggregate metrics and ratios.</p><h4>Position-engagement Weighted Impressions</h4><p>For CTR counter features, we correct the impression denominator by making impressions weigh more proportionally to how biased that position is towards engagement compared to a global engagement rate for the context. This allows us to easily compute aggregate sums of weighted impressions for all real-time engagement ratio counter features use using impressions. That is:</p><p>Rather than counting Impressions 1 by 1 for each, count for an impression at position “i” in a context, for example, iOS, count: <em>CTR@pos(i,iOS)/CTR(allPositions,iOS)</em></p><p>If the position bias for position i is higher than the average (e.g., for position 1), then this impression will count for more than 1. The ratio will be proportionally lower when this weighted impression is used as a denominator in an average CTR counter feature sum(clicks) / sum(weighted impressions). Likewise, the ratio will be proportionally higher for impressions in low positions with less than average CTR.</p><p>For streaming efficiency, the position bias can be computed as a table published daily in a recurring data pipeline and saved in memory in a standard streaming accumulator using a system like Flink.</p><h4><strong>Position Bias Corrected Engagement Metrics</strong></h4><p>For some experiments, like ad load tuning, changing ad slot positions can change the click rates. This complicates typical A/B quality measures, like CTR, clicks, and ad revenue, if ad revenue includes cost-per-click billing. Generally, ad slots in higher positions will inflate click-related metrics, so such metrics cannot be reliably used to measure user experience quality.</p><p>Similar to impressions with engagement weights, weigh engagements like clicks by the inverse position engagement, so that higher, more engaging positions count less and vice versa. For example, for a click iOS in position <em>i</em>, weigh it by: <em>CTR(allPositions,iOS)/CTR@pos(i,iOS)</em></p><h4>Past position counter features</h4><p>Counter features can also accumulate allocated position itself to learn corrections for position bias inherent in past data. At Promoted, we recommend accumulating the impression log-transformed allocation position transformed with dimensions for different product surfaces (search, home page, related products) and devices (iOS, web, mobile web). Add this feature to all ordinary engagement models in combination with allocation position features and historical engagement features and labels to learn a contextual correction of past delivery allocation position patterns to predict future delivery.</p><h3>L3 Model Design</h3><p>“L3” ranking is any allocation system where the priority method includes a dependency on each allocation position. For engineers familiar with Meta-style organic and ads ranking, these systems were typically the “Newsfeed ranking” and “prediction calibration” systems. “L2” is more similar to the AdFinder system, which traditionally computed predictions for items in a position and allocation-independent way. L3 systems typically use the L2 scores as features and models using features computed from allocation positions and other items in the allocation.</p><h4>Non-mutating “Calibration” models</h4><p>These adjustments do not change the allocation. Instead, they are used in ads ranking to correct (typically lower) event predictions in lower position to improve ad auction pricing performance (typically, lower prices and help the auction behave more similarly to a bid-your-true-value dominant strategy auction)</p><p>Promoted can use three types of non-mutating calibration models to improve pricing performance for ads:</p><ul><li>A simple “position curve” fixed multiple, like 100%, 90%, 70%, 60%, 50%, 50%, …</li><li>A model with position and some context as features. This is similar to the position curve, but regularly trained on historical per-position miscalibrations between predictions at position 0 and observed events at an allocated position in a context.</li><li>A real-time pacing system to force the average calibration per item to be near 100% for small windows of time, like an hour, as a correction to the pCTR model as compared to streaming measurement. This can also help mitigate the production impact of ML production failures that result in much too high or too low predictions that cause over and under-delivery of content.</li></ul><h4>Whole Page Optimization</h4><p>Whole page optimization, slate learning, and sequence learning all use techniques of recomputing model inferences for every position. This is a complex topic and beyond the scope of merely handling positions in allocation. Generally, because inference is recomputed for each allocation position, the allocated position is available as a feature, and position bias is modeled directly. Additional after-allocation calibration corrections may still be applied even when directly modeling and computing inferences for each position to align allocation-attribution outcome measurements with per-item inferences.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1fe0ff69904" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/an-unusually-comprehensive-review-of-position-bias-correction-in-search-and-ads-ranking-d1fe0ff69904">An Unusually Comprehensive Review of Position Bias Correction Methods in Search and Ads Ranking</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons Learned using GPUs in Search Ranking at Medium Scales]]></title>
            <link>https://medium.com/promoted/lessons-learned-using-gpus-in-search-ranking-at-medium-scales-6969ea6315f0?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/6969ea6315f0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Fri, 24 May 2024 18:34:40 GMT</pubDate>
            <atom:updated>2024-05-24T19:00:24.680Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By: </em><a href="https://www.promoted.ai/about-us/#james-hill"><em>James Hill</em></a><em>, ML Infra Engineer @ Promoted.ai</em></p><p>GPUs are the latest craze for AI and machine learning. We ported our L2 “second-stage” pClick and pConversion search and ads ranking models to GPUs. In conclusion: we now support GPU for inference. We <em>can</em> use it… we just usually don’t. It’s not economical except for our biggest customers (public company, public facing search use case, &gt;10M searches a day).</p><p>Here’s what we learned.</p><p>We observed mediocre latency improvements (&lt;5ms) and no cost improvements on our average insertion count for most customers. They just don’t have enough items to return on complex enough models to greatly benefit. Only for our largest customers approaching CPU limits, we could see modest infra costs improvements.</p><p>For most search engineering teams asking us if we use GPU, I think “We support it” is a fine answer. If an ML expert asks us, I think the best answer might be “For training or inference?” If they say <em>inference</em>, I think there’s a few relevant talking points we can take from all the investigation. (If they don’t care about the difference, then that’s an awareness and education issue versus a technical specification issue, and let’s focus on building trust in that case.)</p><ul><li>Latency improvement from GPU is strongly tied to model architecture, and we prefer focusing on improving quality as long as the latency is “good enough”</li><li>The vast majority of GPU support in the common ML libraries is focused on training. For inference, you may actually spend more time on auxiliary work (even in C++) than actually using the model</li><li><strong>Basic instances with GPU are ~3x the cost</strong> of even compute-optimized CPU instances</li><li>I didn’t do a cost-performance comparison (where we compare against an instance with 3x as much CPU) because I think it kind of misses the point. I think CPU would win here — but because of our model architecture and not because of how we’re using devices</li><li>Given the above point about auxiliary work, then add in feature loading, Blender, additional complexity with sponsored search, etc. <strong>we’d prefer to just have more CPU in general in Delivery</strong></li></ul><p>We continue to prioritize more and more complex models and more different types of models. Having GPU support will help us expand in both scale and complexity. We don’t see inference compute as a problematic constraint in the foreseeable future, and doubly so for very large use cases where GPUs would help most. It won’t help with small to medium sized customers with limited traffic (&lt;10M queries per day) and smaller models (still using GBDT directly or as a NN feature transform).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a_5eIHrPY4cVg-zqu1gk5Q.png" /><figcaption>GPU hard at work predicting clicks in search</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6969ea6315f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/lessons-learned-using-gpus-in-search-ranking-at-medium-scales-6969ea6315f0">Lessons Learned using GPUs in Search Ranking at Medium Scales</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Ads Dynamic Bidding]]></title>
            <link>https://medium.com/promoted/how-to-build-ads-dynamic-bidding-b29cb706a9d0?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/b29cb706a9d0</guid>
            <category><![CDATA[ad-auction]]></category>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Fri, 26 Apr 2024 07:02:40 GMT</pubDate>
            <atom:updated>2024-04-27T18:23:14.185Z</atom:updated>
            <content:encoded><![CDATA[<p>Amazon Sponsored Products (ads) has an ad bidding product called “<a href="https://advertising.amazon.com/library/guides/dynamic-bidding-sponsored-products">Dynamic Bidding</a>.” It works by bidding up to 200% of your cost-per-click bid to increase ad-attributed sales. <a href="https://advertising.amazon.com/help/GCU2BUWJH2W3A8Z7">From Amazon</a>:</p><blockquote>“With dynamic bids - up and down&#39; strategy, Amazon will increase your bids in for clicks that may be more likely to convert to a sale. With this strategy, we’ll also reduce your bids for clicks that are less likely to convert to a sale… With the fixed bids [CPC] strategy, Amazon will use your exact bid for all opportunities and won’t adjust your bids. Compared to dynamic bidding strategies, you may get more impressions, but fewer conversions for your ad spend with this strategy</blockquote><p>Dynamic bidding works by making a cost-per-click (CPC) ad auction more similar to an ideal ad auction by bidding for sales using machine learning while keeping the CPC click bid in the insertion bid formula. Here’s why dynamic bidding works and how to implement it. This guide is designed for tech leads and product managers creating dynamic bidding in their marketplace or e-commerce app.</p><h3>Step by Step Dynamic Bid Equation Derivation</h3><h4>Step 1: Start with CPC Bid Equation</h4><p>Assume you have a calibrated click prediction model to predict the probability of a click given a specific insertion. It may be personalized and contextual. In simple CPC systems, p(click|impression) may be implemented as a global or per advertiser constant rate for all insertions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*OeFu1OeB91jXUeSZmnrt9g.png" /><figcaption>Equation 1: CPC insertion bid equation</figcaption></figure><h4><strong>Step 2: Show the implied relationship to sales</strong></h4><p><strong>Expected Sales</strong>: Advertisers don’t value clicks. They value sales. Implicitly, advertisers expect some incremental sales per click when bidding for clicks. A <a href="https://www.facebook.com/business/help/552097218528551">lift test</a> would measure this as total incremental ad sales divided by total ad clicks.<em> Note that these are not necessarily “post-click” or “last-click” sales.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*ste9VJyG0BTlxntg-5cZ8g.png" /><figcaption>Equation 2: Implied expected incremental sales from winning an insertion</figcaption></figure><p><strong>Implied CPA Bid</strong>: Likewise, implicitly, advertisers are willing to pay some cost per incremental ad sale, or “Implied CPA Bid,” when bidding for clicks. A past willingness to pay for sales for cost-per-click ad delivery estimates an effective lower bound on the value-per-sale to the advertiser on a cost-per-click basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*TH1PHD3LKSyKZsL3dkkwdQ.png" /><figcaption>Equation 3: B: Implied Sale Bid, or Cost-per-action (CPA) bid.</figcaption></figure><h4>Step 3: Re-combine to show the relationship</h4><p>Multiplying Equations 2 and 3 together, we get back the original cost-per-click Equation 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w544l6EhqippZSrxwi-z0Q.png" /><figcaption>Equation 4: Recombining factors to show the relationship</figcaption></figure><h4>Step 4: Implement “sales per click” numerator with ML model</h4><p>𝑢(Δsale/click) is assumed to be a constant for every ad click in the CPC bid formula. However, some ad clicks are more likely to generate sales than others. Model this directly with a calibrated machine learning model to replace 𝑢(Δsale/click) in the numerator with a post-click conversion model <em>p</em>(sale|click). A reasonable implementation of p(sale|click) uses de-duplicated clicks and sales per insertion, p(∃sale|∃click|insertion), to better represent a user state transition model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*3aXPhGDQFMpOc7x9Wn0CIg.png" /><figcaption>Equation 5: Implement the numerator with post-click sales model, p(sale|click).</figcaption></figure><h4>Step 5: Implement “clicks per sale” denominator to preserve a “stable” mean of 1</h4><p>We can’t re-implement 𝑢(Δsale/click) in the denominator with the same <em>p</em>(sale|click) model because then the ratio between numerator and denominator would always be 1, and there would be no difference from the CPC bid Equation 1. Going back to Step 2, consider that the value-per-action to the advertiser is stable — it may vary over weeks or seasons, but per day, it’s a constant. Reasonable implementations include:</p><ul><li><strong>OK</strong>: The observed average sale per click rate. This estimator is problematic for ads with few recent attributed sales, which may be most ads.</li><li><strong>Better</strong>: The average <em>p</em>(sale|click) prediction over a week or some other “reasonable” observation period. This is like borrowing information across ads when data are sparse. When the per-ad <em>p</em>(sale|click) numerator is persistently miscalibrated, the denominator will also be miscalibrated by the same factor, canceling out the bias.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*I4tZ3aORrS9tRsWBwAI41A.png" /><figcaption>Equation 5: Implement the denominator with the average post-click prediction used in the numerator.</figcaption></figure><p>The ratio of <em>p</em>(sale|click) to avg(<em>p</em>(sale|click)) is the raw bid multiplier, see Equation X. It represents how much higher or lower to bid than the average per click as proportionate to the expected sale attributed to a click on the insertion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*1XMqke0YV1P9_EFHdFOCIg.png" /><figcaption>Equation 6: the “raw” dynamic bid multiplier</figcaption></figure><h4>Step 6: Add Bid Caps, Transform to “Dynamic Bid”</h4><p>The raw bid multiplier is unbounded. To provide the guarantee and avoid unreasonable overbidding, cap the bid multiplier to the guaranteed limits. The upper cap is 200% or 150% for “up and down” and 100% for “down only.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*Auj-fKaPbUp4B2ez_FOauQ.png" /><figcaption>Equation 7: Bid multipliers with limits</figcaption></figure><p>The “Dynamic Bid” is the capped bid multiplier by the click bid. It can be used as a direct replacement for the ClickBid in any CPC auction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*E9XRT4Uc1qC0NGzcb6vMiw.png" /><figcaption>Equation 8: The CPC Dynamic Bid is the Bid Multiplier from Equation 7 by the CPC Max Click Bid</figcaption></figure><p>Below are additional discussion topics on building and designing Dynamic Bidding for ads.</p><h3>Characteristics of a useful p(sale|click) model</h3><p>For Dynamic Bidding to work (see “The Ideal Ad Auction” below), the p(sale|click) model must correctly distinguish between high and low-value clicks for the <em>same ad</em>. This means the model must include features that are either or both:</p><ul><li><strong>Contextual</strong>: certain pages, queries, devices, products, times, or surfaces are higher value than others for the same ad.</li><li><strong>Personalized</strong>: clicks by some users are worth more than others, including authenticated versus unauthenticated users.</li></ul><p>Many ad systems start with a per-ad average rate to implement terms like <em>p</em>(click|insertion) and <em>p</em>(sale|click). Per the discussion above, a constant per ad simplifies to the CPC bid equation, which defeats the purpose of using Dynamic Bidding.</p><p>Further, many ML systems in Search and Discovery are per-item estimates of “quality.” Or, features are predominately per-item historical rates or item features like “price.” These features may help distinguish between overall “better” ads and help explain bias in a personalized and contextual model to improve fit. Still, they will not generate a useful p(sale|click) model for Dynamic Bidding.</p><p>To verify that a p(sale|click) model is promising for use in Dynamic Bidding using offline analysis:</p><ul><li>Verify that some feature importance is attributed to user and contextual features (e.g., regression weights, GBDT feature importance measures, NN dropout loss)</li><li>Show that per-item (versus global) normalized entropy and AUC are promising (&lt;100% and &gt;50% respectively).</li></ul><p>Refinements to p(sale|click) could account for incrementality or the expected value of the sale, including the quantity or order value, probability of cancelation or return, and delivery costs. However, all these improvements must still be realized as improvements per ad versus modeling across ads to work with Dynamic Bidding.</p><h4>Incrementality</h4><p><em>p</em>(sale|click), or “post click sale,” is a model of “incremental sales” <em>caused by</em> an ad impression. (<a href="https://docs.promoted.ai/docs/delivery-system-diagrams#click-conversion-and-post-purchase-joining-and-attribution">See how Promoted estimates incrementality for conversion joining</a>). It’s important to optimize towards “incremental” sales because if the ad does not cause the user to buy, for example, then showing that the ad will not increase sales in live production. Not all post-click sales are incremental, not all non-post-click (aka “view-through”) sales are not incremental, and sales can be “fractionally” incremental. However, an increase in post-click ad sales is usually highly correlated with the total lift in true incremental sales as measured in lift tests. Therefore, <em>p</em>(sale|click) is typically a directionally good estimate of incremental sales, but it may be miscalibrated.</p><p><strong>Example</strong>: Assume a uniform random 50% probability that a “true incremental” conversion has an ad click, and nearly 100% of post-click ad conversions are 100% incremental. Then, a <em>p</em>(click|imp)<em>p</em>(sale|click) model predicts incremental sales with a miscalibration constant of 50%. This miscalibration may not impact the relative priority in value allocation, so <em>p</em>(sale|click) is appropriate for search and ads ranking without adjustments. However, the miscalibration may impact ad pricing and ad value estimation.</p><h3>The Ideal Ad Auction</h3><p>The goal of Dynamic Bidding is to move the CPC ad product closer to an ideal ad auction. In the ideal ad auction, each advertiser bids exactly their expected value in dollars from winning each ad insertion to maximize advertiser profits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muhS-RD3EWSH_4sntWryPA.png" /><figcaption>Equation 9: “Bid your true value” for every ad insertion auction is the theoretical ideal.</figcaption></figure><p>Advertisers who most value each ad insertion win that ad insertion. Given no interactions between insertions, this maximizes total platform advertiser value and total advertiser profits, and given a “reasonable” reserve price, it maximizes platform ad revenue.</p><p>However, in a fixed bids CPC ad auction, advertisers can only bid a fixed price per click, see Equation 1. This CPC design has three problems in practice:</p><ol><li>Advertisers cannot directly bid for each ad insertion. They can only bid for clicks. However, advertisers don’t value clicks directly. They value incremental sales.</li><li>Some clicks are more valuable than others because some are more likely to convert to a sale. However, a “fixed bid” cost-per-click (CPC) bid values all clicks equally.</li><li>Real auctions are not <a href="https://en.wikipedia.org/wiki/Strategic_dominance">“dominant strategy” </a>or “bid your true value” auctions. Maximizing the allocation’s (assumed) advertiser value does not necessarily maximize profits because advertisers may be able to win more profit by bidding lower, and therefore, advertiser bids are not reliable signals of true value.</li></ol><p>Dynamic Bidding addresses allocation maximization problems (1) and (2), allowing the system to bid for “true value” more correctly per insertion on behalf of advertisers. The more correct the value estimates used in bidding, the more that estimated value will be realized in expectation. Problem (3) relates to pricing and bid incentive theory, which are beyond the scope of Dynamic Bidding (this definition). Assume that any advertiser value measurement using past advertiser bids is a lower bound of true advertiser value in most auction designs.</p><p>In summary, bidding closer to the ideal generates more ad value and, therefore, more ad revenue.</p><h3>Measuring Value</h3><p>In brief: sum sales. More details below.</p><p>The goal of Dynamic Bidding is to <strong>increase ad value</strong>. Ad value is the total value to advertisers from ads and the maximum sustainable ad revenue that can be billed. Advertisers value sales, so ad value for Dynamic Bidding is the <strong>sum of incremental sales</strong>. Not all sales are valued equally. They are each valued by, at least, the <strong>effective CPA Bid</strong> or cost-per-sale, see Equation 3. That is, advertisers are willing to pay for more sales rather than others. This is an important consideration in ads value and potential ads revenue.</p><p>If Dynamic Bidding is working correctly, then compared to CPC, Dynamic Bidding will:</p><ul><li><strong>more correctly predict advertiser value (in sales)</strong> per insertion better than any per-ad constant multiplier of any click prediction model. Measure this using per-ad distribution of Normalized Entropy (&lt;100% is good) and AUC (&gt;50% is good). If any constant is better, then either Dynamic Bidding will not outperform CPC, or CPC can perform as well as or better Dynamic Bidding with fixed CPC bid adjustments.</li><li><strong>change the allocation of ads</strong>. If there is no substantial allocation change of ads, then there can be no difference in ad value generated (prices may differ, but this should be negligible if the allocation is unchanged)</li><li><strong>increase total ad value</strong> <em>because ads are more correctly allocated to generate more ad value</em>. Assuming ad volume is constant when compared to CPC ad delivery volume, the total ad system must have become more efficient overall and in expectation per advertiser to generate more total sum ad value if total sum ad value increases.</li><li><strong>does not penalize specific advertisers for realizing value</strong>. If a particular advertiser is heavily penalized, and the additional ad value in an A/B test derives from the penalty of this advertiser, then in production, that advertiser will rationally refuse to use Dynamic Bidding, and the increase in ad value will not be realized.</li></ul><h4>Ad Value Metrics</h4><p><strong>Ad Value</strong> is the sum of incremental ad sales weighted by each sale’s value to the advertiser. The Cost-per-sale bid is a lower bound for each sale’s value. See Equation 3: Implied Sale Bid, or Cost-per-action (CPA) bid.</p><p>Generally, ad-attributed conversions that are rarer and have higher click bids are worth more ad value because they have a higher implied conversion bid. An increase in such higher ad value conversions supports higher ad revenues. This is related to an increase in ad CVR but where some conversions are worth more than others in billable advertiser value.</p><p>In practice, the effective cost-per-action bid can be difficult to estimate, especially for advertisers with sparse historical information. Therefore, all things being equal, more attributed ad sales (conversion rates (CVR), total sales) and more ad revenue are reasonable proxies for increases in ad value.</p><p>An easier heuristic to compute than effective cost-per-action bid is to use the sum click-bid-weighted attributed conversions. This assumes that all conversions have equal conversion rates (false). It is an improvement over CVR and sum sales because it accounts for different click bids, and click bids are not estimates and, therefore, are more appropriate for use in metric definitions.</p><p>Ads Value Metrics for A/B testing from simple to more advanced. Note that the dollar value of sales in Ads Value <em>derives from the bidded advertising potential revenue</em>, not the sale price, sale profitability, or cost-of-revenue.</p><ul><li><strong>Total sales </strong>and CVR. All things being equal, total sales should increase.</li><li><strong>Post-click sales</strong> and CVR. Post-click sales are directionally a better estimate of incrementally.</li><li><strong>Click-bid weighted post-click sales</strong> in $. The click bid is one of two terms used in the impliedCPABid, and it’s not an estimate, so it can be computed exactly, making it a more appropriate metric. Generally, sales are worth more in ads value proportional to how much an advertiser is willing to pay per click.</li><li><strong>Implied CPA-weighted post-click sales</strong> in $. Sales that require more clicks to realize in expectation are worth more in ad value. However, this rate is an estimate, and so is uncertain and may be unobservable for some low-volume advertisers. Uncertainty is an undesirable property of a launch metric even though this is technically a more correct metric. A reasonable heuristic is to assume a global rate that shrinks to an observed post-click sales rate when data are available. If all advertisers used the global rate, then this metric would be functionally the same as Click-bid weighted post-click sales.</li></ul><p>Ad Revenue should also increase because the Ad Value increases. However, it may require advertisers to raise bids or budgets to realize the revenue depending on the ad auction design and billing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYOg8PXxntheHCTHL2GLpg.png" /><figcaption>Equation 10: Ad Value formulas of increasing complexity.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b29cb706a9d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/how-to-build-ads-dynamic-bidding-b29cb706a9d0">How to Build Ads Dynamic Bidding</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Label Definitions in Search and Ads]]></title>
            <link>https://medium.com/promoted/ml-label-definitions-in-search-and-ads-769ed0df7980?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/769ed0df7980</guid>
            <category><![CDATA[advertising]]></category>
            <category><![CDATA[marketplaces]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Sat, 16 Sep 2023 00:49:41 GMT</pubDate>
            <atom:updated>2023-09-16T00:49:41.102Z</atom:updated>
            <content:encoded><![CDATA[<p>To get the maximum performance of scaled recommendation and ad ML systems, model complexity is not the only consideration. What you optimize, or “label definitions,” are also critical to achieving that last +5% in profits at scale. When spending engineering months to reduce model loss by -0.1%, carefully modeling what you have optimized and why can be comparably better investments. Labeling improvements do not cost inference infrastructure costs to deploy and gains compound with model complexity improvements like larger models and more features. For ads, label definitions also determine <em>ad pricing</em>. Large advertisers will demand (and may audit) definitions of billable ad metrics, and advanced ad delivery systems like ROAS optimization will be more tractable to design when ad prices are well-defined.</p><p>Good ML Labeling has the following characteristics:</p><ul><li><strong>Semantics</strong>: predictions mean something</li><li><strong>Casualty</strong>: a decision made by an inference “causes” what is predicted</li><li><strong>State</strong>: limit interference to what’s uncertain</li></ul><h3>Semantics: Why Semantic Inferences</h3><p>Promoted’s model predictions, or “inferences,” are <em>semantic</em>, or have a human-intelligible meaning that is useful analytically. This means inferences should have:</p><ul><li><strong>units</strong> of measurement</li><li>statistical conditional <strong>formulas</strong> p() and E()</li><li><strong>calibration</strong> with real measurements, i.e., the sum of observed clicks should equal the sum of predicted clicks.</li></ul><p>Without inference semantics, there is no systematic way for engineers to improve, combine, or monitor these inference systems other than with continuous trial-and-error tuning in live production with user response metrics. We call these “<strong>biology experiments</strong>,” because the management of this process resembles running cell culture experiments where the true mechanisms cannot be manipulated or known exactly. The results are highly variable and take a long time. This is an inefficient state for an engineering organization compared to solving an equation.</p><p>The “semantic inference” approach contrasts with most recommendation systems, where inference definitions are loosely understood and do not have units. Sometimes, this is by technical necessity. For example, “embedding vector similarity” has no “meaning” other than the relative meaning that higher scores are “better” than lower scores in the context of a single search request. Another example is a ranking score produced by a learn-to-rank (LTR) model. While these techniques can produce an ordering of listings by “goodness” to achieve increased business metrics like increased sales or clicks in search as verified by production A/B testing, there is no observable user response metric to validate the correctness of each inference directly, and there is no clear analytical method to combine these inferences into higher-level models.</p><p>Prediction semantics become critical when building ad systems because the inference value, not just its relative value, is used to compute ad prices. The price will not be intelligible if the inference does not align with the billable or optimized event. This applies to any term included in an ad bid or “utility function,” including “quality scores.”</p><p>Even for non-ad systems, it’s easier to reason about well-defined events. Example benefits include:</p><ul><li>Well-understood methods of using a more common event (like checkout) as a proxy for a more rare but correlated event (like purchase) in both utility functions (proxy objectives) and multi-task modeling labels (e.g., augment sparse purchase labels with similar but more common checkout labels)</li><li>Sensible calibration models enforce event predictions to observed predictions continuously, which help smooth sudden changes in optimization system performance from either user behavior or system failure</li><li>Arithmetic composition of inferences to achieve other higher-level models</li><li>More sensibly designed data-driven attribution models</li></ul><p><strong>Prioritizing Inference Semantics</strong>: In small inference systems, inference semantics are less important because at small volumes of user responses, small improvements in quality are not statistically measurable, and the improvement from no quality control to any reasonable quality control system is large. Therefore, inference semantics may be neglected in original search and recommendation system designs. At large scales, and especially at large e-commerce and tech companies, inference semantics become relatively important in accomplishing realized user metric improvements in sales and clicks compared to increasingly marginal quality improvements from machine learning achieved with more complex models. Furthermore, inference semantics are an orthogonal improvement to model improvements, facilitate the use of more complex models through model stacking, make debugging and analyzing the operation of inference systems more manageable, and, unlike increasing model complexity, do not incur recurring infrastructure costs in computation or latency. The trade-off is that it can be organizationally infeasible to change inference semantics in a large organization across many systems without strong design leadership, synchronization across multiple teams, and potential confusion in measuring impact when previous KPIs are questioned.</p><h4>Promoted’s Semantic Inference for “Click” or “Navigate”</h4><p>Here is our formula. We have similar formulas for all events that we predict.</p><p><em>p(click) = p(any(navigate|joinwindow)|insertion with IAB impression @ relativePosition=0)</em></p><p>A Navigate (aka click) is when a user actively and conscientiously selects a response insertion with a tap or mouse click to “Navigate” to another view, like a product details page. The event is not the Page View of the destination page navigated-to, although this event may be used to construct or infer a navigate event if necessary. “Navigate” comes from iOS nomenclature.</p><p><strong>Definitions:</strong></p><ul><li><strong>P</strong>: a calibrated probability between 0% and 100%. Predictions over 100% indicate an error.</li><li><strong>Any</strong>: a deduplicated navigate, or, there is “any” navigate attributed to this insertion. We deduplicate to the insertion. Per <a href="https://www.iab.com/wp-content/uploads/2015/06/click-measurement-guidelines2009.pdf">2009 IAB Guidelines</a>, we use the “One-Click-Per-Impression Method.”</li><li><strong>Joinwindow</strong>: A click has a limited amount of time to occur before it’s no longer counted as a “valid click” for optimization purposes. For example, one hour. For example, you may load a screen, go to sleep, and then click a search result. We discard that click for optimization purposes. Implementing an unbounded join window is technically infeasible (there will be some limit in practice).</li><li><strong>Insertion</strong>: all training examples (and model inferences) must be mapped to a “response insertion,” or a decision from the backend sent to the user for display. It may not necessarily be displayed (that is an “impression”)</li><li><strong>IAB impression</strong>: all training examples must be an IAB impression. In the absence of an attributed navigate, this requires sufficient visibility logging. If a navigate is attributed to the response insertion by insertion ID, this is an IAB impression (because the user must have seen the insertion to use it for navigation). This is not true for inferred attribution of navigates.</li><li><strong>relativePosition=0</strong>: this is the first position on the page, or the “offset” for paged examples. At inference time, all insertions already share the same paging offset. It’s only the relative position from the offset that cannot be known at inference time for independent point-estimate L2 ranking. To set a pageOffset while setting the position to 0 at inference time is not a good approximation because this combination would never be seen in training data. If multiple position features (e.g., relative and absolute) need to be set as if allocating in the first position in the current page (i.e. 0 and pageOffset respectively).</li></ul><h4>Promoted’s Semantic Inference for “Purchase”</h4><p>A Purchase is our default “objective.” If there is only one “conversion” event, we ask customer to map them to “purchase.”</p><p><em>p(conv) = p(any(conv) * f(credit) | insertion with IAB impression with any(click) @ relativePosition0)</em></p><p><strong>Definitions:</strong></p><p>Definitions are similar to “Navigate” above. However, unlike Navigate, a single Purchase may be joined to many response insertions according to a multi-touch attribution model. There may be multiple attribution models that assign different credit values for the same purchase-insertion join in the future.</p><ul><li><strong>f(credit):</strong> is the value assigned by the multi-touch attribution model. If f(credit) = 0, then the purchase may not be joined to a response insertion (but not necessarily).</li><li><strong>any(conv) </strong>: Was there any distinct purchase decision for this item for this user? A “distinct” purchase decision is a heuristic defined as 24h deduplication windows for all purchases of the same contentID by the same authenticated userID. For optimization purposes, purchases outside this join window for the same item for the same user with new order IDs are “new” purchase decisions with their own attribution logic. Multiple purchases and the post-purchase value is modeled separately and is not described here.</li><li><strong>Post-click “With any click”: </strong>We decompose attributed purchase credit to “clicks” and “non-clicks.” Most or all of the attribution credit is assigned to “click.” When we assign credit to non-clicks, the total purchase attribution summed over all attributed clicks may be less than 100%.</li><li><strong>View-Through</strong>: The fraction of conversion credit attributed to non-clicked insertions.</li></ul><h3>Causality: Solving for Incrementality</h3><p>All predictions are trying to capture “causality.” We want to predict the probability that showing an insertion will <em>cause</em> a response. It’s possible to predict the probability of an event without a causal relationship (e.g., a sale that would happen anyway). Such “correlations” inferences are unlikely to cause sales to increase when used to decide what content to deliver to users in live production.</p><p>We use concepts in ads measurement to model “incrementality” which we apply to our ML labeling systems for inference. This makes sense: you predict what you measure, and to predict incremental value, you need an a model of incrementality.</p><p>Sometimes, in ad systems, causality modeling is disregarded to boost “credit assignment” in return-on-ad-spend (ROAS) calculations. This merely shifts the assignment to credit ads more from the same pool of sales from the e-commerce app or marketplace versus creating new sales as true causality modeling would. We measure this by measuring all sales across both ads and organic systems. We strongly prefer casualty modeling across native ads and organic insertions when possible.</p><p>Causality is primarily a consideration for “conversions,” like purchases. We use a combination of last-click, multi-touch, and data-driven attribution models across insertions, sometimes with weights for the duration of time since the last user action on an insertion from the time to the conversion.</p><h3><strong>State</strong>: limiting interference to the uncertain</h3><p><strong>Stateful Actions</strong>: For some “stateful” events like “save” or “checkout,” once the user has performed this action on an item, additional actions by the same user on the same item are either ambiguous (multiple checkouts on the same item isn’t necessarily better or a higher order value) or undefined (you cannot “save” an already saved item. In a real-time inference system like Promoted’s, this can be a problem when using stateful events in utility functions (i.e., ranking or quality scores). Once the state is achieved, it immediately becomes a user feature, but in the next inference, the event is undefined and the inference may converge to 0. For example, “goodness” is defined to include “probability to like,” then “goodness” will decrease when a user likes an item! The correction is to use the known state (e.g., “user_liked”) as a conditional with its own weight in utility functions.</p><p><strong>Known State</strong>: Promoted estimates post-purchase value like sales revenue, the probability of continuing subscriptions or consummating a delayed transaction like a vacation rental, and cancelations. However, not all of the “future value state” is uncertain. For example, the purchase price of an item is known. If we fit a model to “post-purchase” value across a mixture of purchases for different items, then the model will likely overfit primarily on features we already know at the inference time, like “price.” To correct this, we decompose post-purchase value predictions to only the “uncertain” part of the value, like the user’s probability of completing a delayed transition or canceling. We then combine the known state (like price) with the inferred state (like the probability of completing the transaction) analytically for optimization and bidding. For example: p(completePurchase)*price.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=769ed0df7980" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/ml-label-definitions-in-search-and-ads-769ed0df7980">ML Label Definitions in Search and Ads</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Podcast: The Tech Unifying Search, Feed, Ads, and Promotions for Marketplaces]]></title>
            <link>https://medium.com/promoted/podcast-the-tech-unifying-search-feed-ads-and-promotions-for-marketplaces-5e5e38e3ee0c?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e5e38e3ee0c</guid>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Mon, 11 Jul 2022 23:57:23 GMT</pubDate>
            <atom:updated>2022-09-27T00:15:28.804Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Fepisode%2F5jXlQvR1RZpHS19kj1thrt%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Fepisode%2F5jXlQvR1RZpHS19kj1thrt&amp;image=https%3A%2F%2Fi.scdn.co%2Fimage%2Fc5ff7c20d79848864dcf740f73ae4992ef0cd8d2&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/239eba43d629cad6e6ec23390cf39dbf/href">https://medium.com/media/239eba43d629cad6e6ec23390cf39dbf/href</a></iframe><p><strong>Neil</strong>: Welcome to the tech talks daily podcast, where you can learn and be inspired by real-world examples of how technology is transforming businesses and reshaping industries in a language everyone can understand. Here is your host Neil C Hughes. Welcome back to the tech talks daily podcast. Now confession time:</p><p><strong>Neil</strong>: Being out in a field in Glastonbury for five days and sitting out in the sun today, I’m recording today’s podcast with a slightly sunburnt face. So who knows what today’s guest thought when he saw me on camera, his name is Andrew Yates and he appeared on my radar. When I originally read about their backstory, which involved Andrew and Dan meeting at Pinterest, where they helped build ad systems at Facebook and Google and realized.</p><p><strong>Neil: </strong>They began to think bigger. So they started <a href="http://promoted.ai/">Promoted.ai</a> to match every buyer with every seller across every app. And they’ve been on an amazing journey, which has taken them through Y Combinator, not to mention being the core function of some of the biggest companies in tech, which are Google, Facebook, Amazon, et cetera, all wash down with a great story. But enough from me, buckle up and hold on tight as I beam your ears all the way to San Francisco, where Andrew Yates is waiting to share his story.</p><p><strong>Neil: </strong>So a massive welcome to the show. Can you tell our listeners a little about who you are and what you do?</p><h4>Introduction to Promoted.ai: Selling Discovery</h4><p><strong>Andrew: </strong>Hi. Neil. I’m Andrew Yates, I’m CEO and founder of Promoted.ai. Promoted.ai, we optimized marketplaces. For example, if you have ever used Airbnb or Amazon, we sort search and feed and promote the best listings at the top to increase revenue.</p><p><strong>Neil: </strong>And all those services you just mentioned, people know all about, but what I try and do on this podcast is let people have a peek behind the curtain and look under the hood.</p><p><strong>Neil: </strong>So today I invited you on the podcast to learn more about engineering and optimizing marketplaces. So just to set the scene, can you tell me more about how data engineering is the core function of some of the biggest companies in tech, such as Google, Facebook, and Amazon? Well, so many others you can name too, but can you expand on that for me?</p><p><strong>Andrew: </strong>Yeah, absolutely. For search or presentation, when you are selling discovery, really all you have is that data you have, the measurement you are selling: did someone engage with it? Did seeing this thing cause somebody to take some action later? etc, and that’s the business that runs Silicon Valley.</p><p><strong>Andrew: </strong>That’s the core business of Google. That’s the core business of Facebook. That’s the core business of Amazon and retailers. You just have a screen and some pixels. Someone did something and you log it, and somehow that transforms that into trillion dollar market.</p><h4>Importance of Measurement in Commercial Media</h4><p><strong>Andrew: </strong>To get more concrete into this, it’s being able to maximize the efficiency of these marketplaces or eCommerce apps or social media. The first step of anything is measuring it correctly. And on one side, that’s a lot of data. That’s everything you looked at, did you engage with it? There’s a lot of streaming data making that available. Lots of infrastructures involved with that, then there’s the uncertainty part of it, and this is where it starts to get interesting where if you’ve ever run an ad campaign, maybe if you had small business or for work, there’s a question of did this ad cause somebody to take an action.</p><p><strong>Andrew: </strong>Now you’ve added uncertainty into your data pipeline, now you have to deal with these things like, what if you don’t have a way to join records, but you have to guess, and that really magnifies the complexity of what these data pipelines have to do, and they still have to be real-time. And if you are in the advertising business, they’re auditable. If you get the Wall Street Journal if you’re in the United States, or I guess anywhere, they love publishing when some major ad network or Facebook or somebody misinterprets or somehow miscomputes a metric. That’s data infrastructure and data engineering, and it’s the order of billions of dollars, and it’s a big deal, so that’s the philosophy of data engineering and processing.</p><p><strong>Andrew: </strong>And this is all separate by the way, from like the data privacy issues. I kind of wanna separate all of that privacy consideration, which is also very important. But I think it sucks up all the oxygen in the room regarding the technical discussion of what the technology is doing, and what the challenges are versus the challenges of just collecting all this data, aggregating, joining it, and dealing with uncertainty.</p><h4>Optimizing What You Correctly Measure</h4><p><strong>Andrew: </strong>And then once you have this measurement, then you can either sell it, in terms of discovery, impressions, clicks, and conversions, and you can also optimize against it. So that’s getting into the other piece of uncertainty, which is uncertainty about the future: did things happen in the past? If you can accurately measure them, it helps you predict what’s likely to happen in the future. And that’s what all of these recommendation engines and search engines are automatically learning about. This was a good result for this query, or this was a good entry to put into a feed for this user that measurement is driving the optimization side. So these are the two things that we focus on, and then the promotion side and the ad tech side, that’s like a layer on top of the maximization, which is what if you wanna sell the ability to be discovered.</p><p><strong>Andrew: </strong>Our philosophy, first you need to have a really good concept of here’s what good matches are and here’s the value of it. And if you want to change that, here’s how much it costs. And as you said a few moments ago, the subjects have ads that do have a bad reputation around privacy, et cetera.</p><h4>“Ads” as Seller Take-Rate Optimization</h4><p><strong>Neil: </strong>It does circle all the energy out in the room. It’s already well documented and talked about just about everywhere, so it is nice to have a different approach today. Before you came on, I was reading your belief that another way: Thinking of ads, as a way to raise prices on sellers on a much fairer basis. Can you tell me more about that?</p><p><strong>Andrew: </strong>Yeah. Think of it as like Amazon, for example, I think this is a little bit different than a brand advertisement where you have a blog or your social media and you are just allowing people to put banner ads.</p><p><strong>Andrew: </strong>This is more of like the abstraction of, for example, Amazon or Doordash where they’re doing paid promotions and it’s things that are already for sale in their marketplace. Therefore, the way to think of this from the seller’s perspective is that it&#39;s math, and it’s money in and it’s money out. Here’s how much I made in revenue, and here’s as much I had to pay in costs.</p><p><strong>Andrew: </strong>Already, the marketplace’s core business is a fraction of sales on Amazon or Airbnb, or any of these e-commerce or marketplaces to take a fraction of the sale, plus like some sort of merchandising fee or et cetera. One way for the marketplace to become more profitable is just to say: before we were charging 20% of the sale, now it’s 25%, or now it’s an extra dollar fee. You just say: hey everyone, you need to pay more. Well, if it was so easy, they would just keep raising the prices forever, why not just make infinite money, right? Why not a hundred percent, right?</p><p><strong>Andrew: </strong>Well because it’s inefficient. Some sellers are willing to pay higher and more fees to the marketplace. Other sellers will pay less. They’re not gonna tell you outright. They’re not gonna say, hey I raise my hand, and I would love to pay more. No, they want to pay the minimum amount to maximize their profits.</p><p><strong>Andrew: </strong>So you can think of ads as a way to optimize for that take rate for sellers. Some sellers are effectively paying, let’s say 30% and other sellers are effectively paying, let’s say 20% and it’s not like you get to choose a select box. Like, I choose 20%, and I choose 30%. It’s more like, okay, you make trade-offs.</p><h4>Raising Prices on Sellers Requires Delivering More Value to Sellers</h4><p><strong>Andrew: </strong>So if you pay a higher fraction to the marketplace, you need to get something back in return. What you get back in return is usually more volume, so you’re trading off. Variability is another one like you’re trading off margins for volume and prominence. And if you think of it that way, then when you see, let’s say Amazon, you see these numbers, I forget the latest number, but billions of dollars are generated by ads on Amazon. And then the rest of the business is just sort of a lost leader. I know, that’s funny to think about.</p><p><strong>Andrew: </strong>I know it’s funny to think about, could they just do ads? Of course not. What’s happening here is that they’ve created their ad system like it’s profit optimization. Lo and behold, if you just attribute profits to the last layer of the profit optimizer, Sure, It looks like it’s all being driven by that, but if you take a step back and use some common sense, it’s like, no, no, no.</p><p><strong>Andrew: </strong>This is the last optimizer for profits and attributing it to this system, so I think once you start zooming out a little bit to let’s say Amazon case or these other big marketplaces where you look at their ads business and it’s driving all of these quote-unquote huge profits, it’s more like that’s the whole business, and that’s just the last layer where they’re optimizing. Here’s how much profit you can pull out in the most efficient way.</p><h4>Silos in Search, Ads, and Merchandising Engineering Teams</h4><p><strong>Neil: </strong>And if we were to take a look under the hood, can you tell me a bit more about how search and discovery teams are built at these large companies that we’re talking about?</p><p><strong>Andrew: </strong>Typically there exists a search team. There’s an ads team and there’s a merchandising team and there’s a data team and maybe a data science team on the side. It’s organically built, and people have their areas. The challenge here though is you only have one marketplace and one e-commerce app, and these teams are competing for the same user’s attention.</p><p><strong>Andrew: </strong>How these teams are built: frequently, you have a search team that’s really good at retrieval, and they’re really good at query relevance. You’ll have an ads team that’s really good at exact measurements, like data streaming, and getting those metrics in place. And then the merchandising team may have a lot of really fantastic, different formats and different ways of like controlling the product.</p><p><strong>Andrew: </strong>But what you really want is all of those things combined into one function, and some companies are really good at this. Like Amazon or Facebook have been, and some other companies are still developing their expertise.</p><p><strong>Neil:</strong> And as someone that’s right in the heart of this space, you’re probably seeing it continuously evolve. And it’s gonna continue to do that too. So I’m curious how do you see the future of commercial search and discovery tech and where it’s heading? Is there anything that stands out to you?</p><h4>Optimizing Phone Screens is Increasingly Critical to Commerce</h4><p><strong>Andrew:</strong> It’s certainly becoming much more important because people are on their phones all of the time. Yeah. As retail becomes more and more abstract and electronic, and it’s just your phone deciding what goes on your phone screen is incredibly important. That’s your portal to people’s attention and their wallets. Ultimately, unlike maybe the past where it was retail, you walk into the store and own that specific location and you come into the store and the experience of it.</p><p><strong>Andrew:</strong> So it’s like taking some of the ideas from old world retail and moving them to phone and some of them translate well and some of them are totally new. I don’t think the core philosophy of measuring people’s attention and doing this economic trade-off between the seller’s interest, the buyer’s interest, and the platform or the marketplace interest. I think philosophically that’ll continue to remain the same. The trend that I see is, especially recently, that people are much more interested in that unified optimization. Like you can’t get away with free money for growth forever. When is it going to be profitable? And that’s when the trend that we’re seeing is people are a lot more serious now about whether or not this is going to be a sustainable business.</p><p><strong>Andrew:</strong> And not only that, but users are buyers, households are much more serious about whether or not it’s something that they really need to buy. For sellers, it’s like, do we have to raise our prices? How and what are we gonna do? Ads are too expensive, and our costs are going up.</p><p><strong>Andrew:</strong> We’re really serious, so I think the gravity of being able to solve these sorts of profitability economic equations has become much more important recently. Whereas I’d say a year ago, it was like a peaked bubble time and money is free, so you can not optimize. It’s pretty easy when you have like these three-sided optimizers, let’s just focus on one of these sides. Growth and Maximum GMV. Let’s say the chickens have come to roost and people are much more serious about profits at scale as opposed to just scale at scale.</p><h4>Ads Engineering at Facebook, Google, and Pinterest</h4><p><strong>Neil:</strong> We will have people listening from a variety of backgrounds, so for those people outside of the space, could you just offer a very brief overview on how ad engineering works behind the scenes, and also for the ad engineers listening, are there any tips or advice that you would offer on how to design great ad marketplaces?</p><p><strong>Andrew:</strong> We have a unique perspective that there’s a whole multi-decade area of ad tech that we really don’t have any experience with or use like real-time bidding or like banner ads, web ads. I know very little about this, frankly, and not that interested. Our perspectives here are what if you work at Facebook, Google, or Pinterest. Dan and I, my co-founder, were EMs together at Pinterest in Pinterest ads. It’s a different philosophy in the sense that you’re thinking in terms of the entire experience, like the entire app versus the real-time bidding and transacting on consumer data.</p><p><strong>Andrew:</strong> It’s much more focused on search and discovery plus, it’s like you’re solving recommenders without the ads. Forget the ads part, it’s just search and discovery for a marketplace. You also have to be accountable for measuring it correctly, communicating those metrics to sellers and it’s other people’s money and you have to communicate your decisions and dollars.</p><h4>“Ads” as Adversarial Search and Discovery with Other People’s Money</h4><p><strong>Andrew:</strong> So from a technology perspective here, the big difference between ads engineering from search and discovery and ordinary search and discovery is that it’s other people’s money and all of your decisions are denominated in dollars and there are trade-offs. It’s not just maximizing, it’s like you have this, and you try to maximize one part, but that minimizes another part, and you have to do this balancing.</p><p><strong>Andrew:</strong> From an engineering perspective, the philosophy of taking the learnings from Facebook ads engineering, Google ads engineering, or Pinterest ads engineering which has much more emphasis on correct measure versus directionally correct. Like A/B experimentation where every data point needs to be correct. Because people measure it, and that’s what you sell.</p><p><strong>Andrew:</strong> On the other hand, it’s not just the presentation of what you show. It’s not just you showing the right thing. It’s how much did it cost the price? How much did showing this do? Was it a profitable decision? Not just a good decision and that these two layers of complexity. It’s our superpower at Promoted for being able to do such a fantastic job of search and discovery because the data bar is higher, so we have better quality data, and we can do better quality optimization. That’s longing the short of it, but it’s also like how you can power performance ads at top companies.</p><h4>Promoted’s Plan to Match Every Buyer and Seller Across Every App</h4><p><strong>Neil: </strong>So I guess the big question is how is Promoted.ai planning to better match every buyer with every seller across all marketplaces on the web or mobile, et cetera. Can you expand on that and how you’re doing that? Rather than say, hey, we have a network, and join our network.</p><p><strong>Andrew: </strong>That’s not what we’ve been doing. In fact, we don’t have an ad network today. What we have are fantastic top marketplaces that we help optimize and measure as infrastructures. Our philosophy is you need to have your own house in order before you can start opening up your house to others. And that starts with measuring everything happening all in your current app, like getting the right data, then maximizing your own conversions and sales.</p><p><strong>Andrew: </strong>Also, doing your own promotions internally and only when you can do it for yourself, then you can open up and start taking your inventory and put it elsewhere, or bring in inventory from elsewhere. Because that way, this is incrementally an improvement on top of the best that you could already be doing.</p><p><strong>Andrew: </strong>So our strategy here has been to work with the absolute best marketplace’s best engineering teams. Get them to the same standardized level of sophistication and quality. Now you have a standardized interface for measurement and optimization that you can start networking together on the backend to do these cross-promotions. That’s our ultimate vision here.</p><h4>Promoted and Y Combinator Winter 2021</h4><p><strong>Neil:</strong> And before you came on the podcast today, I was doing a little bit of research and I noticed that Promoted.ai got into Y Combinator. So I’ve gotta ask: What was that experience like? Were there any big takeaways from that?</p><p><strong>Andrew:</strong> Yeah. I highly recommend Y Combinator. Generally. It’s like, should I go to Harvard? And if you have to ask, then you should probably go for it. If you know better — like you actually wanna go to CMU or Tsinghua or something, then yeah, go to where you want. You don’t need to go to YC.</p><p><strong>Andrew: </strong>But otherwise, it’s a good default place to go if you don’t know any better. A great community and a lot of support. I think their biggest advantage is that they accelerate you past the crappiest and most fragile part of your business, and also help you avoid some of the lamest mistakes that magnify. It’s like kind of your Genesis, these sorts of early mistakes just stick around forever.</p><p><strong>Andrew: </strong>They help you smooth that out. It’s expensive, I do wanna call out that. It’s not free, although they upped their other offer recently, which is awesome.</p><h4>How Do I Get into Y Combinator?</h4><p><strong>Andrew: </strong>Hey, Should I join Y Combinator? I’m sure that’s a pretty popular question. Yes. And: “How do I get into Y Combinator?” is usually the follow-up question. “What’s your advice?” Say fewer words… and have a convincing story about why you specifically would be successful. If you don’t have that answer, you should make one. If you don’t know the answer, then they’re not gonna know, and that’s not good. Right? Have an answer ready.</p><h4>Y Combinator During COVID</h4><p><strong>Andrew: </strong>And then from my experience personally, we did it in peak COVID, so we were winter 2021. No one really enjoyed COVID, I don’t think, and we didn’t enjoy it. The experience was different. It was extremely efficient, but it wasn’t fun. What you did is that you just sat in a chair and faced your computer for a year and you didn’t move and you just worked.</p><p><strong>Andrew: </strong>No parties, no networking. If you need to meet someone for raising money, you send an email and they do a zoom call and they’re like: How much do you want? You say, I want X and I’m like, okay or no, and no coffee chat. There are advantages to this, and there are disadvantages to this. My experience with Y Combinator won’t be replicated and definitely hope it won’t ever be because that was the peak lockdown period, and it was a weird time.</p><p><strong>Andrew: </strong>There were special advantages and disadvantages to that. That being said, Y Combinator is now opened up and it’s in person now, and you can meet people and they’re going back to some of their classes and saying that they can meet you in person now, which is great.</p><p><strong>Andrew: </strong>By the way, by the time this goes out, we’ll have already announced. I was just talking to TechCrunch right before this, and we’re gonna announce our continuation of funding from Y Combinator. Y Combinator invested another several million dollars into our company.</p><p><strong>Andrew: </strong>So that’s another thing about Y Combinator is that it’s not just a program and that’s the end, Y Combinator logo forever. No. It’s forever. It’s a part of your life. You’re part of the community, and that means part of the community. It doesn’t end at just their seed stage or demo day. It’s a continuation forever.</p><h4>What’s Next for Promoted: Keep Doubling Down on Top Customers and Get More Like Them</h4><p><strong>Neil: </strong>Wow. That’s a fantastic story. And you’ve made it through, and you enjoy the benefits now and with a post-pandemic world. So I’ve gotta ask what’s next for Promoted.ai. Where’d you go from here?</p><p><strong>Andrew: </strong>Yeah, the biggest is to just keep doubling down on our current top customers and getting more of them. There was an interview the other day. They asked when did you know the product worked? And I’m like yesterday, we’re at double-digit improvements for Outschool and Hipcamp.</p><p><strong>Andrew: </strong>We’ve got some fantastic other partners that are some public and some not yet public, but we’re never satisfied. We want more, and our biggest challenge and something that makes us unique in comparison to lots of like ML in a box and easy mode to get started, like lots of ads mode get started. There’s Algolia out there for Search.</p><h4>The Race of Selling Optimization to Top Engineering Companies</h4><p><strong>Andrew: </strong>We really focus on top marketplaces that are already scaled or have engineering teams, so that means the bar is really high and if you are kind of like, oh yeah, we got double digits, you know, we’re great, and be happy forever. No, no, no, no, no. That was good yesterday, but today, we need another percent improvement. And the day after that we need another percent improvement and the nature of this business with the data infrastructure and machine learning is multiplicative.</p><p><strong>Andrew: </strong>Right? I’m adding this extra component, but it interacts with every other component as well. So voila, you’ve just created much more complexity for smaller and smaller gains. So that’s the race that we have is taking what we currently have and continuing to invest in making it even better.</p><p><strong>Andrew: </strong>But then that also increases your cost a lot, so from our side, it’s like this forever goal of adding more data, which increases our costs, which then we have to increase our infrastructure, but then increases latency. So we have to reduce the latency. And then that’s the main thing for us: is to just keep that loop going to make the absolute best optimizer, and then once you have a really fantastic optimizer and the data infrastructure to support it. Then there are lots of great things you can do, like ads or discount promotions. Like how do you optimize, how do I know if I was in slot 10? And if I was in slot one, that’s gonna increase my sales, and by how much and how much is that worth?</p><p><strong>Andrew: </strong>Well, you just think about all the pieces into that, right? You need to know what the value is. If you didn’t do it, you need to measure the value for all of it. And then you need to predict all of the conversion optimizations, and then you need to measure the user experience and put at the top and maybe degrade the user experience of what that’s worth and does the price you’re willing to pay to exceed that.</p><p><strong>Andrew: </strong>Like just keep hitting that loop, so keep investing in our core product for the top customers, and then bringing on even more similar top customers, and then eventually it’s that cross promotions vision of us actually powering all of the greatest marketplaces in the world. That’s a pretty interesting book of inventory if you’re an ads business, isn’t it?</p><h4>Talking about Favorite Music and Outro</h4><p><strong>Neil: </strong>It really is, and it’s a story I’ll be following very closely and it’d be great to stay in touch with you and get you back on maybe next year and find out how that journey’s going. But before I do let you go, I’m gonna have a bit of fun with you. Now. I always like to ask my guests to leave a personal note of inspiration with everyone listening. Now that can either be a song choice that we can add to our Spotify playlist, a song that inspires you or helps you get your head in the zone, or a book that can add to our Amazon wish list. What are you gonna leave everyone listening with today?</p><p><strong>Andrew: </strong>You know what I think, I think I looked at your Spotify list and I’m gonna be that guy and kind of blow up your Spotify list, but something doesn’t quite fit.</p><p><strong>Neil: </strong>I had a guy a few weeks ago, put on pirate metal, so you’re gonna have to go there, man.</p><p><strong>Andrew:</strong> Pirate metal. you have to, there you go. Yeah. I really like Cowgirl by the Underworld.</p><p><strong>Neil: </strong>Oh, that’s a good tune.</p><p><strong>Andrew: </strong>Yeah. Like that early nineties. Yeah, UK, electronica scene. There’s just something about that era. It’s like, wow, Computers! They’re amazing. And you have like these memes of the kid with spiky hair on the computer and big old sea monitor and like the nineties hacker I’m in.</p><p><strong>Andrew: </strong>but it was just like, wow, the internet and you can be in your house anywhere in the world and be nobody just like for me, frankly, I’m just a kid from the Midwest, you know? but you could be somebody if you are really good at this and that song is just that reminder of wow, this magical world.</p><p><strong>Andrew: </strong>And it turns out that actually making things work is a lot of work and it’s a job, but that song kind of hearkens back to a more innocent time of when you sit down at the computer to write code and that’s a cool thing.</p><p><strong>Andrew: </strong>You find your sunglasses and your trench code or something. I’m like, yeah, they’re gonna go out to the rave next door, the matrix sort of thing. The funny thing is that I live in San Francisco and if you want to do that, you can, but you don’t. It’s ridiculous. You mean, first of all, I need to see my computer screen and sunglasses aren’t gonna work and like going to the rave, man, I’m getting old. Right?</p><p><strong>Andrew: </strong>You did it a couple of times and then it’s like that’s what it is, and then I’m gonna stay at home with my wife, and Netflix and chill is kind of what I’m really passionate about these days. Anyways, cowgirl by the underworld, reliving the 16 to 25-year-old you about the dreams of being in technology.</p><p><strong>Neil:</strong> I absolutely love that. I’ll be listening to that as soon as we finish this call. You’re never too old to rave definitely. But before you go for anyone listening, just want to find out more information about promoted ai, dig a little bit deeper on any of the topics we talked about, reach your team or contact you even. What’s the best starting point for everything.</p><p><strong>Andrew:</strong> Our website: promoted.ai.</p><p><strong>Neil:</strong> Awesome. Well, I had those links to the show notes so people can find you. Love sharing with you. Great story! And I love how you’ve simplified everything. Put it in a language that everyone can understand, which is something I try and do on here. But I meant what said a few moments ago, we’ll stay in touch and it’d be great to see how your journey continues to evolve and maybe get you on later in the air or next year. Thanks for sharing that today.</p><p><strong>Andrew: </strong>Well, thank you, Neil. It was a pleasure.</p><p><strong>Neil: </strong>I know we’ve only scratched the surface of this topic, but I really enjoy today’s conversation.</p><p><strong>Neil: </strong>And if there is anybody else listening in the industry, you’ve got different insights or any story that you’d like to share with me, or even if you’ve just got a few questions, remember to email me at techblogwriter.co, Twitter, Instagram, LinkedIn, just @ me or C Hugh, send me a DM. Don’t just hit the follow button.</p><p><strong>Neil: </strong>Let me know if you’ve listened to the show. What you like, what you don’t like, whatever it might be. And now we’ll keep this conversation going. But other than that, it’s time for me to get outta here and I’ll return again tomorrow with another great guest. So thank you for listening as always. And until next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e5e38e3ee0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/podcast-the-tech-unifying-search-feed-ads-and-promotions-for-marketplaces-5e5e38e3ee0c">Podcast: The Tech Unifying Search, Feed, Ads, and Promotions for Marketplaces</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Promoted.ai on TechCrunch! $6M Seed]]></title>
            <link>https://medium.com/promoted/promoted-ai-on-techcrunch-6m-seed-875897e9317e?source=rss-48973c0b7bfa------2</link>
            <guid isPermaLink="false">https://medium.com/p/875897e9317e</guid>
            <dc:creator><![CDATA[Andrew Yates]]></dc:creator>
            <pubDate>Mon, 11 Jul 2022 23:51:24 GMT</pubDate>
            <atom:updated>2022-07-11T23:51:24.250Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://techcrunch.com/2022/07/01/promoted-grabs-more-funding-to-help-e-commerce-marketplaces-achieve-profitability/">Promoted grabs more funding to help e-commerce marketplaces achieve profitability</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=875897e9317e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/promoted/promoted-ai-on-techcrunch-6m-seed-875897e9317e">Promoted.ai on TechCrunch! $6M Seed</a> was originally published in <a href="https://medium.com/promoted">@ Promoted</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>