<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Understanding Recommenders - Medium]]></title>
        <description><![CDATA[Understanding Recommenders is a research-driven effort to demystify recommender systems and their impact on society. A project of the Center for Human-Compatible AI at the University of California, Berkeley. - Medium]]></description>
        <link>https://medium.com/understanding-recommenders?source=rss----47b079b44caf---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Understanding Recommenders - Medium</title>
            <link>https://medium.com/understanding-recommenders?source=rss----47b079b44caf---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:50:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/understanding-recommenders" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Experiments are the Best Kind of Transparency]]></title>
            <link>https://medium.com/understanding-recommenders/experiments-are-the-best-kind-of-transparency-ea35c1661479?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/ea35c1661479</guid>
            <category><![CDATA[recommender-systems]]></category>
            <dc:creator><![CDATA[Luke Thorburn]]></dc:creator>
            <pubDate>Sun, 09 Jun 2024 00:51:57 GMT</pubDate>
            <atom:updated>2024-06-09T00:51:57.648Z</atom:updated>
            <content:encoded><![CDATA[<p>— <a href="https://lukethorburn.com/">Luke Thorburn</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SvYITzqww0ELV90gE05_2A.png" /></figure><p>Many of the big questions regulators of online platforms care about are about <em>causality</em>. Do current recommender systems harm mental health? Do they make politics more divisive? Do they undermine journalism? If so, by how much? We can get <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">some insight</a> into these questions from outside platforms, but the most reliable answers come from experiments conducted on-platform with real content, real interfaces, real users and real patterns of use.</p><p>For now, only platform employees (and their occasional academic collaborators) have the ability to conduct such experiments, and the results of experiments that <em>are</em> conducted — <a href="https://partner.booking.com/en-us/click-magazine/industry-perspectives/role-experimentation-bookingcom">hundreds every week</a>, for large platforms — are rarely made public. This lack of access to strong causal evidence about the impacts of platform design decisions makes it difficult for academics to conduct public interest research, and difficult for policymakers to design effective, evidence-based regulation.</p><p>In this piece we argue that for those who play a role in governing recommenders, the most useful kind of transparency is experiments, because they provide the strongest evidence for (1) whether intervention is warranted and (2) which interventions will be effective. They also (3) level-up policy debates and (4) facilitate the <a href="https://www.platformaccountability.com/">regulation of outcomes</a> in a technology-agnostic way. We review how non-experimental forms of transparency fall short, assess the limitations of current proposals (both under Europe’s Digital Services Act, which is law, and the Platform Accountability and Transparency Act that is stalled in the US Senate) with respect to experimental access, and sketch what a regulatory framework for allowing experimentation could look like.</p><h3>Why experiments?</h3><h4>Causal evidence is important</h4><p>Causal evidence is important for regulators, first, because it can tell us <em>whether</em> intervention is justified. There are widespread concerns that recommenders are implicated in political polarization, mental health, declining local news, and many other problems. But if recommenders are not significantly contributing to such problems — and if building recommenders differently would not help — why intervene? Attempts to regulate in the absence of causal evidence risk placing unjustifiable state-enforced constraints on speech and other human activities, and spending valuable resources (including the time and attention of regulators) in ways that ultimately don’t improve the outcomes we care about.</p><p>Second, causal evidence is important because it can tell us <em>which</em> specific alternatives — which alternative algorithms, which evaluation metrics, which changes to the user interface — would be better. As a regulatory domain, online platforms are particularly amenable to evidence collection. All aspects of their design can be experimentally manipulated, and most established platforms will already have infrastructure and processes in place to perform such A/B tests. In principle, this makes it possible to test the impact of interventions and write evidence-based <a href="https://www.prosocialdesign.org/">design</a> <a href="https://uscneelycenter.substack.com/p/introducing-the-neely-center-design">standards</a> for social platforms in a way that is not possible in other domains, and can help avoid encouraging alternatives that don’t actually address the problem.</p><p>Third, causal evidence enables better policy discourse. For example, consider the ongoing and deeply felt <a href="https://www.vox.com/24127431/smartphones-young-kids-children-parenting-social-media-teen-mental-health">debates</a> over how social media impacts the mental health of young people. There is <a href="https://www.afterbabel.com/p/social-media-mental-illness-epidemic">some causal evidence</a> available, but it is often not as definitive or granular as could be provided by on-platform experiments (see, e.g., Instagram’s <a href="https://www.wsj.com/articles/facebook-knows-instagram-is-toxic-for-teen-girls-company-documents-show-11631620739">Project Daisy</a>), and this leaves space for interpretation. Some are concerned that we fail young people by unnecessarily waiting for stronger evidence, and others are <a href="https://zephoria.medium.com/struggling-with-a-moral-panic-once-again-f126528c7715">equally</a> <a href="https://www.nature.com/articles/d41586-024-00902-2">concerned</a> that we fail young people by mistakenly or simplistically concluding that social media is the primary driver of declines in mental health. Access to stronger causal evidence could help resolve contentious claims, and shift the mainstream discussion from “whether or not to blame social media” to the <a href="https://www.nature.com/articles/s44159-024-00307-y">details</a> of the causal dynamics at play.</p><p>Fourth, causal evidence is important because it would allow regulators to address the <em>impacts</em> of design choices, and not the design choices themselves. While technology-specific design standards provide helpful guidance, it can be difficult to write laws that dictate what people can build in a way that (a) adapts to technological innovation and (b) avoids banning harmless uses of the same technology. For these reasons, it might be better to <a href="https://www.platformaccountability.com/">regulate outcomes</a>. But to do this, regulators need regular, reliable access to evidence about the causal impacts of platform design choices on a standardized set of outcomes.</p><h4>To answer causal questions, you need experimental access</h4><p>There <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">are ways</a> to get some causal evidence without running on-platform experiments, but they all have significant limitations. Approaches that academics and civil society researchers commonly use are:</p><ul><li><strong>Mock platforms</strong> — Sometimes researchers build their own model <a href="https://osf.io/preprints/osf/tygec">social</a> <a href="https://www.nature.com/articles/s41562-023-01655-0">media</a> <a href="https://www.yourfeed.social/about">apps</a> and pay people to use them. This allows the researcher to get evidence about how design choices on this mock platform impact users. But there will always be question marks over the extent to which user behavior on the mock platform reflects their behavior on real apps, especially because niche platforms don’t experience the <a href="https://medium.com/analytics-vidhya/network-effects-a95c42c10ff2">network effects</a> of major players.</li><li><strong>Browser extensions</strong> — Sometimes researchers pay people to <a href="https://humancompatible.ai/news/2024/01/18/the-prosocial-ranking-challenge-60000-in-prizes-for-better-social-media-algorithms/">install</a> <a href="https://www.recalign.com/">a</a> <a href="https://dl.acm.org/doi/10.1145/3555643">browser</a> <a href="https://www.tandfonline.com/doi/full/10.1080/08838151.2020.1757365">extension</a> that changes how a platform works and collects data on how the user responds to this change. But this method can’t be used for mobile apps, which by some estimates account for <a href="https://www.statista.com/statistics/477368/us-social-media-visits-share/">more than 80%</a> of social media use, and the possible interventions are limited by the data available, the defensive measures many platforms have implemented to make scraping difficult, and the need to implement changes without substantial delays during page load.</li><li><strong>User controls</strong> — Sometimes platforms give users control over their recommender. For example, X (formerly Twitter) allows users to switch between an engagement-based feed and chronological feed, and this makes it possible to <a href="https://arxiv.org/abs/2305.16941">compare the content</a> that is surfaced by each algorithm. But this approach can only be used to study alternatives that are already made available by a platform.</li><li><strong>Observational data</strong> — In some situations it is possible to extract causal evidence from observational data, using statistical methods that exploit <a href="https://mixtape.scunning.com/09-difference_in_differences">natural</a> <a href="https://mixtape.scunning.com/07-instrumental_variables">experiments</a> or carefully <a href="https://mixtape.scunning.com/10-synthetic_control">simulate a control group</a> where one didn’t exist. These methods <a href="https://dx.doi.org/10.2139/ssrn.3919760">have</a> <a href="https://www.dropbox.com/s/5v9c6ajvmu2of8d/Guo_social_media.pdf?dl=0">been</a> <a href="https://arxiv.org/pdf/2404.02803.pdf">used</a> in the social media context, but are limited to questions for which the data has a particular structure, and the answers they provide are never quite as convincing as conducting an actual experiment. For example, the ways that these methods try to account for confounding variables are proxies for the true randomization that would be used in an experiment.</li></ul><p>Access to data alone is not very helpful for understanding causality. To get reliable information about what causes what, researchers really need access to run on-platform experiments.</p><h4>We don’t yet have experimental access</h4><p>Currently, there are no reliable ways for academics, regulators, or civil society researchers to run experiments on platforms.</p><p>There <a href="https://www.science.org/content/article/does-social-media-polarize-voters-unprecedented-experiments-facebook-users-reveal">are</a> <a href="https://www.nature.com/articles/s41586-023-06297-w">some</a> <a href="https://arxiv.org/abs/2008.12623">examples</a> of platforms voluntarily collaborating with academics to perform on-platform experiments. Such collaborations are valuable, but infrequent. They are built on serendipitous relationships between academics and platform employees, often <a href="https://www.wired.com/story/platforms-engagement-research-meta/">fall</a> <a href="https://www.wired.com/story/meta-social-media-polarization/">through</a> because they are not a priority for platforms, and fundamentally leave the power with platforms to decide what experiments they will let researchers run. This has been called “<a href="https://www.science.org/doi/10.1126/science.adi2430">independence by permission</a>,“ which is not true independence.</p><p>Are there legal avenues to require platforms to provide experimental access? In the EU, the <a href="https://eur-lex.europa.eu/eli/reg/2022/2065/oj">Digital Services Act</a> (DSA) has two related provisions. <a href="https://eur-lex.europa.eu/eli/reg/2022/2065/oj#art_37">Article 37</a> requires platforms to subject themselves to annual independent audits. These audits, according to <a href="https://digital-strategy.ec.europa.eu/en/library/delegated-regulation-independent-audits-under-digital-services-act">additional guidance</a> provided by the EU, may require the auditing organization to perform “tests,” the stated definition for which includes “experiments.” While the scope of these experiments is limited to assessing compliance with the DSA, the definition of “systemic risks” (for which the DSA requires risk assessment and mitigation) is <a href="https://eur-lex.europa.eu/eli/reg/2022/2065/oj#art_34">very broad</a>, and arguably includes most of the experiments that would be of interest. That said, the extent to which the word “experiments” in the <a href="https://digital-strategy.ec.europa.eu/en/library/delegated-regulation-independent-audits-under-digital-services-act">delegated act</a> is interpreted to include actual randomized controlled trials, and the extent to which auditors make use of them, remains to be seen. (We encourage those involved in the implementation of the DSA to push for experimental access via this route.)</p><p><a href="https://eur-lex.europa.eu/eli/reg/2022/2065/oj#art_40">Article 40</a> of the DSA requires platforms to make data available for third-party researchers, and to some extent it remains to be seen exactly what forms of data access this will enable. But it seems unlikely that Article 40 would allow researchers to request data that a platform would need to conduct an experiment in order to generate.</p><p>In the US, the proposed <a href="https://www.congress.gov/bill/117th-congress/senate-bill/5339">Platform Accountability and Transparency Act</a> (PATA) would, if enacted, require platforms to cooperate with any “qualified research project” that has been approved by the National Science Foundation via a newly-established review process. However, there is no mention of experimentation in the draft text, and passing the bill does not appear to be a priority for Congress in the near term.</p><h4>What about past experiments?</h4><p>While there isn’t yet a reliable route to <em>conduct</em> on-platform experiments, the data access provisions under Article 40 of the DSA probably do allow for compelled release of the results from experiments that a platform has <a href="https://fbarchive.org/doc/odoc6094798026">already run</a>. Release of such experiment results is also required by the <a href="https://www.revisor.mn.gov/bills/text.php?number=SF4097&amp;version=latest&amp;session=ls93&amp;session_year=2024&amp;session_number=0">Prohibiting Social Media Manipulation Act</a> which recently passed in Minnesota, and takes effect on July 1, 2025.</p><p>This is potentially a rich source of insight. Large platforms run <a href="https://partner.booking.com/en-us/click-magazine/industry-perspectives/role-experimentation-bookingcom">hundreds</a> of experiments every week, and seeing the results of these could <a href="https://www.lawfaremedia.org/article/how-tech-regulation-can-leverage-product-experimentation-results">significantly</a> <a href="https://future.com/forget-open-source-algorithms-focus-on-experiments-instead/">contribute</a> to policy debates. For example, the documents leaked by Facebook whistleblower Frances Haugen mention experiments testing the impact of <a href="https://www.documentcloud.org/documents/21748432-tier2_news_ir_0218">engagement-based ranking</a> relative to a chronological feed, and the impact of removing most <a href="https://www.documentcloud.org/documents/23605723-tier0_civ_ir_0419">platform integrity measures</a>. There are very few public experiments of comparable scale or validity to inform debates over recommender design.</p><p>Whether the Minnesota law survives legal challenges, and the extent to which Article 40 will be used in practice to access such results, however, remains to be seen. It’s difficult for researchers to know what data to ask for without knowing which experiments have been run. Further, there is a problem of perverse incentives: without additional safeguards, compelling the disclosure of experimental results might discourage them from evaluating their impact on society out of fear that the results of such experiments would become public. For these reasons, at least some power to initiate experiments should lie outside platforms.</p><h3>Implementation</h3><p>So how could regulation facilitate third-party experiments?</p><h4>Sketch of a law</h4><p>A new law could look similar to <a href="https://www.congress.gov/bill/117th-congress/senate-bill/5339">PATA</a> — which would establish a formal approval process for qualified researchers to access platform data — but with an explicit focus on experimental access. It would need to answer a number of key questions, including:</p><ul><li><strong>Who decides which experiments are run?</strong> — This could be an existing scientific body (e.g., the NSF). Proposals for experiments would also need to undergo ordinary academic <a href="https://scholarship.law.cornell.edu/facpub/1487/">ethics review processes</a>, which usually require participants to explicitly consent to participate in research. This differs from the way platforms normally conduct in-house experiments, where they rely on the blanket permissions granted in platform terms of service.</li><li><strong>Who pays?</strong> — This could be existing academic funders, the platforms, or some combination thereof.</li><li><strong>Who does the work of actually conducting the experiment?</strong> — This is likely to be some combination of external researchers (who are framing the research) and platform employees (who can directly work with the internal systems of a platform).</li></ul><h4>Challenges</h4><p>Any legal requirement for experimental access would need to responsibly navigate concerns around privacy and research ethics, but these are solvable problems. There are standard ethics review processes in place at most academic institutions, and there are many concrete ways to mitigate privacy risks, both technical (e.g., <a href="https://en.wikipedia.org/wiki/Differential_privacy">differential privacy</a>) and procedural (e.g., <a href="https://techpolicy.press/remote-query-execution-a-powerful-way-to-do-privacy-protecting-research-on-platform-data/">remote query execution</a>).</p><p>A more fundamental challenge relates to free speech. In the US, compelling a platform to change the order in which it ranks content for some users would potentially violate the First Amendment, even if such a change was in the service of a temporary, public-interest experiment that was designed by academics without government influence. Platforms have <a href="https://www.theguardian.com/us-news/2023/sep/08/elon-musk-twitter-lawsuit-california-free-speech">already</a> <a href="https://www.techpolicy.press/evaluating-the-argument-over-the-california-age-appropriate-design-code-act/">challenged</a> less-interventional transparency measures on constitutional grounds, with <a href="https://www.reuters.com/legal/judge-blocks-california-law-meant-protect-childrens-online-safety-2023-09-18/">some success</a>. The US legal system <em>can</em> <a href="https://lawreview.sf.ucdavis.edu/sites/g/files/dgvnsk15026/files/media/documents/49-4_Balkin.pdf">enforce private contracts</a>, so if a platform is challenged in court for causing an outcome they previously told users they wouldn’t cause, it might be in their self interest to produce experimental evidence on the matter. But in general, direct legal mandates for experimental access might only be possible in other jurisdictions.</p><h3>Conclusion</h3><p>There are many different <a href="https://medium.com/understanding-recommenders/a-menu-of-recommender-transparency-options-7b1379f12dda">kinds of transparency</a> for recommenders, which all serve different needs, so when calling for transparency it’s important to <a href="https://arxiv.org/abs/1806.07552">be specific</a> about the intended audience and how the type of transparency requested will be helpful for them. For regulators and all those who play a role in governing recommenders, the most useful kind of transparency is the ability to run experiments on platforms, because experiments provide strong causal evidence to clarify whether intervention is justified, and if so, which specific alternative algorithmic design choices would be better. Here, we have focused on the context of recommender systems, but the same basic point is also true of generative AI.</p><p><em>Thank you to Nathaniel Persily for helpful guidance on the scope of PATA and the DSA. Any errors remain those of the authors.</em></p><p><em>Luke Thorburn was supported in part by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (safeandtrustedai.org), King’s College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea35c1661479" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/experiments-are-the-best-kind-of-transparency-ea35c1661479">Experiments are the Best Kind of Transparency</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Media Diversity and Do Recommender Systems Have It?]]></title>
            <link>https://medium.com/understanding-recommenders/what-is-media-diversity-and-do-recommender-systems-have-it-b2c86be9f08f?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/b2c86be9f08f</guid>
            <dc:creator><![CDATA[Priyanjana Bengani]]></dc:creator>
            <pubDate>Mon, 13 Nov 2023 15:28:15 GMT</pubDate>
            <atom:updated>2023-11-17T11:32:12.352Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFyKzJEYbsZlOsJmzrOjVw.png" /></figure><p>— <a href="https://twitter.com/acookiecrumbles"><em>Priyanjana Bengani</em></a><em>, </em><a href="https://twitter.com/jonathanstray"><em>Jonathan Stray</em></a><em>, </em><a href="https://lukethorburn.com/"><em>Luke Thorburn</em></a></p><p>One of the classic criticisms of recommender systems arises when they show you a narrow range of content, be it popular items or items similar to whatever you’ve already clicked on. This can be a failing of more elementary approaches to recommendation systems, but production recommender systems typically include diversification algorithms for myriad reasons. We’ve <a href="https://medium.com/understanding-recommenders/when-you-hear-filter-bubble-echo-chamber-or-rabbit-hole-think-feedback-loop-7d1c8733d5c">previously discussed</a> the mechanics of filter bubbles, echo chambers, and other types of information-limiting environments. Here we examine the many meanings of “diversity” in media, why society in general and platforms in particular might want it, whether existing platforms have it (and what that even means), and how recommender systems can help achieve it.</p><p>There are many different definitions of diversity used by communications scholars, as well as a variety of metrics and diversification algorithms used in recommenders that cover much of the same ground from different perspectives. We might want each user to see items from a variety of sources, across diverse viewpoints, on multiple topics, or including multiple media formats. We could also think of diversity in terms of novelty and serendipity, thereby ensuring users don’t see posts covering the same topics every time they log in, which can lead to boredom and engagement dropping off in the longer term. These conceptions of diversity are all from the perspective of the consumer — that is, which items each user sees. We can also look at diversity from the view of the producer — that is, what users each producer reaches. Measures such as coverage or popularity inequality are used to ensure that the system displays content from more than a few “superstars,” such that it is viable for a wider range of content producers.</p><h3>Why society wants diversity in media</h3><p>Media diversity has been a key concept in social science and media policy since at least the <a href="https://en.wikipedia.org/wiki/Media_cross-ownership_in_the_United_States#Cross-ownership_rules_of_1975">mid 20th century</a>, with no single agreed-upon definition but many different ideas. A <a href="https://doi.org/10.1080/21670811.2020.1764374">recent systematic review</a> notes that “research on this topic has been held back by the lack of conceptual clarity about media diversity and by a slow adoption of methods to measure and analyze it.”</p><p>There are a myriad reasons to care about media diversity: informed citizens, inclusive public discourse, or preventing large companies from dominating the media landscape. Different conceptions of diversity have focused on item <em>content</em> (e.g. what each article contains), the <em>structure </em>of the media ecosystem (e.g. the number of different publishers and who owns them or the demographics of journalists), or individual <em>exposure </em>(what each person sees, as determined both by self selection and recommender systems). Recent works give several different conceptions and reasons for diversity in media, including:</p><ul><li>“[the] heterogeneity of media content in terms of one or more specified characteristics.” (<a href="https://arxiv.org/ftp/arxiv/papers/2005/2005.09495.pdf">Bernstein et al, 2019</a>)</li><li>“Diversity refers to the idea that in a democratic society informed citizens collect information about the world from a diverse mix of sources with different viewpoints so that they can make balanced and well-considered decisions.” (<a href="https://www.tandfonline.com/doi/epdf/10.1080/1369118X.2016.1271900?needAccess=true&amp;role=button">Helberger et al, 2016</a>)</li><li>“support and seek to give voice to a plurality of competing views — from those with different backgrounds, histories and stories. Help build a more inclusive, less fragmented society” (<a href="https://www.adalovelaceinstitute.org/wp-content/uploads/2022/11/Ada-Lovelace-Institute-Inform-Educate-Entertain...and-Recommend-Nov-2022.pdf">European Broadcasting Union</a>)</li><li>“news consumers emphasize that they find it important to be able to hear and read about topics they haven’t thought of, viewpoints they don’t quite understand, and perspectives that are unknown.” (<a href="https://royalsocietypublishing.org/doi/10.1098/rsta.2018.0088">Harambam, 2018</a>)</li></ul><p>Ultimately, from a societal standpoint, the type of media diversity you care about often depends on <a href="https://doi.org/10.1080/21670811.2019.1623700">how you think democracy should work</a>.</p><h3>Why platforms want media diversity</h3><p>Platforms, on the other hand, have other motivations for incorporating media diversity. When users consume a wider array of items, it <a href="https://www.cs.toronto.edu/~ashton/pubs/alg-effects-spotify-www2020.pdf">can improve</a> long-term metrics including user conversion and retention. Platforms also need to ensure that the system doesn’t simply recommend the most popular items or it won’t be attractive to the smaller or newer content creators. TikTok’s algorithm, for example, <a href="https://knightcolumbia.org/blog/tiktoks-secret-sauce">prioritizes content</a> over the number of followers an account has, thereby allowing even small creators to gain an audience.</p><p>Most real-world recommenders run diversification algorithms prior to presenting users with a list of items. The rationale for diversification given by the different platform operators differs based on priorities and values. Examples from large platforms include:</p><ul><li><strong>avoiding monotony</strong> — “don’t see multiple posts from the same person or the same seed account” (<a href="https://ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommender-system/">Instagram</a>)</li><li><strong>encouraging novelty</strong> — “help users discover new content or inculcate new tastes” (<a href="https://research.atspotify.com/2021/03/shifting-consumption-towards-diverse-content-via-reinforcement-learning/">Spotify</a>)</li><li><strong>preventing rabbit holes</strong> — “avoid recommending a series of similar content — such as around extreme dieting or fitness, sadness, or breakups — to protect against viewing too much of a content category that may be fine as a single video but problematic if viewed in clusters” (<a href="https://newsroom.tiktok.com/en-us/an-update-on-our-work-to-safeguard-and-diversify-recommendations">TikTok</a>)</li><li><strong>accounting for evolving or unexpressed preferences</strong> — “intersperse recommendations that might fall outside people’s expressed preferences, offering an opportunity to discover new categories of content” (<a href="https://newsroom.tiktok.com/en-us/an-update-on-our-work-to-safeguard-and-diversify-recommendations">TikTok</a>)</li><li><strong>minimizing popularity bias</strong> — “spread consumption across artists and facilitate consumption of less popular content” (<a href="https://research.atspotify.com/2021/03/shifting-consumption-towards-diverse-content-via-reinforcement-learning/">Spotify</a>)</li></ul><p>These different goals mean that there is no one ideal way to incorporate diversity in recommender systems. Instead, approaches may differ widely depending on a platform’s priorities and use cases.</p><h3>Are platforms diverse today?</h3><p>For those outside of a platform, measuring media diversity can be challenging because good data on who sees what is often inaccessible, and the core algorithms and policies are ever-changing. Nonetheless, there have been a variety of <a href="https://arxiv.org/pdf/2102.04256.pdf">audits</a>, <a href="https://ora.ox.ac.uk/objects/uuid:d6b9a202-9a1c-41f7-881e-08256178aeb7/download_file?safe_filename=fletcher%2Bcornia%2Bnielsen%2B-%2Bhow%2Bpolarized%2Bare%2Bonline%2Band%2Boffline%2Bnews%2Baudiences.pdf&amp;file_format=pdf&amp;type_of_work=Journal+article">surveys</a>, <a href="https://www.science.org/doi/10.1126/science.ade7138">collaborations</a> with the platform companies along with <a href="https://research.atspotify.com/2020/12/algorithmic-effects-on-the-diversity-of-consumption-on-spotify/">corporate research</a> which provide a window into what’s actually happening, and can help confirm or dispel widely-believed myths based on anecdotal evidence. In this section, we’ll try to summarize what is publicly known.</p><ul><li>Many have voiced concerns about algorithmic systems creating <a href="https://www.google.co.uk/books/edition/Echo_Chambers/sEgHAAAACAAJ">echo chambers</a> and <a href="https://www.google.co.uk/books/edition/_/zhRGngEACAAJ">filter bubbles</a> of like-minded content that could potentially result in more polarization across the board. Yet survey research shows that there is more <a href="https://ora.ox.ac.uk/objects/uuid:728f1036-6ed3-4b85-b897-8763073a6d74/download_file?safe_filename=Incidental%2Bexposure%2BAAM.pdf&amp;file_format=application%2Fpdf&amp;type_of_work=Journal+article"><em>incidental</em> <em>exposure</em></a><em> </em>to different sources for those who consume news online, while those who only read news offline tend to engage in a more self-selected media diet associated with their own partisan leanings (<em>selective exposure</em>)<em>.</em></li><li>Two different studies into <a href="https://www.researchgate.net/publication/318256136_Burst_of_the_Filter_Bubble_Effects_of_personalization_on_the_diversity_of_Google_News">Google News</a> (Germany) and <a href="https://www.nature.com/articles/s41562-020-00954-0">Google Search</a> (US) show the concentration of news sources. In the case of Google News, over a six-day period, researchers found that 86% of all articles were from just eleven outlets, with Focus Online and Die Welt making up 37% of these recommendations. These eleven outlets weren’t necessarily the ones that had the most reach according to data compiled by the <a href="http://ausweisung.ivw-online.de/">German Audit Bureau of Circulation</a> (IVW). Meanwhile, on Google Search, an audit of location-specific news results in 3,000 US counties found three outlets making up just over 15% of all results, with the platform favoring national outlets over local news outlets, indicating the presence of superstar economics.</li><li>Similarly, an <a href="https://arxiv.org/pdf/1908.00456.pdf">audit of Apple News</a> over a two-month period found that over 45% of stories shown in the platform’s algorithmic-driven “trending stories” section came from just three sources (CNN, Fox News, People). Comparatively, 24% of the human-driven “top stories” section came from three sources (Washington Post, CNN, NBC News).</li><li>For Facebook, one axis of diversity is cross-cutting content, i.e. ensuring that users are exposed to content from across the partisan aisle. In 2015, the company <a href="https://scontent-lhr8-1.xx.fbcdn.net/v/t39.8562-6/240831368_322090006325589_2566761670306389107_n.pdf?_nc_cat=103&amp;ccb=1-7&amp;_nc_sid=ad8a9d&amp;_nc_ohc=nc_oyxGaugoAX8vJoHB&amp;_nc_ht=scontent-lhr8-1.xx&amp;oh=00_AfBQS5fQHfoKOm15EYf0GIGclfrQNIYgliHb-zF31U2SjA&amp;oe=64D0DB2F">conducted a study</a> on exposure to ideologically diverse news and opinion with 10.1 million US users and 7 million links over a six-month period and found “~45% of the hard content that liberals would be exposed to would be cross-cutting, compared with 40% for conservatives.” More recently, a <a href="https://www.science.org/doi/10.1126/science.ade7138">collaborative study</a> conducted by academics with Facebook focusing on the 2020 US election, found “[conservative] audiences are more homogeneously conservative and, therefore, more isolated.” This is a result of both individual preferences and platform decisions.</li><li>Spotify has run <a href="https://ide.mit.edu/wp-content/uploads/2020/03/SSRN-id3555927.pdf">multiple</a> <a href="https://research.atspotify.com/2020/12/algorithmic-effects-on-the-diversity-of-consumption-on-spotify/">experiments</a> trying to ascertain the impact of recommendations on consumption diversity for podcasts as well as music. In the case of music, the company found that personalized recommendations (i.e. recommendations based on past listening behavior) led to 25% more streams and 10% fewer skips for users who listened to less diverse music (“specialists”). Song streams even went up by 10% for users with more diverse tastes. In the case of podcasts, Spotify found that personalization led to users consuming more homogenous podcasts (decreasing the individual-level diversity of podcast streams by 11.5%) even though the aggregate diversity of podcasts streamed increased by almost 6%.</li></ul><p>The different conceptualizations of diversity means that there isn’t a single optimal measure or a consensus as to what the baseline should be. If you are unhappy with some of the numbers reported above, what do you think the numbers should be? As <a href="https://www.sciencedirect.com/science/article/abs/pii/S0747563218303650?via%3Dihub">Nechustai &amp; Lewis</a> put it in their study of news recommender diversity, “in most cases, public-facing algorithms … do not function catastrophically or perfectly, but somewhere in between. Precisely where they fall on such a spectrum remains open for debate.”</p><h3>Algorithmic approaches to media diversity</h3><p>Most recommender systems today include some mechanism for diversifying the content shown to users as well as the users or creators whose content is shown. These systems account for <a href="http://rishabhmehrotra.com/papers/recsys2020-divergent-recs.pdf">multiple conflicting objectives</a> (including trade-offs between diversity and <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3555927">engagement</a>, <a href="https://www.proquest.com/openview/cbead732d695acef578e851eac7628ee/1?pq-origsite=gscholar&amp;cbl=18750&amp;diss=y">fairness</a>, <a href="https://www.sciencedirect.com/science/article/pii/S0306457320309511%23!">accuracy</a>, and <a href="https://www.sciencedirect.com/science/article/pii/S163107051930043X?via%3Dihub">other</a><a href="https://link.springer.com/article/10.1007/s11390-020-0135-9"> objectives</a>), of which diversity is simply one goal — and probably not the primary one.</p><p>Different recommender systems incorporate diversification algorithms at different stages of the pipeline, <a href="https://www.sciencedirect.com/science/article/pii/S0306457321002053">typically in a re-ranking step</a> that comes after initial item-ranking. This re-ranking approach tries to re-order the list of selected items to prevent homogeneity, such as having multiple posts in a row from the same person or on the same topics. In this design, if the <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">candidate generation</a> step did not produce a sufficiently diverse set, the subsequent diversification step won’t help.</p><h3>Diversity for consumers</h3><p>Almost all of the <a href="https://www.sciencedirect.com/science/article/pii/S0950705117300680">many available diversification algorithms</a> ultimately depend on the existence of “a <a href="http://www.cs.toronto.edu/~ashton/pubs/alg-effects-spotify-www2020.pdf">consistent notion</a> of similarity that can compare any pair of items.” This can also be referred to as “similarity metric,” “dissimilarity metric,” or “distance metric.” There are many different item attributes that such a metric might consider, as in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9H4R7UCaU0VUzSX7kjyjkQ.png" /><figcaption>Different attributes of an item that a similarity metric might take into account.</figcaption></figure><p>While a similarity function compares only two items, the diversity of an entire list of items is called <a href="https://castells.github.io/papers/recsys2011.pdf">intra-list diversity</a>. Intra-list diversity is sometimes defined as the average pairwise dissimilarity between the items recommended to a user on a single “slate,” the set of items displayed to a user in a single session, though there are <a href="https://linkinghub.elsevier.com/retrieve/pii/S0950705117300680">many other definitions</a>.</p><p>For users consuming content, diversity is important both within a single slate (as measured by intra-list diversity), and across multiple slates.</p><ul><li><a href="https://link.springer.com/article/10.1007/s00607-018-0687-5"><em>Per slate</em></a>:<em> </em>Diversity in a single session, to maintain short term engagement, prevent boredom, and avoid redundancy. A user is served a diverse slate of recommendations in a single go that takes into account what they are likely to engage with as well as what they might find novel or serendipitous. (Example: <a href="https://link.springer.com/article/10.1007/s10844-013-0252-9#Sec3">ClusterDiv</a>, <a href="http://www2.informatik.uni-freiburg.de/~cziegler/papers/WWW-05-CR.pdf">Topic Diversification</a>, <a href="https://dl.acm.org/doi/pdf/10.1145/3269206.3272018">Determinantal Point Process</a>)</li><li><a href="https://link.springer.com/article/10.1007/s00607-018-0687-5"><em>Across slates</em></a><em> (longer term)</em>:<em> </em>Diversity across sessions, to maintain longer-term retention/engagement. A secondary goal is to avoid rabbit holes. As a user’s interests evolve, a user’s recommendations over time must be diverse; this can also help prevent the user from going down rabbit holes. (Example: <a href="https://amatriain.net/pubs/lathia_sigir10.pdf">Temporal Switching or Temporal User-based Switching</a>, <a href="https://www.arxiv-vanity.com/papers/1905.12767/">SlateQ</a>)</li></ul><p>Homogenous recommendations in a single session can cause boredom, while recommendations over multiple sessions that don’t account for changes in user interests or preferences can reduce longer-term engagement. Repetitive recommendations can lead to rabbit holes. Moreover, monotonous recommendations are <a href="https://www.pnas.org/doi/10.1073/pnas.1000488107">not necessarily useful</a>: “real value is found in the ability to suggest objects users would not readily discover for themselves.” Diversity generally <a href="https://papers-gamma.link/static/memory/pdfs/153-Kunaver_Diversity_in_Recommender_Systems_2017.pdf">minimizes overfitting</a> to a user’s preferences and habits, which can be a problem when systems optimize only for accuracy.</p><p>Additionally, different users may want different amounts of diversity. For example, some users may be satisfied listening to a small set of popular artists, while others may value musical exploration. To estimate the amount of diversity that a user finds most satisfying, recommenders can use measures like the <a href="https://www.sciencedirect.com/science/article/pii/S0020025516318035">Shannon’s entropy</a> or <a href="https://link.springer.com/article/10.1007/s10115-019-01371-0">root mean square diversity error</a> to quantify the difference between a user’s preferred level of diversity and the diversity of the recommendations. The <a href="http://iir2015.isti.cnr.it/proceedings/IIR_2015_submission_17.pdf">intuition</a> is, if a user has explored items with different characteristics in the past, they are more likely to be amenable to diverse recommendations. On the other hand, if a user has stayed within the confines of specific interests, they’ll be less satisfied with diverse recommendations.</p><h3>Diversity for producers</h3><p>Diversity can also be evaluated in terms of the audience for each item, rather than the items for each user. This is known as producer diversity, and it can have a large effect on the economics of content production. <a href="https://web-ainf.aau.at/pub/jannach/files/Conference_UMAP_2013.pdf">Multiple</a> <a href="http://neconomides.stern.nyu.edu/networks/07-10_Fleder_Hosanagar_Blockbuster_Culture&#39;s.pdf">studies</a> have shown that some recommender systems — especially those using collaborative filtering — can expose individuals to a wide range of items (high consumer diversity), while selecting those items only from a small set of sources (low producer diversity). This results in the system being profitable for only a small subset of providers, also known as the “superstar economics” effect. In such an environment, new entrants fail to find an audience and smaller players fail to grow their consumer base beyond their typical traditional audience.</p><ul><li><a href="https://www.contrib.andrew.cmu.edu/~ravi/simods19.pdf"><em>Single item</em></a>: Ensure that the same item is shown to a diverse set of users irrespective of their demographics, interests, and ideological beliefs.</li><li><a href="https://www.jstage.jst.go.jp/article/transinf/E100.D/12/E100.D_2017EDR0003/_pdf/-char/en"><em>Item-group</em></a>: Ensure that all items within a group (e.g. movies) get shown to at least some users. Ensure that recommendations are dissimilar even within an item group; for example, show different genres of movies if the system selects movies to serve a user.</li></ul><p>Single-item diversity ensures that diverse user segments — be it based on demographics, interests, ideological beliefs, or some other metric — see each item regardless of popularity. Item-group diversity, on the other hand, refers to the idea that a diversity of items within a group are recommended. For example, on a music streaming platform, users don’t just see different genres of music, but also see a diverse set of artists within each genre. This ensures that small or new producers in each genre have a chance at success.</p><p>Producer diversity can be measured in a variety of ways. <a href="https://dl.acm.org/doi/10.1145/1864708.1864761">Coverage</a> is the percentage of items in the catalog recommended at least once. High coverage ensures that recommendations are extracted from the long tail as opposed to favoring popular items. Alternatively, the <a href="https://www.site.uottawa.ca/~nelkadri/CSI5389/Papers/143-Recommender%20Systems%20and%20their%20Impact.pdf">Gini coefficient</a>, widely used in economics to measure the inequality of income distributions, can be used to measure inequality of item popularity instead. <a href="https://ide.mit.edu/wp-content/uploads/2020/03/SSRN-id3555927.pdf">Spotify</a> has used the Gini coefficient to determine the impact of personalization on podcast consumption, finding that it reduced market concentration and distributed a larger fraction of streams to less popular podcasts.</p><h3>Where next</h3><p>There are times when the different interpretations and implementations of diversity can be at odds with one another, and there are times they might complement one another. Because the term is so loaded, it’s important to hone in on specifics when discussing how to incorporate and evaluate diversity in recommenders.</p><p>Societal objectives might be at odds with corporate objectives. Even within the same platform, diversity might be treated differently for different contexts — be it the main content feed, suggested friends, or recommended groups — with no agreement as to what is the optimal baseline, how much personalization is too much personalization, and the extent to which these things should stay constant or change over time. Designers of recommender systems must determine what types of diversity the relevant stakeholders care about, and measure it over time.</p><p>Stakeholder desires and chosen metrics can act as guiding principles, but key questions around what optimal diversity looks like in any system remain subject to discussion. As Nechustai &amp; Lewis<a href="https://www.sciencedirect.com/science/article/abs/pii/S0747563218303650?via%3Dihub"> ask</a>, “What standards should be used to assess [recommender] performance? Which visions of the public and of public life should such algorithms embody and encourage?”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2c86be9f08f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/what-is-media-diversity-and-do-recommender-systems-have-it-b2c86be9f08f">What is Media Diversity and Do Recommender Systems Have It?</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s the Difference Between Search and Recommendation?]]></title>
            <link>https://medium.com/understanding-recommenders/whats-the-difference-between-search-and-recommendation-c32937506a29?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/c32937506a29</guid>
            <category><![CDATA[recommender-systems]]></category>
            <dc:creator><![CDATA[Luke Thorburn]]></dc:creator>
            <pubDate>Wed, 25 Oct 2023 07:47:20 GMT</pubDate>
            <atom:updated>2023-10-25T07:47:20.354Z</atom:updated>
            <content:encoded><![CDATA[<p>— <a href="https://lukethorburn.com/">Luke Thorburn</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6FJD-8eHTDJ4haUBfqH9Ow.png" /></figure><p>The question of whether there is a meaningful difference between search and <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">recommendation</a> comes up frequently in discussions of platform harms, where recommendation is sometimes said to show people things they didn’t ask for. There are of course differences, but in many ways these types of systems differ in degree, not in kind, so it’s hard to draw a clean line between them — especially the sort of line you’d want when designing regulation.</p><p>Both search and recommendation are types of ranking algorithms. These are the systems which filter and order everything in the digital world, reducing it to the scale of human consumption. In this article we’ll map out dozens of real-world ranking products and features along five axes. These are:</p><ol><li>the explicitness of the ranking signals,</li><li>the degree of personalization,</li><li>the explore-exploit tradeoff,</li><li>the openness of the inventory, and</li><li>the balance between engagement and content signals.</li></ol><p>We’ll pay particular attention to the many systems that fall “in the middle” and confound clean distinctions. For instance, YouTube has a search bar and recommendations on the homepage, but it also lists related videos after the user has already clicked on something, which is somewhere between the explicit query of search and the query-free operation of the homepage.</p><p>Grappling with this complexity is important because consequential decisions — such as those relating to legal liability or auditing requirements — can hinge on how ranking systems are categorized. For example, in the recent US Supreme Court case <em>Gonzalez v Google</em>, the plaintiffs <a href="https://www.supremecourt.gov/DocketPDF/21/21-1333/247780/20221130182721608_GonzalezPetMerits.pdf">argued</a> that search engines are different from recommender systems because they “only provide users with materials in response to requests from the users themselves.” But in amicus briefs to the Court, several parties <a href="https://www.supremecourt.gov/DocketPDF/21/21-1333/252639/20230119133058729_FINAL%20NYU%20Stern%20Amicus%20Brief%20-%20Gonzalez%20v.%20Google%20LLC.pdf">argued</a> <a href="https://www.supremecourt.gov/DocketPDF/21/21-1333/252677/20230119145528517_21-1333_CRAIGSLIST%20INC.%20Amicus%20Brief.pdf">that</a> <a href="https://www.supremecourt.gov/DocketPDF/21/21-1333/252653/20230119140359911_21-1333%20Brief%20of%20Information%20Science%20Scholars.pdf">no</a> <a href="https://www.supremecourt.gov/DocketPDF/21/21-1333/252674/20230119145120402_Gonzalez%20-%20Reddit%20bottomside%20amicus%20brief.pdf">clear</a> <a href="https://www.supremecourt.gov/DocketPDF/21/21-1333/252658/20230119140759469_21-1333%20bsac%20Twitter%20Amicus%20Brief.pdf">line</a> can be drawn between the two.</p><p>While social media users may not explicitly enter search terms when viewing a feed, the underlying recommender system effectively conducts a search based on a combination of previous user actions and the current context. There are many ways a user might “ask for” content, such as following accounts, joining groups, or navigating to particular topics or channels, and this information often feeds into recommender systems. Meanwhile, search results are often personalized based on the system’s best guess of what the user wants. Because of this ambiguity, interpreting laws that hinge on whether someone “asked” to see something is tricky.</p><blockquote>[Consider] just a regular search engine … you put in a search and something comes back, and in some ways, you know, that’s one giant recommendation system.</blockquote><blockquote><em>— </em><strong><em>Justice Elena Kagan</em></strong><em>,</em><strong><em> </em></strong><a href="https://techpolicy.press/transcript-gonzalez-v-google-oral-argument/"><em>Transcript of Gonzalez v Google Oral Argument</em></a><em> (2023)</em></blockquote><p>Historically, there was perhaps more of a technical difference, with web search using algorithms like <a href="https://en.wikipedia.org/wiki/PageRank">PageRank</a> and recommenders using techniques like <a href="https://en.wikipedia.org/wiki/Collaborative_filtering">collaborative filtering</a>. But modern approaches are more alike. At the center, ranking systems have a “<a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">value model</a>” or “scoring function” which combines many different types of signals into an overall score for each item to be ranked, and then sorts the items by this measure of value. The boundary is also blurry in academic literature, where both “search” and “recommendation” are included within a larger field of “information retrieval.”</p><p>Fundamentally, search and recommendation and everything in between are computational tools to help us manage the <a href="https://arxiv.org/pdf/2301.09976.pdf#page=8">allocation of human attention</a>. It’s difficult to imagine navigating the modern world without such systems, but there are personal, social, political and economic consequences to the way these systems work, which is why we will probably always be arguing over how to categorize them for policy purposes. Rather than trying to put everything into either the “search” or “recommendation” box, we have tried to map the space of ranking approaches.</p><h3>Explicitness of Ranking Signals</h3><p>It is <a href="https://stackoverflow.com/a/17876253">often</a> <a href="https://www.supremecourt.gov/DocketPDF/21/21-1333/247780/20221130182721608_GonzalezPetMerits.pdf#page=25">said</a> that recommenders are different from search algorithms because they don’t require an explicit prompt from the user, that recommenders show people things that they didn’t explicitly “ask for.” But there are many cases in which “whether you asked for it” is not well defined, and it is more accurate to think of ranking systems as positioned on an axis from those that use entirely explicit signals for ranking (e.g., search terms) to those that rely mostly on implicit signals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PH13gz8V1zqZ7JJ4bFpdUQ.png" /></figure><p>At one end of the axis lie search algorithms, such as file search in Google Drive, that essentially rank items based on the degree to which they match the explicitly-entered search term. At the other extreme, recommenders like the <a href="https://newsroom.tiktok.com/en-us/how-tiktok-recommends-videos-for-you">TikTok “For You”</a> feed, or <a href="https://transparency.fb.com/features/explaining-ranking/ig-explore/">Instagram “Explore”</a>, predominantly use implicitly-collected engagement data such as dwell time.</p><p>But many high-profile ranking algorithms lie somewhere in between these extremes. Consider three examples from YouTube: the homepage, “Up Next” videos, and search. All of these algorithms use both explicit signals (e.g., the accounts you subscribe to) and implicit signals (e.g., predicted engagement based on your historical engagement data), but weight each type of signal differently.</p><ul><li><strong>YouTube homepage </strong>— Mostly based on historical engagement (implicit).</li><li><strong>YouTube Up Next </strong>— Based on both historical engagement (implicit) and the current video (explicit).</li><li><strong>YouTube search </strong>— Mostly based on search terms (explicit).</li></ul><p>Other examples of ranking algorithms that combine implicit and explicit signals include:</p><ul><li><strong>Google Maps search</strong> — Based on both the search terms (explicit) and your location (implicit).</li><li><strong>Spotify playlist continuation </strong>— Based on both the songs you added to the playlist (explicit) and predictions about what similar songs you would like (implicit).</li></ul><p>These examples show that in many cases it is not clear what it means to “ask for” certain content, which undermines attempts to regulate around a principle that “if people ask for <em>x</em> they should be able to see it, but if they don’t ask for <em>x</em> they shouldn’t be shown it”. There is too much gray area. If you are browsing deep in the <em>Politics</em> podcast category on Spotify and find a podcast by an extremist political group, did you “ask” to see it? If you join a Facebook group in which faulty medical advice is often shared, did you “ask” to see it?</p><p>Even search interfaces (where people do explicitly “ask” for something) involve a bunch of complex value judgements, whether or not the system designers are aware of them. If someone searches for advice about abortion, what should you show them? How do you rank products in Amazon search to be fair to different sellers? These are important questions which get obscured by a focus on providing users only what they ask for.</p><h3>Personalization</h3><p>A second axis is the degree to which the outputs of a ranking algorithm are personalized. Our focus here is on interest-based personalization, rather than the more generic customization that occurs when tailoring outputs to a large category of users (such as those in a particular region or language community).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BvvZS8Ys-2FTfbxJJAQIhA.png" /></figure><p>At one end of the axis lie highly personalized recommenders such as the TikTok “For You” feed or the suggested files at the top of the root directory in Google Drive. At the other end are ranked outputs that are the same for everyone, such as Wikipedia search results or the ordering of posts within a subreddit.</p><p>In between are ranking algorithms that are personalized to varying degrees. These include — perhaps counter-intuitively — search results, which are often personalized and function much like engagement-optimizing recommender systems. This is true of search algorithms on <a href="https://doi.org/10.1145/3392854">YouTube</a>, <a href="https://transparency.fb.com/features/explaining-ranking/fb-search/">Facebook</a> and <a href="https://doi.org/10.1145/3534678.3539080">X</a> (formerly Twitter). LinkedIn tries to <a href="https://doi.org/10.1145/2806416.2806615">guess your intent</a> (e.g. “content consuming”, “job seeking”, or “hiring”) and uses this information, along with <a href="https://doi.org/10.1145/2911451.2927018">information from your connections</a>, to personalize search results. Search engines including Google also <a href="https://support.google.com/websearch/answer/12410098?hl=en">personalize some results</a> (and have done since <a href="https://youtu.be/EKuG2M6R4VM">at least 2009</a>). While such personalization sparked <a href="https://www.ted.com/talks/eli_pariser_beware_online_filter_bubbles">early concern</a> about filter bubbles, much of this is fairly innocuous. For example, when you search for a trade such as “plumber,” Google will use your location to return businesses that are in your area.</p><h3>Explore-Exploit Tradeoff</h3><p>A third axis captures the degree to which a ranking algorithm engages in <strong>exploitation</strong> (giving users what it already knows they want) versus <strong>exploration</strong> (giving users new things, sometimes randomly, to find out if they like them). The amount of exploration is closely related to the concepts of diversity and <a href="https://doi.org/10.1007/s11390-020-0135-9">serendipity</a> in recommender systems, both of which have been widely studied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AORx5Cvt1WLhFDgCAAP4lg.png" /></figure><p>Most recommenders sit somewhere in the middle of the axis. For example, the TikTok “For You” feed and the YouTube homepage will try to strike a balance between exploration and exploitation, recommending content they can confidently predict you like, but also more diverse, exploratory recommendations to discover new interests. In contrast, search algorithms like Google Search are weighted mostly towards exploitation, assuming the user knows what they are looking for.</p><p>Social media platforms often provide versions of their recommenders that are deliberately more exploratory, such as the Instagram “Explore” feed or the YouTube “New to you” recommendations. Amazon’s “Based on your history” recommendations are at the other extreme, heavily exploiting knowledge of what products you have recently viewed. Spotify tries to <a href="https://www.cs.utoronto.ca/~ashton/pubs/alg-effects-spotify-www2020.pdf">personalize the explore-exploit tradeoff</a> for each user depending on how much diversity they like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlubTfvn2_sjLoDoK5nCZA.png" /><figcaption><em>YouTube’s “New to you” feature, which is more exploratory than the default homepage recommender.</em></figcaption></figure><p>Another example comes from online advertising. The <a href="https://doi.org/10.1145/3383313.3412214">algorithms</a> that choose which alternative ads from the same campaign to show people can be thought of as ranking algorithms, and these tend to change from being highly exploratory early in a campaign (to find out which ads work), to highly exploitative late in a campaign (to maximize sales).</p><h3>Openness of Inventory</h3><p>A fourth axis captures whether the inventory — the set of items being ranked — is <a href="https://ejlt.org/index.php/ejlt/article/view/686">open, curated, or closed</a>. Can anyone submit anything to be ranked? Or does the inventory only include items vetted or created by the platform?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1CTksx8qf7PHKDhPlTG2jg.png" /></figure><p>At one end of the axis are ranking algorithms with inventories that are almost completely open. Anyone can create a web page that appears in Google Search. Anyone with an account can post content on social media platforms like Facebook or X, or on other user-generated content sites like DeviantArt. At the other extreme are ranking algorithms on platforms with completely closed inventories, including The New York Times and BBC iPlayer. These platforms personally commission or create the content they host.</p><p>In the middle lie news recommenders such as Google News and The Factual, which will only include items they classify as news content, perhaps only from a curated list of domains known to be associated with news media organizations. Recommenders on platforms such as Audible, Spotify, or the AppStore are also somewhere in the middle — they are open, but it takes a bit of effort to satisfy the platform requirements and submit your content. Sometimes open platforms can also give the impression of being exclusive and closed (think podcast recommendations in Apple Podcasts) because content without any accrued reputation is so heavily downranked that it is never seen.</p><h3>Content versus Engagement</h3><p>A fifth axis describes the relative weight a ranking algorithm gives to two kinds of information: semantic information about the content, and information about the behaviors or engagement it tends to generate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5MEe1RuyUkrIg6oJwAJ1g.png" /></figure><p>At one end lie algorithms which rank items based solely on the properties of the content itself. A good example of this is The Factual, a recommender which ranks news articles <a href="https://www.thefactual.com/how-it-works/">based on</a> their tone and the reliability of their publisher, author and sources, without using any popularity, voting, or engagement feedback. At the other end are engagement-maximizing recommenders such as the TikTok “For You” feed, which ranks videos <a href="https://www.nytimes.com/2021/12/05/business/media/tiktok-algorithm.html">according to</a> which are most likely to increase behavioral outcomes such as user retention and time spent in the app.</p><p>In between lie ranking algorithms — primarily search algorithms — which use a mixture of content and engagement signals. For example, a leaked list of signals used by Russian search engine Yandex <a href="https://webmarketingschool.com/complete-internal-list-of-yandex-ranking-factors/">includes</a> “text relevance” and whether the content is news (content), but also “how often a given URL is clicked for a given query” (engagement). Certain engagement patterns can also help identify content that people don’t want (e.g., if you click on a search result and then immediately come back, that suggests it wasn’t what you were looking for). Search algorithms on platforms like Netflix and Spotify also use a combination of content and engagement signals, balancing relevance to your query with the probability that you will watch a video or listen to a song.</p><h3>The Allocation of Attention</h3><p>Search engines and recommenders are both types of ranking algorithms, and there are all sorts of hybrid forms between them. It is difficult to draw clean lines that make sense as legal distinctions, and in particular, the claim that search is categorically different because it involves users “asking” to see things is too simplistic. Instead of thinking in binary “search vs. recommendation” terms, we encourage policymakers to consider the whole space of possible ranking algorithms. The axes above provide a map of the space that might help to think through the consequences of proposed categorizations.</p><p>Beyond discrete rankings, there are many other algorithmic systems that do similar kinds of prioritization, including large language models, reputation systems, and “<a href="https://www.cybersecurepolicy.ca/policy-brief-contextualization-engines">contextualization engines</a>”. Instead of drawing distinctions between these algorithms, it may be more fruitful to acknowledge that we care about them because they systematically <a href="https://arxiv.org/pdf/2301.09976.pdf#page=8">allocate human attention</a> in certain ways, and to consider what standards should be met by the algorithms to which we delegate this responsibility.</p><p><em>Luke Thorburn was supported in part by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (safeandtrustedai.org), King’s College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c32937506a29" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/whats-the-difference-between-search-and-recommendation-c32937506a29">What’s the Difference Between Search and Recommendation?</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Amplification Measurable]]></title>
            <link>https://medium.com/understanding-recommenders/making-amplification-measurable-2be548e5986c?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/2be548e5986c</guid>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Luke Thorburn]]></dc:creator>
            <pubDate>Sun, 07 May 2023 10:01:38 GMT</pubDate>
            <atom:updated>2023-05-07T10:01:38.696Z</atom:updated>
            <content:encoded><![CDATA[<p>— <a href="https://lukethorburn.com/">Luke Thorburn</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxJA84Jokc5tX3JwAy_snQ.png" /></figure><p>The term “amplification” is ubiquitous in discussions of recommender systems and even <a href="https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32022R2065&amp;qid=1666857835014">legislation</a>, but is not well-defined. We have <a href="https://medium.com/understanding-recommenders/what-will-amplification-mean-in-court-a6b94bad6354">previously argued</a> that the term is too ambiguous to be used in law, and that the three most common interpretations — comparison with a counterfactual baseline algorithm, mere distribution, or a lack of user agency — are either difficult to justify, or difficult to measure consistently. That said, the way recommender systems are designed <em>does</em> influence how widely different types of content are distributed. Amplification is an evocative term to describe this reality, and remains a <a href="https://knightcolumbia.org/events/optimizing-for-what-algorithmic-amplification-and-society">prominent</a> subject of research. For these reasons, it would be useful to have a common understanding of what amplification is and how to measure it.</p><p>In this piece, we propose five properties that measures of amplification should have if they are to be useful in discussions of recommender policy. Specifically, they should: (1) define the content of interest, (2) define an appropriate baseline, (3) focus on impressions as the relevant outcome, (4) isolate the effect of the algorithm from that of human behavior, and (5) specify the time horizon over which they apply. The first three properties are relatively easy to satisfy, while the latter involve <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">causal inference</a> and systemic equilibria, and pose greater challenges. A measure that satisfies these properties can be said to measure <em>relative algorithmic amplification</em>.</p><blockquote><strong>relative algorithmic amplification: </strong><em>A change in the distribution under one algorithm as compared to an alternative algorithm, holding user behavior constant.</em></blockquote><p>We think that this definition is both clarifying and useful, and that, as a research community, our goal should be to develop measures of relative algorithmic amplification which satisfy each of the above conditions. We discuss each in more detail below.</p><h3>Easy-ish Properties</h3><h4>1. Content</h4><p>When measuring amplification we first need to answer the question: amplification of what? There are many types of content for which we might be interested in measuring amplification, such as <a href="https://doi.org/10.1177/1329878X20946113">conspiracy theories</a>, <a href="https://doi.org/10.1126/science.aap9559">false news</a>, posts with a <a href="https://doi.org/10.1073/pnas.2025334119">certain political slant</a> or those with <a href="https://doi.org/10.1038/s41562-021-01133-5">morally-charged language</a>.</p><p>However, none of these descriptions is enough, by itself, to define what content is in scope. For example, the amount of “fake news” <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3340173">varies greatly</a> depending on how it’s defined. So, attempts to measure amplification should <strong>be transparent and precise about how content was identified as in-scope</strong>. For example, if the content was identified manually, what were the instructions given to human raters? What was their inter-rater reliability? Sometimes, content is identified using <a href="https://arxiv.org/pdf/2302.04450.pdf">hashtags</a>, <a href="https://osf.io/wku5b/">key words</a>, or <a href="https://journals.sagepub.com/doi/full/10.1177/14614448221137324">seed accounts</a>, which may or may not capture the sense of broad categories like “conspiracy” or “extreme,” which may themselves be contested terms. Or, if the content was identified using a classification algorithm, what data was it trained on? How accurate was the classification as compared to human raters, and what <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">trade-offs</a> does it create between false positives and false negatives? For the most part, studies on amplification already document such information.</p><h4>2. Baseline</h4><p>When measuring amplification we need to answer the question: amplified relative to what? The claim that content has been amplified implies a counterfactual scenario in which, had the algorithm been designed differently, the content would not have been distributed so widely. There are many baseline algorithms which might serve as useful benchmarks, depending on the context. These include reverse chronological feeds, a feed of randomly selected items, or a previous version of the recommender system.</p><p>Attempts to measure amplification should <strong>be transparent and precise about which alternative recommendation algorithm they are using as a baseline</strong>. There is no universally “correct” or “neutral” baseline — can you really say, quantitatively, how many people <em>should</em> see any particular item? — although certain baselines might be better or worse comparisons. Indeed, it is probably more useful to compare amplification under multiple (i.e., more than two) candidate algorithms. Currently, some discussions of amplification <a href="https://doi.org/10.1073/pnas.2025334119">do</a> specify a baseline, but others <a href="https://policyreview.info/pdf/policyreview-2021-2-1565.pdf">do not</a>.</p><h4>3. Outcome</h4><p>Third, when measuring amplification we need to specify what it is that amplification increases. The main two candidates are impressions (that is, the number of times the content was seen) and engagement (that is, reactions such as the number of times the content was liked or shared).</p><p>Of these, we think measures of amplification should <strong>use impressions as the outcome variable</strong>. Arguably, impressions <a href="https://www.theverge.com/interface/2020/7/22/21332774/facebook-crowdtangle-kevin-roose-nyt-tweets-interactions-reach-engagement">do a better job</a> of characterizing the algorithm (people usually have little control over the specific posts which appear in their feed), whereas other engagement measures better reflect user behavior. It matters if content is shown, regardless of whether the user explicitly reacted to it.</p><h3>Difficult-ish Properties</h3><h4>4. Algorithm v. User Behavior</h4><p>We are interested in measures of amplification that <strong>distinguish the contribution of the recommender from that of user behavior</strong>. If posts from the purple party receive a relatively large number of impressions, is that because the algorithm is biased towards the purple party, or because people tend to engage more with purple party content?</p><p>We are not dismissing the role of human behavior. The ways and degree to which human behavior and psychology contribute to the spread of certain types of content is an important research question. But in the context of recommender policy discussions, we are interested in understanding the effect of varying the algorithm, because that is where platforms and regulators have the most control.</p><p>Fundamentally, this is a problem of <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">causal inference</a>. We would like to know: if we change the algorithm but “hold user behavior constant”, what will be the effect on impressions? When thinking about questions of causality, it is helpful to draw simple graphs which represent whether different variables influence one another. For example, here is one possible causal model of amplification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpJXKDKa7eoUb4hBUsQM6Q.png" /></figure><p>In this model, both user behavior and the choice of algorithm affect what is shown. There are no other arrows, so it is assumed in this model that there are no other direct causal relationships between these variables. If this simplistic model was accurate, then it would be relatively easy to distinguish the effect of the algorithm from that of user behavior. One could simply sequentially try out different algorithms and observe the change in what is shown. The pair of converging arrows — often referred to as a <a href="https://en.wikipedia.org/wiki/Collider_(statistics)">collider</a> or “v-structure” — is a standard situation in causal inference.</p><p>Unfortunately, reality is not that simple. A <a href="https://www.commerce.senate.gov/services/files/62102355-DC26-4909-BF90-8FB068145F18">more accurate</a> causal graph would look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*noDmI-VUHoEIQ9ouQE7eNQ.png" /></figure><p>In this model, it is no longer straightforward to tease apart the effect of the algorithm from that of user behavior — both producers and consumers of content. This is because changing the algorithm will also cause user behavior to change: users develop an intuition for how the algorithm works, and their preferences may be influenced by the content they are shown. In addition, this model has three closed loops, which makes it trickier to do causal inference. Arguably, this model is still too simplistic and misses core aspects of reality such as external cultural norms, the decisions made by competing platforms, or inauthentic forms of engagement such as astroturfing.</p><p>This is not to say that distinguishing the contribution of the algorithm from that of user behavior is impossible, only that it is difficult. Strategies which can help include (1) <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">on-platform experiments or A/B tests</a> to compare two algorithms, (2) stipulating that research participants engage in a <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4114905">predetermined manner</a> to prevent user behavior from being affected by other variables (e.g., “always click on the third recommended video”), and (3) removing loops by distinguishing between variables measured at <a href="https://doi.org/10.1111/j.1468-2885.2007.00296.x">different points in time</a>.</p><h4>5. Equilibrium Effects</h4><p>Finally, attempts to measure amplification should <strong>specify the time horizon over which they apply</strong>. When a new recommender algorithm is introduced, other actors — users, journalists, editors, politicians, content creators, competing platforms, foreign governments — all respond by changing their behavior, and it takes time for these changes to play out. In some experiments, users have taken <a href="https://dl.acm.org/doi/abs/10.5555/3367471.3367481">3–6 months</a> to adapt. Relative algorithmic amplification is effectively a prediction about what will happen to impressions if you change from one algorithm to another, so it is important that it is clear about the time period over which this change is expected.</p><p>By analogy, consider a shoe manufacturer who starts making sneakers with an innovative new design and wants to measure how this decision affected its revenue. If the new design is popular, there might be an initial increase in revenue as consumers — who would otherwise have bought their shoes elsewhere — purchase the new design. This is the short run, non-equilibrium effect. But as time progresses, all the people who like the design will have likely bought it, the design might become mainstream and seem less cool, and rival firms might introduce similar sneakers which directly compete. After all these changes wash out, the sneaker market reaches a new equilibrium. The corresponding long run, equilibrium effect on the annual revenue of the firm might be quite different to the short run effect of introducing the new design.</p><p>When a platform changes its recommender algorithm, there are similar dynamics at play. Producers of content <em>strategically adapt</em> the style of content they produce to attract attention in the new regime. (For example, when Facebook reduced the visibility of sensationalistic posts about health and politics, some publishers started <a href="https://www.wsj.com/articles/facebook-politics-controls-zuckerberg-meta-11672929976">posting more sensationalistic posts about crime</a>, and lay users commonly learn to use <a href="https://www.washingtonpost.com/technology/2022/04/08/algospeak-tiktok-le-dollar-bean/">euphemisms and codewords</a> to avoid having the visibility of their posts restricted.) There are <em>saturation effects</em> as users’ appetite for certain types of content becomes satiated, <em>rival platforms</em> who change their own product to compete, decisions by some users to <em>switch platforms</em>, and <em>network effects</em> that can accelerate or hamper changing levels of engagement.</p><p>Comparing the long run, equilibrium effects of different recommender algorithms is <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">really hard</a>, and perhaps impossible to do perfectly. Because the world is so interconnected, it is not possible to completely isolate an experimental control group. A platform could trial a new recommender system with, say, 50% of its users, but those users still interact with everyone else, so the effects of using the new algorithm may cause changes to and “contaminate” the baseline against which the platform is trying to measure the impact of the new recommender. There are methods that can help (e.g. trialing algorithms in one country at a time, and maintaining a long term control group like the <a href="https://www.pnas.org/doi/10.1073/pnas.2025334119">1% of Twitter users</a> who have only ever had a chronological feed), but they are not perfect solutions. It is also debatable whether a long run equilibrium is ever reached in the context of constantly evolving mass communication.</p><p>It may be that we can only estimate the short run, non-equilibrium effects of algorithmic changes. This is analogous to an <a href="https://en.wikipedia.org/wiki/Elasticity_(economics)">elasticity</a> in economics. In any case, it is important that measures of amplification are transparent about the time horizon over which they apply.</p><h3>A Realistic Definition</h3><p>Research on amplification has been held back by a lack of consistent definitions, and a corresponding shortage of practical metrics. In part, this is because “amplification” involves many moving pieces. We propose <strong>relative algorithmic amplification</strong> to emphasize the need to choose a baseline (relative) and factor out user behavior (algorithmic).</p><p>This definition has the great advantage of being clear about what the quantity to be measured actually is. But as the above discussion shows, that doesn’t mean it’s easy to measure. If “amplification” is something different from “distribution” we need to answer “amplified compared to what?” And if we want to use “amplification” to understand the effect of algorithm design, we need to be able to separate it somehow from user behavior, both in the short run and the long run. It may seem that our preferred definition brings in a lot of complexity — but this is a feature, not a bug. Amplification is an inherently complex idea, so it’s better that we use a definition that tells us where the difficulties lie.</p><p><em>Luke Thorburn was supported in part by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (safeandtrustedai.org), King’s College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2be548e5986c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/making-amplification-measurable-2be548e5986c">Making Amplification Measurable</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When You Hear “Filter Bubble”, “Echo Chamber”, or “Rabbit Hole” — Think “Feedback Loop”]]></title>
            <link>https://medium.com/understanding-recommenders/when-you-hear-filter-bubble-echo-chamber-or-rabbit-hole-think-feedback-loop-7d1c8733d5c?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/7d1c8733d5c</guid>
            <category><![CDATA[recommender-systems]]></category>
            <dc:creator><![CDATA[Luke Thorburn]]></dc:creator>
            <pubDate>Thu, 30 Mar 2023 07:57:52 GMT</pubDate>
            <atom:updated>2023-04-04T10:36:22.085Z</atom:updated>
            <content:encoded><![CDATA[<h3>When You Hear “Filter Bubble”, “Echo Chamber”, or “Rabbit Hole” — Think “Feedback Loop”</h3><p>— <a href="https://lukethorburn.com/">Luke Thorburn</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xcc2SAAe-9WBRgjVom7TQw.png" /></figure><p>While filter bubbles, echo chambers, and rabbit holes are part of the popular wisdom about what is wrong with recommender systems, and have received significant attention from academics, evidence for their existence is mixed. It seems like almost every research paper uses <a href="https://xkcd.com/927/">different definitions</a>, and reaches different conclusions depending on how the concepts are formalized. When people try to make the question more precise, they usually head in different directions, so the results they come up with are no longer comparable.</p><p>In this post, we’ll try to recap the history of these concepts, describe the limitations of existing research, and argue that the concepts are ultimately too muddied to serve as useful frameworks for empirical work. Instead, we propose that research should focus on <a href="https://medium.com/understanding-recommenders/is-optimizing-for-engagement-changing-us-9d0ddfb0c65e">feedback loops</a> between three variables: what is engaged with, what is shown, and what is thought. This framework can help us understand the strengths and weaknesses of the wide range of previous work which asks whether recommenders — on social media in particular — are causing political effects such as polarization and radicalization, or mental health effects such as eating disorders or depression.</p><p>Currently, there are studies which show that bots programmed to watch <a href="https://arxiv.org/pdf/2203.10666.pdf">partisan</a> or <a href="https://www.wsj.com/articles/how-tiktok-inundates-teens-with-eating-disorder-videos-11639754848">unhealthy weight loss</a> videos tend to be shown more of them. There are experiments showing that <a href="https://doi.org/10.1126/sciadv.abc4046">political ads</a> and <a href="https://www.nber.org/papers/w20798">cable news</a> can cause small changes in political opinions, and that manipulating the <a href="https://doi.org/10.1073/pnas.1320040111">emotional valence</a> of the Facebook feed can influence the emotions expressed by users in their posts. There is also a long history of “selective exposure” research showing that people often <a href="https://kf-site-production.s3.amazonaws.com/media_elements/files/000/000/133/original/Topos_KF_White-Paper_Nyhan_V1.pdf">engage more</a> with supporting information, and are <a href="https://doi.org/10.1145/2487788.2487807">more likely</a> to share news articles if they agree with them.</p><p>In other words, all of the parts of the engagement-to-recommendations-to-beliefs-to-engagement loop have been demonstrated in at least some context. But the effect sizes tend to be small or context-dependent, and because the specific measures used are not consistent, it is not straightforward to assemble these component results into evidence for a complete feedback loop. To do this, <a href="https://newsflows.eu/wp-content/uploads/2022/06/epsa2022.pdf">we need studies</a> that use consistent measures for the three variables — what is engaged with, what is shown, what is thought — and research designs that can empirically show the causal relationships between them.</p><h3>How we got here</h3><p>Distinctions are often drawn between filter bubbles, echo chambers, and rabbit holes, but they can all be thought of as types of <a href="https://www.darden.virginia.edu/sites/default/files/inline-files/05_16371_RA_KitchensJohnsonGray%20Final_0.pdf">information-limiting environments</a>. Concerns about the role of media and technology in creating such environments have been around since at least the 1950s — from Philip K. Dick’s “<a href="http://www.technovelgy.com/ct/content.asp?Bnum=637">homeostatic newspaper</a>”, to Joseph Klapper describing mass communication as an “<a href="https://archive.org/details/effectsofmasscom00klap/mode/2up">agent of reinforcement</a>”, to Nicholas Negroponte’s vision of a “<a href="http://assets.press.princeton.edu/chapters/s8468.pdf">Daily Me</a>”.</p><p>In the context of the internet, “<a href="https://www.google.co.uk/books/edition/Echo_Chambers/sEgHAAAACAAJ">echo chamber</a>” was popularized by Cass Sunstein in 2001, “<a href="https://www.google.co.uk/books/edition/_/zhRGngEACAAJ">filter bubble</a>” by Eli Pariser in 2011, and “rabbit hole” — a reference to <em>Alice in Wonderland</em>, or perhaps <em>The Matrix</em> — has been used as a metaphor for independent online investigations since at least the <a href="https://journals.sagepub.com/doi/abs/10.2190/YKU8-XDDK-2XUP-8CQK">mid 1990s</a>. All three phrases started as informal or quasi-academic concepts that have been repeatedly defined and redefined by researchers attempting to quantify the degree to which such phenomena are real.</p><p>We want to be able to answer the question: <em>Do recommender systems cause filter bubbles (or echo chambers, or rabbit holes)?</em> But this question is not specific enough. There are many different recommender systems, and many different ways of defining and measuring information-limiting environments. When people rephrase this question to make it precise and falsifiable, they do so in different ways, so their findings are no longer comparable.</p><p>Take “filter bubble” as an illustrative example. The following list gives a sample of the various ways the concept has been defined.</p><ul><li>“a unique universe of information for each of us” (<a href="https://www.google.co.uk/books/edition/_/zhRGngEACAAJ">Pariser, 2011</a>)</li><li>“[a technological] information network from which relevant voices have been excluded by omission” (<a href="https://doi.org/10.1017/epi.2018.32">Nguyen, 2018</a>)</li><li>“when a group of participants, independent of the underlying network structures of their connections with others, choose to preferentially communicate with each other, to the exclusion of outsiders” (<a href="https://www.politybooks.com/bookdetail?book_slug=are-filter-bubbles-real--9781509536443">Bruns, 2019</a>)</li><li>“the exclusion of the uninteresting at the expense of democracy” (<a href="https://doi.org/10.2478/nor-2021-0002">Dahlgren, 2021</a>)</li><li>“a decrease in the diversity of a user’s recommendations over time, in any dimension of diversity, resulting from the choices made by different recommendation stakeholders” (<a href="https://doi.org/10.1145/3511047.3538028">Leysen et al, 2022</a>)</li></ul><p>The above definitions are qualitative. When trying to measure filter bubbles quantitatively, people use numbers like:</p><ul><li>The weighted average of the squared difference between the expressed opinion of an individual with their social network neighbors, in a theoretical model of opinion dynamics simulated on the follower networks of real social media platforms. (<a href="https://arxiv.org/pdf/1906.08772.pdf">Chitra and Musco, 2019</a>)</li><li>A “time-weighted percentage of news sites visited [during a 4 week period] with political slant scores opposite to a user’s base ideology”. (<a href="https://www.darden.virginia.edu/sites/default/files/inline-files/05_16371_RA_KitchensJohnsonGray%20Final_0.pdf">Kitchens et al, 2020</a>)</li><li>The average ideological slant of videos that appear in Homepage or Up Next recommendations on YouTube, where the slant of the video is estimated by identifying the partisan elites that are followed by Twitter users who linked to the video. (<a href="https://arxiv.org/pdf/2203.10666.pdf">Haroon et al, 2022</a>)</li></ul><p>Inconsistent definitions and measurements are not the only limitations of existing research. Portions of the existing literature <a href="https://www.darden.virginia.edu/sites/default/files/inline-files/05_16371_RA_KitchensJohnsonGray%20Final_0.pdf">have</a> <a href="https://doi.org/10.2478/nor-2021-0002">been</a> <a href="https://dx.doi.org/10.2139/ssrn.4114905">criticized</a> for using <em>non-representative datasets</em> (e.g. only considering the most active social media users), lacking <em>temporal validity</em> (because platforms so frequently change their algorithms), using only <em>observational data</em> (which cannot disentangle the contribution of user behavior from that of the algorithm), making the <a href="https://medium.com/understanding-recommenders/what-does-it-mean-to-give-someone-what-they-want-the-nature-of-preferences-in-recommender-systems-82b5a1559157"><em>revealed preference</em></a> assumption (conflating user behavior with user preferences), falling for the <a href="https://web.stanford.edu/class/ed260/freedman549.pdf"><em>ecological fallacy</em></a> (for example, by showing a correlation between rates of internet use and political antipathy across regions — there is no guarantee that the individuals who use the internet a lot are the same people who dislike their political opponents) and, conversely, for making an <em>unsubstantiated leap</em> from individual effects (e.g. information consumption) to societal effects (e.g. democratic resilience).</p><p>Because of these definitional and methodological issues, we propose shifting attention from filter bubbles, echo chambers, and rabbit holes to a different concept: feedback loops between humans and machines. This concept seems better defined, can be broken into falsifiable questions about each stage of the process, and might help us better coordinate research efforts to produce lasting insight.</p><h3>The feedback loop model</h3><p>In theory, the recommender feedback loop works like this:</p><p>When faced with a <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">slate</a> of recommendations, a user will engage with some types of content more than others. This is called <a href="https://www.sciencedirect.com/science/article/abs/pii/S0065260108602129">selective exposure</a>. For example, they may engage with ingroup content more than outgroup content (due to <a href="https://www.annualreviews.org/doi/full/10.1146/annurev.soc.27.1.415">homophily</a> or <a href="https://doi.org/10.1016/S0079-7421(08)60315-1">confirmation bias</a>), with mainstream content more than indie content (due to cultural pressures), or with radical content more than moderate content (due to sensationalism or outrage). The recommender, because it is at least partially <a href="https://medium.com/understanding-recommenders/whats-right-and-what-s-wrong-with-optimizing-for-engagement-5abaac021851">optimizing for engagement</a>, starts to show more of the types of content that <em>are</em> engaged with. This changes what the users see, which causes their beliefs and preferences to change, becoming more aligned with what they see. Finally, this preference change causes the user to be even more selective, perhaps by explicitly following more homogeneous sources, or more consistently ignoring other types of content. Then the loop begins again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLmYBl1LOcyY6DztXrY3cQ.png" /></figure><p>The feedback loop concept isn’t new. Versions of it have <a href="https://arxiv.org/abs/2203.10666">been</a> <a href="http://eprints.lse.ac.uk/101413/1/Levy_echo_chambers_and_their_effects_accepted.pdf">mentioned</a> <a href="https://arxiv.org/pdf/2112.05084.pdf">previously</a> in the literature on filter bubbles, echo chambers, and rabbit holes. In the <a href="https://www.google.co.uk/books/edition/_/zhRGngEACAAJ">original book</a> on filter bubbles, Eli Pariser described a recommender “you loop”, which can behave “like reverb from an amplifier.” It is the focus of a <a href="https://responsibledecisionmaking.github.io/assets/pdf/papers/30.pdf">recent paper</a> on safe recommender design, and is fundamental to concerns that recommenders might be <a href="https://medium.com/understanding-recommenders/is-optimizing-for-engagement-changing-us-9d0ddfb0c65e">influencing us</a> (as has been shown to <a href="https://arxiv.org/pdf/2009.09153.pdf">occur</a> <a href="https://arxiv.org/pdf/2109.04083.pdf">in</a> <a href="https://dl.acm.org/doi/10.1145/3460231.3478849">simulations</a>). However, feedback loops are usually not the way that people think about and research these types of problems.</p><p>Feedback loops are also not unique to recommenders. They occur in traditional media such as cable news and newspapers, albeit over longer time periods as producers and editors adjust their coverage to chase ratings. Also, not all information-limiting environments involve a feedback loop. “Educational” curricula used by extremist organizations for recruitment can be viewed as rabbit holes that are not influenced <em>by</em> the “student”, yet still influence <em>them</em>.</p><p>We think there are at least three good reasons to think in terms of feedback loops.</p><p>First, the feedback loop framework <strong>clarifies the specific research questions which need to be answered</strong> to measure the extent to which information-limiting environments are a concern. Concretely, we need to quantify three variables — what is engaged with, what is shown, what is thought — and need to measure three causal relationships between them — (1) how what is engaged with affects what is shown, (2) how what is shown affects what is thought, and (3) how what is thought affects what is engaged with. These variables can be quantified in different ways, so there are different versions of the feedback loop depending on how this is done. But if the three variables are quantified consistently, and there is reliable evidence for the three causal relationships between them, then there is a strong, precise case for a “filter bubble”-like phenomenon.</p><p>Second, the model makes explicit that <strong>there must be a change in what is <em>thought</em></strong> — the user’s beliefs, opinions, preferences, emotions, etc. — for the phenomenon to be of interest. It is common for existing studies on filter bubbles or echo chambers to only consider partial fragments of the loop. For example, a study may reveal a limitation with regard to what is shown, but <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4114905">not demonstrate</a> that this limitation affects what is thought. But if the user isn’t influenced by what they see, what’s the problem? The concerns about echo chambers, filter bubbles, and rabbit holes assume that people are affected by the content they are exposed to — as opposed to using their own intelligence and agency to decide what to think — and this assumption needs to be tested.</p><p>Third, the feedback loop framework can <strong>help us talk about practical constraints on the power of recommenders to influence</strong>. The feedback loop does not continue forever. At some point, a form of saturation or equilibrium will be reached, perhaps because a <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">diversification stage</a> in the recommender ensures that the recommended content cannot exceed some level of homogeneity, or because people have a finite number of hours with which to engage a recommender, or because people are also influenced by other information sources (e.g. friends and family). The question of how much people can change before equilibrium is reached, and how long this process takes (one loop? more?) is an important one.</p><p>Before we move on we note that such feedback loops, if they exist, cannot be attributed solely to a recommender system. Only one of the three segments of the loop, the relationship between what is engaged with and what is shown, is determined by the algorithm and can be (relatively) easily influenced by recommender design changes. The other two relationships — between what is shown and what is thought, and what is thought and what is engaged with — are determined by human psychology and a complex set of other societal factors, and are thus harder to influence. Also, users and creators will adjust whenever the recommender is changed. As an example of the second-order effects involved, consider that content producers will be incentivized to produce content that reinforces feedback loops which lead to them receiving more attention. This in turn changes the distribution of content available for recommendation.</p><h3>Is there evidence for feedback loops?</h3><p>Feedback loops seem plausible in theory, but are they supported by evidence? Below, we review existing evidence for each of the three causal arrows in the feedback loop model.</p><h4>Does what is engaged with affect what is shown?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*D9iYphHem_vFqfqyKuqqOQ.png" /></figure><p>The question of whether a recommender shows more of the types of content that a user engages with might be the least interesting of the three, because this is (part of) what recommenders are <a href="https://medium.com/understanding-recommenders/whats-right-and-what-s-wrong-with-optimizing-for-engagement-5abaac021851">designed to do</a>. Responding to our interests makes these information filtering systems much more useful. Nonetheless, the question can be studied empirically, and it is important to do so because the strength of the effect can vary, and most real platforms have built-in safeguards that disrupt the effect for certain types of content (e.g. terrorist content).</p><p>Much of the existing literature on filter bubble-like phenomena focuses on the link between engagement and recommendation. For example, recent large-scale studies of <a href="https://arxiv.org/pdf/2203.10666.pdf">YouTube</a> and <a href="https://www.youtube.com/watch?v=nfczi2cI6Cs">TikTok</a> show that when a bot is programmed to “watch” certain types of content, they tend to get recommended more of those types of content. Explicitly “following” or interacting with a specific account is also a type of engagement, so studies showing that the accounts users interact with affect what they get shown — such as on <a href="https://arxiv.org/pdf/2004.09603.pdf">Facebook and Twitter</a> — also speak to this question.</p><p>Future work on this question would benefit from more consistency in how what is shown is measured. Specifically, the methods used to quantify subjective, qualitative properties of content — such as its political slant — <a href="http://dx.doi.org/10.18653/v1/D19-5548">vary</a> <a href="https://arxiv.org/abs/1607.06795">greatly</a>, and are difficult to compare. It is also important to <a href="https://www.darden.virginia.edu/sites/default/files/inline-files/05_16371_RA_KitchensJohnsonGray%20Final_0.pdf">distinguish</a> between the <em>slant</em> and <em>diversity</em> of content when assessing changes to what is shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*HSko21Itz1XBvHAlre1N7g.png" /></figure><p>Finally, studies that use bots to assess this question can only tell us so much. Bots may be more fixed or dogmatic in their habits of information consumption than real people, and may not respond to the content they are exposed to in the same way, particularly over the long term. In particular, people do not engage with content deterministically or uniformly at random, and modeling them with bots that <a href="https://arxiv.org/pdf/2203.10666.pdf">behave in this way</a> can <a href="https://arxiv.org/abs/2302.11225">significantly overstate</a> the degree to which extreme content is recommended. For this reason, it is important to either study this question in the context of <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">real users</a>, or to put <a href="https://arxiv.org/abs/1805.10000">considerable</a> <a href="https://arxiv.org/pdf/2208.01534.pdf">effort</a> into simulating users faithfully.</p><h4>Does what is shown affect what is thought?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*OryZIVUq3s3Q2MWp5MFYSw.png" /></figure><p>The question of whether the media people are exposed to affects their thoughts, beliefs, or attitudes is the focus of a sizable literature on <a href="https://doi.org/10.1146/annurev-psych-122414-033608">media effects</a>. To estimate a media effect, it is necessary to focus on particular kinds of thoughts or attitudes or emotions (e.g. political ideology, mental health) and to have some way of measuring them, which usually amounts to either (1) asking people what they think or (2) inferring what they think from what they do.</p><p>Most of the existing literature on media effects focuses on advertising or traditional media rather than recommenders. For example, the impact of non-personalized political advertising on voting preferences seems to be <a href="https://www.science.org/doi/10.1126/sciadv.abc4046">small regardless of context</a>, while spending more time watching Fox News has been shown to have a <a href="http://www.nber.org/papers/w20798">modest effect</a> on voting behavior. In the context of recommenders, the largest examples of media effects experiments we know of are cases where Facebook used a political mobilization message (pinned to the top of the feed) to <a href="https://doi.org/10.1038/nature11421">increase voter turnout</a>, and changed the emotions expressed in users’ posts by <a href="https://doi.org/10.1073/pnas.1320040111">manipulating what they saw in their feeds</a>.</p><p>On the one hand, current evidence appears to suggest that the direct effects of individual media items are small, on average (though they <a href="https://psyarxiv.com/dhg6k/">can be increased</a> with the use of personalized persuasive messaging). Further, a sizable fraction of people are simply <a href="https://doi.org/10.1177/2053168017702990">not interested</a> in overtly political media.</p><blockquote>Commentators often neglect how little political news most people consume — much of the public is not attentive to politics and thus unlikely to be in an echo chamber of any sort.</blockquote><blockquote><em>— </em><strong><em>Andrew Guess et al.</em></strong><em>, </em><a href="https://kf-site-production.s3.amazonaws.com/media_elements/files/000/000/133/original/Topos_KF_White-Paper_Nyhan_V1.pdf"><em>Avoiding the echo chamber about echo chambers</em></a><em> (2018)</em></blockquote><p>If true, media effects overall may be minimal. On the other hand, everything we think is in some sense a response to the information we are exposed to, and changes to that information environment, if large enough, should have significant effects.</p><p>There are at least two ways to resolve this paradox. The first possibility is that the media effects of recommenders may be small for most people most of the time, but occasionally <a href="https://www.ashudeepsingh.com/publications/facctrec2020_singh_et_al.pdf">much larger for a few individuals</a> (<strong>subgroup effects</strong>). Current research on recommenders tends to look at average or aggregated effects, and may not be able to detect these sorts of relatively rare large effects. The second possibility is that changes in our thinking do not usually arise from one-off media exposure but from exposure to recurring narratives and themes over the long term (<strong>cumulative effects</strong>). Both these possibilities can be difficult to observe and need further research attention.</p><h4>Does what is thought affect what is engaged with?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*B0LQFCNBzjY2Ae1DaxIbDg.png" /></figure><p>The research area that studies whether what is thought affects what is engaged with is called <a href="https://www.sciencedirect.com/science/article/abs/pii/S0065260108602129">selective exposure</a>. Generally, it is hypothesized that people tend to engage more with information if it is consistent with their existing thoughts. To assess this claim, we need a way to measure thoughts that is <a href="https://medium.com/understanding-recommenders/what-does-it-mean-to-give-someone-what-they-want-the-nature-of-preferences-in-recommender-systems-82b5a1559157">distinct</a> from the way we measure behavior — we can’t assume that people agree with the content they engage with. The easiest option is to simply ask people what they think. Indeed, studies on selective exposure in the context of recommenders usually <a href="https://keep.lib.asu.edu/_flysystem/fedora/c7/224446/Liu_asu_0010N_19743.pdf">just</a> <a href="https://doi.org/10.1177/0093650212466406">ask</a> <a href="https://doi.org/10.1177/1940161219862988">participants</a> to self-report their political affiliation.</p><p>It is straightforward to observe a <em>correlation</em> between what is thought and what is engaged with, but that does not reveal whether the relationship is <em>causal</em>. The most straightforward way to assess causality would be to conduct an experiment in which you manipulate what people think and observe the impact on engagement. However, manipulating what people think can be both difficult and unethical. For this reason, many of the experiments on selective exposure manipulate other variables (such as whether <a href="https://journals.sagepub.com/doi/10.1177/0093650212466406">like counts are shown</a>) and observe the extent to which this changes the degree of selective exposure observed.</p><p>On average, it <a href="https://sciendo.com/article/10.2478/nor-2021-0002">seems</a> that selective exposure usually has only weak effects.</p><blockquote>[These] tendencies are asymmetric; people tend to prefer pro-attitudinal information to a greater extent than they avoid counter-attitudinal information. Selective exposure can also be overridden by other factors such as social cues. In addition, behavioral data shows that tendencies toward selective exposure do not translate into real-world outcomes as often as public discussion would suggest.</blockquote><blockquote><em>— </em><strong><em>Andrew Guess et al.</em></strong><em>, </em><a href="https://kf-site-production.s3.amazonaws.com/media_elements/files/000/000/133/original/Topos_KF_White-Paper_Nyhan_V1.pdf"><em>Avoiding the echo chamber about echo chambers</em></a><em> (2018)</em></blockquote><p>The story is complicated by the fact that those with highly partisan political views seem to be <a href="https://academic.oup.com/qje/article/126/4/1799/1924154">more likely</a> to engage with content from the opposite side than moderates.</p><blockquote>One study found that people who visited sites with white supremacist content were “twice as likely as visitors to Yahoo! News to visit nytimes.com in the same month”. As Bruns suggests, “they must monitor what they hate”. Meta-analytic findings similarly suggest that the more confident an individual is in their belief, attitude or behavior, the more exposure they have to challenging information.</blockquote><blockquote><em>— </em><strong><em>Peter Dahlgren</em></strong><em>, </em><a href="https://doi.org/10.2478/nor-2021-0002"><em>A Critical Review of Filter Bubbles and a Comparison with Selective Exposure</em></a><em> (2021)</em></blockquote><p>These findings appear to undermine the hypothesis that what is thought has a strong influence over what is engaged with. However, high-level analysis of this sort might obscure what is really happening. For example, there is <a href="https://youtu.be/AYBMSbR2Ndo">some evidence</a> to suggest that when, say, a committed conservative reads the New York Times, they are selectively reading those articles that are relatively consistent with their worldview. Alternatively, there might be framing effects: when a conservative engages with a New York Times article, they may arrive at that URL via a dismissive post from a fellow partisan, which changes how they interpret it. Finally, while selective exposure may be weak on average, it may be stronger for certain subgroups of people. It would be helpful for future research to account for these subtleties.</p><h4>Assembling a feedback loop</h4><p>Based on the above, we think that each of the three causal arrows that make up a feedback loop likely exists in some form, albeit with effect sizes that may be small, inconsistent, and context-dependent. Can we conclude, then, that a feedback loop exists? The way previous work has measured what is shown, what is thought, and what is engaged with varies, so it is difficult to assemble these causal segments into a closed loop.</p><p>The closest work we know of is a <a href="https://doi.org/10.1111/jcom.12108">family</a> <a href="https://doi.org/10.1177/0267323119830056">of</a> <a href="https://doi.org/10.1080/1369118X.2021.1907435">studies</a> that use a framework called “<a href="https://doi.org/10.1111/j.1468-2885.2007.00296.x">reinforcing spirals</a>” to study the interaction between selective exposure and media effects. However, most of this work focuses on traditional media (e.g. TV, radio, newspapers, or specific online news websites) rather than recommender systems, so does not consider the link between what is engaged with and what is shown. It also tends to use only high-level, aggregate measures of what is engaged with (e.g. the number of days in a week that a particular news source was accessed). The best studies on recommender systems we know of, such as <a href="https://dx.doi.org/10.2139/ssrn.4114905">this YouTube one</a>, only consider one or two sides of the loop. One <a href="https://newsflows.eu/">research group</a> led by Damian Trilling at the University of Amsterdam appears to be <a href="https://newsflows.eu/wp-content/uploads/2022/06/epsa2022.pdf">thinking clearly</a> about feedback loops, though they haven’t run any experiments yet. And <a href="https://doi.org/10.1145/3306618.3314288">this study</a> by DeepMind researchers models all sides of the loop, but only in simulation.</p><h3>Where next?</h3><p>If you wanted to collect evidence for a complete feedback loop, you could design a research project something like this:</p><p>Say you suspect that a recommender on a video sharing platform is pushing users toward violent extremism. First, decide on how to measure and quantify three variables: what is engaged with, what is shown, and what is thought. To do this, build a classifier that quantifies the “extremity” of a given video. <strong>What is engaged with</strong> might then be the observed probability that a user watches at least 2 minutes of a video conditioned on the fact that it was offered to them, its position in the feed, and its extremity. <strong>What is shown</strong> could be the average extremity of videos recommended, and <strong>what is thought</strong> could be measured by a survey, such as the <a href="http://dx.doi.org/10.1177/13634615211000550">Sympathy for Violent Radicalization Scale</a>.</p><p>Next, conduct three separate experiments, with real platform users, to assess whether these variables are causally related. To assess whether <strong>what is engaged with</strong> affects <strong>what is shown</strong>, artificially increase the engagement with extreme content above users’ true interaction patterns, and observe how this changes their recommendations. This could be done, for example, using a browser extension that engages with content in the background. To assess whether <strong>what is shown</strong> affects <strong>what is thought</strong>, artificially add videos to their feed that are more or less extreme than those they would have otherwise seen, and observe how this changes their sympathy for violent radicalization between pre- and post-surveys. (This raises some complex ethical issues; perhaps the videos shown in the experiment should only be less extreme than the status quo recommender output, never more.) Finally, to assess whether <strong>what is thought</strong> affects <strong>what is engaged with</strong>, monitor engagement for those same users, who will have been experimentally “de-extremized.” All these experiments should be run over a substantial period of time (perhaps 6 months) and with enough power to detect subgroup and cumulative effects.</p><p>This method, while not intended to be a gold standard, gives one concrete example of the kind of research that is needed. To demonstrate a complete feedback loop, we need to measure three variables consistently, and experimentally test the three relationships between them. A single study that takes this approach might only demonstrate a feedback loop of a particular sort in a particular context. But through a series of such studies, we might be able to develop a sense of the overall prevalence of human-machine feedback loops in recommender systems.</p><p><em>Luke Thorburn was supported in part by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (safeandtrustedai.org), King’s College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d1c8733d5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/when-you-hear-filter-bubble-echo-chamber-or-rabbit-hole-think-feedback-loop-7d1c8733d5c">When You Hear “Filter Bubble”, “Echo Chamber”, or “Rabbit Hole” — Think “Feedback Loop”</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Optimizing for Engagement Changing Us?]]></title>
            <link>https://medium.com/understanding-recommenders/is-optimizing-for-engagement-changing-us-9d0ddfb0c65e?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/9d0ddfb0c65e</guid>
            <category><![CDATA[recommender-systems]]></category>
            <dc:creator><![CDATA[Luke Thorburn]]></dc:creator>
            <pubDate>Mon, 10 Oct 2022 08:02:45 GMT</pubDate>
            <atom:updated>2022-11-23T17:50:34.175Z</atom:updated>
            <content:encoded><![CDATA[<p>— <a href="https://twitter.com/LukeThorburn_">Luke Thorburn</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CuK3h964lmGDfU_sdzU6gA.png" /></figure><p>Of the many <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">potential harms</a> of recommendation algorithms, perhaps the most unsettling is the idea that they might be changing us — our opinions, preferences, beliefs, and behaviors — in ways we don’t understand. Our concern here is not marketing or propaganda, but the strange ways that automated systems can affect us in their algorithmic pursuit of maximization. For example, many are familiar with the idea of a social media “<a href="https://dx.doi.org/10.2139/ssrn.4114905">rabbit hole</a>”, in which the algorithm nudges users towards believing content that is increasingly extreme. One explanation for such rabbit holes is that users with radical beliefs are more likely to engage, and because the recommender has been coded to <a href="https://medium.com/understanding-recommenders/whats-right-and-what-s-wrong-with-optimizing-for-engagement-5abaac021851">optimize for engagement</a>, it tries to change the users so that they become more radical.</p><p>To date, such influence has been shown to <a href="https://arxiv.org/pdf/2009.09153.pdf">occur</a> <a href="https://arxiv.org/pdf/2109.04083.pdf">in</a> <a href="https://doi.org/10.1145/3460231.3478849">simulations</a>. At a high level, these simulations compare how preferences in a population evolve when people are exposed to a recommender system, with a baseline preference trajectory that occurs “naturally”, without the possibility of influence by the recommender. Under some assumptions, these studies have demonstrated that certain types of recommenders can significantly influence the ways in which preferences evolve over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0jnZfe0RfpfUiq-wrjnW0w.png" /><figcaption>Examples of simulations that compare baseline preferences in a population over time with the changes in those preferences that occur when the population is exposed to a recommender (adapted from <a href="https://doi.org/10.1145/3460231.3478849">here</a> and <a href="https://doi.org/10.1145/3460231.3478849">here</a>).</figcaption></figure><p>Simplified models suggest that influence is possible, but is it actually happening? The effects of recommenders are <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">hard to measure</a>, and we leave the complex question of whether social media recommenders are changing political preferences to a future post. But there are sources which suggest influence is already taking place. For example the following comment, made by a <a href="https://www.youtube.com/watch?v=HEqQ2_1XRTs">YouTube employee</a> developing reinforcement learning-based recommender systems, seems to describe a type of influence effect.</p><blockquote>We actually do observe this really interesting user learning process as well during our experiments … there is this growing impact of improvement during even just a 2–3 week period. We see this move from a neutral-ish improvement towards a bigger gain, where the user is actually also adapting to this new behavior exhibited by the system.</blockquote><p>So if influence is already happening, how can we avoid it? We can’t simply tell the algorithm to “prevent all changes” — to do so would prevent opportunities for positive change such as education or new hobbies. Nor can we give a list of every type of preference change we want it to avoid. This problem has been well explored in the <a href="https://arxiv.org/abs/1606.06565">context of robotics</a>. Imagine programming a robot to get coffee as fast as possible. You run it, and in its haste it steps on the cat. You add code to tell it not to hurt pets or people, but next time it breaks a vase. You add code penalizing changes in its environment, but then it tries to prevent you from washing the dishes. Analogously, there are countless ways a recommender could influence people in undesirable ways and we cannot articulate them all, nor do we know how to formulate simple general prohibitions.</p><p>In this post we tackle the challenges of defining, detecting, and preventing “manipulation” by increasingly intelligent recommender algorithms. We’ll explain how this accidental influence results from a combination of mis-specified objectives and feedback loops, then dig into the ethical challenge of deciding when influence is good (e.g. “education”) versus bad (e.g. “blackmail”). Finally we summarize the best thinking to date on how we might detect undesirable influence, and how to prevent it from happening.</p><h3>Accidental Influence Through Mis-specified Objectives (and all objectives are mis-specified)</h3><p>Recommenders, like almost all AI systems, operate by maximizing some objective or reward, usually some form of <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">engagement</a>. This works to the extent that it actually aligns with <a href="https://medium.com/understanding-recommenders/what-does-it-mean-to-give-someone-what-they-want-the-nature-of-preferences-in-recommender-systems-82b5a1559157">what users want</a>. But there is a risk that the algorithm might instead learn to change the preferences of users so that they are easier to satisfy. For example, a recommender might learn that outrage reliably produces clicks, and therefore develop strategies to keep people outraged.</p><blockquote>Like any rational entity, the algorithm learns how to modify the state of its environment — in this case, the user’s mind — in order to maximize its own reward.</blockquote><blockquote><em>— </em><strong><em>Stuart Russell</em></strong><em>, Human Compatible (2019)</em></blockquote><p>We’ll use the word “influence” as a value-neutral way to talk about the idea that recommenders might be changing people. This covers a lot of ground: advertisers influence consumers by placing ads in printed newspapers, teachers influence their students through education, and politicians influence citizens by giving speeches and interviews. We think a good definition for talking about influence in recommender systems is:</p><blockquote>The act of changing a person’s context-specific distribution of beliefs or behavior.</blockquote><p>In this definition, <em>behavior</em> is what people do — their “external state” — and <em>beliefs</em> is used as a catch-all for their “internal state” — any aspect of the individual that isn’t expressed externally, including preferences, opinions, emotions and beliefs of fact. We use the phrase “context-specific distribution …” to acknowledge the fact that internal states, such as preferences, can be uncertain, inconsistent and <a href="https://medium.com/understanding-recommenders/what-does-it-mean-to-give-someone-what-they-want-the-nature-of-preferences-in-recommender-systems-82b5a1559157">constructed on the fly</a>.</p><p>There are three criteria which, in the context of recommender systems, define the particular type of “accidental” influence we are concerned about here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o1ah_6m8phZB9c99PxsOjQ.png" /></figure><p>First, we are only talking about influence that is <strong>unintended</strong>. While there’s a lot to say about data-driven persuasion methods such as the <a href="https://www.nature.com/articles/d41586-018-03880-4">psychographic microtargeting</a> used by Cambridge Analytica — both the political implications and the question of their actual effectiveness — in this piece we’re concerned with the influence that is not intended by any of the humans who build or interact with the system. This excludes, for example, claims of platform designers <a href="https://web.archive.org/web/20190418055610/https://www.businessinsider.com/project-veritas-sting-facebook-plans-tackle-targeted-harassment-2019-2">explicitly biasing</a> algorithms in political directions.</p><p>Influence can occur as a <a href="https://arxiv.org/abs/1606.06565">side effect</a> of the recommendation algorithm pursuing its objective, which is to optimize a measure of value (the <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">value model</a>) by showing items to the user. This optimization pressure creates a strong incentive for the recommender system to take any actions or strategies available to it that increase its measure of value, including influencing its users, even if such influence was never intended by the system’s designer.</p><blockquote>We care which <em>means</em> the algorithm used to solve the problem, but we only told it about the <em>ends</em>, so it didn’t know not to “cheat”.</blockquote><blockquote><em>— </em><strong><em>Krueger et al.</em></strong><em>, </em><a href="https://arxiv.org/pdf/2009.09153.pdf"><em>Hidden Incentives for Auto-Induced Distributional Shift</em></a><em> (2020)</em></blockquote><p>Second, we are only talking about influence that requires a <strong>feedback loop</strong> between the recommender system and the user. By this we mean the recommender changes in response to the user, and the user changes in response to the recommender. This interactive loop is what makes recommender influence different from reading a book, and from non-personalized search engines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TZCPNG2VPFC92pekaaYKjw.png" /></figure><p>Some types of recommenders cannot produce these feedback loops. For example, the recommenders for Reddit and <a href="https://www.thefactual.com/how-it-works">The Factual</a> do not change their strategy in response to user behavior. But feedback loops can be found, to varying degrees, in most modern recommenders because they are based on <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">engagement prediction</a>, and their predictions are regularly updated using recent user behavior. Feedback loops may turn out to be a particular problem for a new generation of <a href="https://arxiv.org/abs/2101.06286v1">recommenders based on reinforcement learning</a>. In such systems, an “agent” (the recommender) is constantly learning and adjusting its strategy by observing the effect its actions have on the “environment” (which includes the user).</p><p>Feedback loops can also arise in more subtle ways, such as through A/B tests. Platform designers may expose two groups of users to different versions of a recommender system, and then adopt the version that most improves their measure of value. If they do this repeatedly, then the recommender will be changing in a way that is informed by the effect it has on users. The designers themselves become the means by which the recommender system changes in response to user behavior, completing the feedback loop. This kind of feedback loop is not personalized to each user, and indeed personalization is not required for influence to occur. A news video channel is an example of a “recommender” that is constantly learning and adapting to a population of users, but not personalizing for each member of the audience. All of these techniques can influence the population as a whole, but don’t have a fine-grained ability to influence individual users.</p><p>Concretely, what would it look like for a recommender to try to influence you? It might simply show you content that it wants you to engage with. In general, all of us tend to become more aligned with the content we are exposed to — whether through <a href="https://www.aeaweb.org/articles?id=10.1257/aer.20190390"><em>experience effects</em></a>, <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2285042"><em>anchoring</em></a>, learning new facts, discovering new interests, <a href="https://en.wikipedia.org/wiki/Asch_conformity_experiments">pressure</a> to <a href="https://en.wikipedia.org/wiki/Communal_reinforcement">conform</a>, or the <a href="https://en.wikipedia.org/wiki/Illusory_truth_effect">illusory truth effect</a>. Alternatively, a recommender might show you content you find strongly disagreeable in order to <a href="https://www.pnas.org/doi/abs/10.1073/pnas.1804840115">more firmly entrench your current preferences</a>. It might show you articles suggesting that everyone else is interested in a particular topic, to generate <a href="https://en.wikipedia.org/wiki/Fear_of_missing_out">FOMO</a> and increase the chance you engage. It might show you a lot of conflicting news accounts to generate <a href="https://www.wired.co.uk/article/deepfakes-reality-fiction">reality apathy</a>, then feed you a false conspiracy theory which makes it all make sense. This list is not meant to be exhaustive. There will be many more subtle ways in which a sufficiently capable recommender could influence us, including some that we would not be able to foresee. The above examples simply demonstrate that recommenders could plausibly influence us, if they are able to learn how to do so.</p><p>The third and final criteria is that we are only talking about influence that is <strong>unethical</strong>.</p><h3>When is Influence Bad?</h3><p>It can be tempting to claim that all influence is unethical. At the same time, almost everything we think and do is a response to our environment, and in that sense has been influenced by the information to which we are exposed. Some forms of influence (e.g. education, good advice) are usually positive, but others (e.g. gaslighting, blackmail) are not.</p><p>If we are to prevent recommenders from influencing us in unethical ways, we need to draw a line between ethical and unethical influence that is widely-accepted — to determine our collective “<a href="https://arxiv.org/pdf/2203.10525.pdf">preference-change preferences</a>”. There are at least two resources that may help.</p><p>The first resource is the study of influence, persuasion, manipulation, and related topics, which goes back thousands of years. For example, the <a href="https://plato.stanford.edu/entries/ethics-manipulation/">Stanford Encyclopedia of Philosophy</a> characterizes manipulation in three ways: as the bypassing of reason, as trickery, or as the application of pressure. Literature on <a href="https://www.scu.edu/ethics/focus-areas/journalism-and-media-ethics/">journalism and media ethics</a> highlights particular forms of unethical influence, such as the <a href="https://en.wikipedia.org/wiki/Werther_effect">Werther effect</a>. <a href="https://uk.sagepub.com/en-gb/eur/propaganda-persuasion/book250869">Jowett &amp; O’Donnell</a> discuss the differences between propaganda and persuasion, arguing that persuasion — unlike propaganda — is “interactive and attempts to satisfy the needs of both persuader and persuadee”. This kind of theory can be used as a basis for defining unethical influence in the context of recommender systems.</p><p>The second resource is the pool of existing laws and regulatory frameworks that record our previous judgments about manipulation and influence, and hence set a precedent. In the US, for example, there are Federal Trade Commission rules about <a href="https://www.ftc.gov/news-events/topics/truth-advertising">truth in advertising</a> and product <a href="https://www.ftc.gov/news-events/topics/truth-advertising/advertisement-endorsements">endorsements</a>, Federal Communications Commision rules about <a href="https://www.fcc.gov/general/consumer-deception-and-fraud-0">consumer deception</a>, and public affairs office-holders within the Department of Defense are <a href="https://www.jcs.mil/Portals/36/Documents/Doctrine/pubs/jp3_61.pdf">required to be truthful</a>. Such regulations do not cover all forms of unethical influence, but a practical approach may be to start by identifying a subset of unethical influence, and then refine the definition over time.</p><p>While there is a wealth of previous thinking about unwanted influence, encoding these ideas computationally is a major challenge. More than building each of these existing rules into a system, the challenge is to design systems that can flexibly interpret these principles and extend them to new situations. There is some work <a href="https://arxiv.org/abs/2110.07574">using language models to make simple moral judgements</a>, but we are a long way away from nuanced evaluations of the acceptability of influence.</p><h3>Detecting and Preventing Unwanted Influence</h3><p>Assuming we can clearly define the types of influence we are concerned about, we are faced with two technical questions: how to detect it, and how to prevent it.</p><h4>Detection</h4><p>There are two broad approaches to detecting influence in general. The first is to observe how people’s <em>behavior</em> changes over time and, via a model that connects beliefs to behavior, make causal inferences about whether their beliefs are changing due to exposure to the recommender system. The second is to directly ask people what their beliefs (or preferences, or opinions) are, and again try to infer whether such changes are caused by the recommender.</p><p>Both approaches have their limitations. <a href="https://medium.com/understanding-recommenders/what-does-it-mean-to-give-someone-what-they-want-the-nature-of-preferences-in-recommender-systems-82b5a1559157">Behavior does not always reflect beliefs</a>, and there are multiple incompatible ways of modeling the link between beliefs and behavior. It is not clear which, if any, is correct. Moreover, doing causal inference to determine the effects of recommenders is <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">challenging</a>. The effects of recommender-caused influence <a href="https://doi.org/10.1126/sciadv.abc4046">may be small</a>, or only affect a small number of people in specific circumstances, and thus be hard to detect.</p><p>In practice, it may be more plausible to detect whether a group or population of recommender users is being influenced (such as by becoming more polarized), rather than to detect influence on an individual basis. Alternatively, it may be practical to simply <a href="https://doi.org/10.1145/3460231.3478849">classify certain preference shifts as undesirable</a>, regardless of whether they were caused by the recommender. This removes the challenges associated with causal inference, but means you may take action against changes that were not caused by the recommender, raising other ethical issues.</p><h4>Prevention</h4><p>Say we are clear on the philosophical and ethical questions, have measured the amount of bad influence that is taking place and think we need to reduce it. Or perhaps we want to mitigate the risk of bad influence preemptively. What could we do?</p><p>There is a growing literature on how to prevent or mitigate the effects of influence in recommender systems. Broadly, proposals fall into three groups.</p><p>The first set of proposals are to <strong>avoid giving recommenders the capability to influence</strong>. Concretely, this could mean that recommenders <a href="https://arxiv.org/pdf/2009.09153.pdf">get tested</a> in simulated environments, and only those that don’t demonstrate the ability to influence get deployed. It could mean that recommenders are designed to <a href="https://arxiv.org/pdf/2009.09153.pdf">only observe the short term effects of their actions</a>, which would remove their ability to learn to manipulate preferences over the long term. More simply, it could mean that recommenders are just not used in sensitive contexts, such as when Facebook decided to <a href="https://www.reuters.com/article/us-facebook-groups/facebook-says-it-will-permanently-stop-recommending-political-groups-to-users-idUSKBN29X00C">stop recommending political groups</a> or Twitter <a href="https://twitter.com/jack/status/1189634360472829952">banned political advertising</a>.</p><p>The second set of proposals amount to <strong>making influence more difficult</strong>. This could be achieved by <a href="https://arxiv.org/pdf/1606.06565.pdf">rewarding the recommender for being uncertain</a> about what it is the user actually wants, which could lead it to be more conservative in its recommendation strategies and avoid pursuing strategies if it is uncertain whether the user would approve of them. (For example, this would effectively stop the recommender from reading too much into our actions when we click occasionally on a clickbait headline.) Another approach would be to mandate a certain level of <a href="https://arxiv.org/pdf/2205.13026.pdf">randomness or noise</a> in the recommendations — perhaps by interleaving a sample of randomly selected posts into users’ feeds — which would make any attempt at influence by the recommender less effective.</p><p>The third set of proposals explicitly <strong>penalize influence</strong>, to remove the incentive for recommenders to cause it. Of the three, this is the most difficult direction, because it requires a formal definition of which types of influence are unwanted, and the ability to distinguish between changes caused by the recommender and changes caused by other factors. Moreover, as in the example of the robot barista in the introduction, it may not be possible to articulate every possible form of unwanted influence. A conservative strategy is to define categories of “<a href="https://dl.acm.org/doi/abs/10.1145/3460231.3478849?casa_token=CJ3b7dLZWYoAAAAA:chO34mwIy3MMePYjq6LAr62Kk_xdPaVNPWoxTELek6aacRA3lqS45hOJvY6vksmVClVmkvAGeRbreQ">safe</a>” changes — such as those expected to occur in the absence of the recommender — and to flag all changes that don’t fall into these categories. Alternatively, it may be possible to learn which changes are undesirable in a bottom up fashion, by asking users. For example, the YouTube recommender could learn to avoid <a href="https://foundation.mozilla.org/en/youtube/findings/">regret</a>, as measured by survey responses. Another approach attempts to ensure the recommender has <a href="https://www.aaai.org/AAAI22Papers/AAAI-1140.FarquharS.pdf">no incentive to increase its reward by influencing the user</a>.</p><h3>Conclusion</h3><p>Inadvertent influence may not just be an issue for recommender systems on large platforms with the potential for <a href="https://arxiv.org/pdf/2203.17232.pdf">steering a population</a>, but also for small platforms if sufficiently harmful to a small number of users. Further, the strange effects of strong optimization may increasingly become an issue as more of the content we are exposed to becomes personalized, and perhaps synthetically generated to maximize an objective (à la <a href="https://openai.com/api/">GPT-3</a> or <a href="https://openai.com/dall-e-2/">DALL-E 2</a>). To retain human agency, we need to be able to define, detect, and prevent such influence from happening. These are challenging problems and all require significant research attention, but there are promising precedents and directions on which we can build.</p><p><em>Thank you to David Krueger for feedback on this post.</em></p><p><em>Luke Thorburn was supported in part by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (safeandtrustedai.org), King’s College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d0ddfb0c65e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/is-optimizing-for-engagement-changing-us-9d0ddfb0c65e">Is Optimizing for Engagement Changing Us?</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Menu of Recommender Transparency Options]]></title>
            <link>https://medium.com/understanding-recommenders/a-menu-of-recommender-transparency-options-7b1379f12dda?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/7b1379f12dda</guid>
            <category><![CDATA[recommender-systems]]></category>
            <dc:creator><![CDATA[Priyanjana Bengani]]></dc:creator>
            <pubDate>Wed, 17 Aug 2022 10:51:17 GMT</pubDate>
            <atom:updated>2022-08-17T10:51:17.764Z</atom:updated>
            <content:encoded><![CDATA[<p>— <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://lukethorburn.com/">Luke Thorburn</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KNoao3MgswRhIwyCH5gBmA.png" /></figure><p>Recommender systems are ubiquitous, yet there are major unanswered questions about their operation and impact. The users of recommender-based platforms often don’t know why they’re seeing certain items, the public and policy-makers don’t understand how feeds are produced, and the extent to which the “algorithms” can be blamed for societal problems (from <a href="https://www.nature.com/articles/s41467-022-29296-3">teen depression</a> to <a href="https://arxiv.org/abs/1908.08313">radicalization</a> to <a href="https://www.ohchr.org/sites/default/files/Documents/HRBodies/HRCouncil/FFM-Myanmar/A_HRC_39_CRP.2.pdf">genocide</a>) remains unclear. Platforms have adopted a host of transparency measures, from high-level statistics to <a href="https://developers.google.com/youtube/v3/">APIs</a>, <a href="https://www.crowdtangle.com/">tools</a>, and privacy-protected <a href="https://transparency.twitter.com/en/reports/information-operations.html">data sets</a>, but these remain <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">insufficient</a> to understand content selection algorithms and their actual influence on society.</p><p>Platforms have pushed back on calls for additional transparency measures citing user privacy concerns, especially in the aftermath of the <a href="https://www.wired.com/story/cambridge-analytica-facebook-privacy-awakening/">Cambridge Analytica scandal</a>. There are other legitimate reasons to keep things secret, as certain disclosures might <a href="https://journals.sagepub.com/doi/full/10.1177/2053951715622512">expose trade secrets</a> or <a href="https://www.judiciary.senate.gov/imo/media/doc/Persily%20Testimony.pdf">compromise system security</a>. Yet much of the resistance to transparency seems to be about avoiding embarrassment. Executives at Meta have <a href="https://www.nytimes.com/2021/07/14/technology/facebook-data.html">voiced concerns</a> that they were “giving away too much [information],” allowing users to dig up data that didn’t reflect well on the company.</p><p>This has led to a lively policy discussion around platform transparency, and a variety of proposed legislation that takes <a href="https://cdt.org/insights/overview-of-transparency-obligations-for-digital-services-in-the-dsa/">different</a> <a href="https://law.stanford.edu/press/the-platform-transparency-and-accountability-act-new-legislation-addresses-platform-data-secrecy/">approaches</a> to compelling disclosure for recommender systems. But disclosure of what? To ground this discussion in specifics, we offer a menu of things that recommender “transparency” could mean. This includes UI affordances for individual users, documentation detailing system behavior, anonymized data sets, and research collaborations. We broadly categorize these options into documentation, data, code, and research. Each type of disclosure has its advantages and drawbacks, and each might serve particular purposes for particular groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xL5jRGZZG_SOriZ9_Yw36w.png" /><figcaption><a href="https://docs.google.com/spreadsheets/d/1Um3w7knhRA-lJKRd2qTe99E3xcCsyvQOCGopw5jsUhk/edit?usp=sharing">Click here</a> for a more comprehensive version of this table, with<br>descriptions and examples of each transparency option.</figcaption></figure><h3>Documentation</h3><p>Documentation is the simplest form of transparency, where the platform relies on natural language to explain what’s going on to its users and other interested parties. We break this down into user-specific documentation and system-level documentation.</p><h3>User-specific documentation</h3><p>Algorithmic feeds are highly dependent on individual user activity and documentation generated for each user can explain why they’re seeing what they’re seeing. Real-world examples include Facebook’s “<a href="https://www.cnbc.com/2019/04/01/facebook-new-tool-explains-why-am-i-seeing-this-post-on-news-feed.html">Why am I seeing this post?</a>” and Amazon’s “Frequently bought together” panel recommending other items to buy. Explainability and transparency are not interchangeable, but transparency is <a href="https://dl.acm.org/doi/pdf/10.1145/3331184.3331211">one of the aims of explainability</a>, and transparency measures “should give an honest account of how the recommendations are selected and how the system works.”</p><p>This “honest account” should extend to informing the users about the <a href="https://anupamdas.org/paper/CSCW2022.pdf">interests</a><strong> </strong>(e.g. skiing, black-and-white movies, or bird-watching) and <a href="https://www.theguardian.com/technology/2016/mar/22/facebooks-ethnic-affinity-advertising-concerns-racial-profiling">attributes</a> (e.g. race, age, or sexual orientation) that the platform may have <a href="https://www.tandfonline.com/doi/abs/10.1080/1369118X.2019.1642934">inferred</a> about them, including how the system made these inferences. What specific actions taken by the user led to a given inference? If user behavior was <a href="https://lig-membres.imag.fr/gogao/papers/fb_ad_transparency_NDSS2018.pdf">combined with off-platform activity</a>, how was the data obtained? This level of disclosure would also help address both the “<a href="http://www.cs.cmu.edu/~byl/publications/lim_chi09.pdf">Why</a>” questions users are likely to ask when they don’t understand the reasons a specific post is shown (e.g. the case of a user who <a href="https://www.buzzfeednews.com/article/katieheaney/facebook-knew-i-was-gay-before-my-family-did">saw an ad on his feed</a> asking, “COMING OUT? NEED HELP?”), and the “<a href="http://www.cs.cmu.edu/~byl/publications/lim_chi09.pdf">Why Not</a>” questions when they expect certain results that the recommender system does not produce. Creating additional UI affordances to let users rectify incorrect or unwanted inferences can also ensure users have more control over their online experience.</p><h3>System-level Documentation</h3><p>Beyond individual users, ranking decisions applicable to the entire recommender system should be documented. Such documentation could be modeled on existing transparency reports that provide high-level information on requests from law enforcement and government, the number of content takedowns for different reasons, and various security and integrity initiatives.</p><p>These are important numbers, but they do not provide any information about algorithm design. Additional documentation could include the types of content the platform <a href="https://newsroom.fb.com/news/2018/01/news-feed-fyi-bringing-people-closer-together">prioritizes</a> and <a href="https://transparency.fb.com/en-gb/features/approach-to-ranking/types-of-content-we-demote/">downranks</a>, and whether it is operating under heightened “break-glass” conditions due to external factors like <a href="https://www.nbcnews.com/tech/tech-news/facebook-says-it-could-restrict-some-content-if-election-leads-n1240722">elections</a> or <a href="https://www.buzzfeednews.com/article/ryanmac/mark-zuckerberg-joel-kaplan-facebook-alex-jones">other</a><a href="https://www.theatlantic.com/ideas/archive/2021/04/facebook-should-dial-down-toxicity-much-more-often/618653/"> events</a> likely to increase civic unrest. Customers should also be informed if a platform is prioritizing its <a href="https://themarkup.org/amazons-advantage/2021/10/14/amazon-puts-its-own-brands-first-above-better-rated-products">own</a><a href="https://themarkup.org/google-the-giant/2020/07/28/google-search-results-prioritize-google-products-over-competitors"> brands</a> or <a href="https://www.accc.gov.au/media-release/trivago-misled-consumers-about-hotel-room-rates">partners</a>. Advocacy groups have suggested publicly <a href="https://www.newamerica.org/oti/reports/why-am-i-seeing-this/promoting-fairness-accountability-and-transparency-around-algorithmic-recommendation-practices">logging all major changes</a> made to the system, including technical adjustments, new features, and changes in policy.</p><p>Documentation does not necessarily allow for independent oversight. Further, it will remain difficult to understand how recommenders and their effects differ across regions or demographic groups. Providing deeper access to data is one way to address these deficiencies.</p><h3>Data</h3><p>To understand what is happening on a platform — including what a recommender algorithm is doing — it is necessary to have data on what users are seeing and how they are responding. Twitter regularly releases <a href="https://transparency.twitter.com/en/reports/information-operations.html">datasets</a> containing tweets and account details related to state-linked information operations, while Facebook has <a href="https://research.facebook.com/blog/2020/02/new-privacy-protected-facebook-data-for-independent-research-on-social-medias-impact-on-democracy/">released</a> a <a href="https://socialscience.one/blog/unprecedented-facebook-urls-dataset-now-available-research-through-social-science-one">large dataset of URLs</a> in partnership with the Social Science One academic consortium. (However, this data set was plagued with <a href="https://www.fastcompany.com/90412518/facebooks-plan-for-radical-transparency-was-too-radical">delays</a> and ended up <a href="https://www.washingtonpost.com/technology/2021/09/10/facebook-error-data-social-scientists/">missing data from roughly half of Facebook’s US users</a>.)</p><p>Another approach, application programming interfaces (APIs), provide ongoing access to platform data. This includes tools like Facebook’s <a href="http://crowdtangle.com/">CrowdTangle</a> (which allows approved users and institutions to access data about public Facebook and Instagram accounts) and <a href="https://developer.twitter.com/en/docs/twitter-api">Twitter’s Developer API</a> (which gives approved users programmatic access to public tweets, among other things). Platforms can provide interfaces for journalists, researchers, academics and other vetted stakeholders to access additional data.</p><p>Researchers have frequently called for access to platform data that is not currently public. Some <a href="https://misinforeview.hks.harvard.edu/article/tackling-misinformation-what-researchers-could-do-with-social-media-data/">common categories</a> of data requested include reliable access to historical data (e.g. what items have trended on Twitter or Reddit historically?), impressions alongside engagements (i.e. how many people saw a piece of content irrespective of whether they interacted with it), and access to longitudinal data that allows user behavior to be (anonymously) traced over time.</p><p>One of the major challenges to wider data disclosures is the tension between <a href="https://medium.com/stanfordcyber/user-privacy-vs-platform-transparency-the-conflicts-are-real-and-we-need-to-talk-about-them-d9dc6cff4a7a">transparency and privacy</a>. From 2010 to 2015, before the Cambridge Analytica scandal, Facebook’s Graph API effectively provided an <a href="https://medium.com/tow-center/the-graph-api-key-points-in-the-facebook-and-cambridge-analytica-debacle-b69fe692d747">all-access pass</a> where developers didn’t require consent from a user’s friends to collect their profile data. Since then, APIs have become more stringent. In <a href="https://blog.twitter.com/developer/en_us/topics/tools/2018/new-developer-requirements-to-protect-our-platform">2018</a> Twitter tightened their developer application process to include “case reviews and <a href="https://developer.twitter.com/en/developer-terms/agreement-and-policy">policy</a> compliance checks.” These sorts of restrictions provide important privacy protections, but privacy can also be used as an excuse to avoid disclosure.</p><p>There are numerous strategies that help meet privacy obligations while releasing large datasets, including <a href="https://desfontain.es/privacy/friendly-intro-to-differential-privacy.html">differential privacy</a> which strategically adds noise to aggregated data to prevent the re-identification of individuals. However, there are <a href="https://edmo.eu/wp-content/uploads/2022/02/Report-of-the-European-Digital-Media-Observatorys-Working-Group-on-Platform-to-Researcher-Data-Access-2022.pdf">concerns</a> that this intentional randomness may make it difficult to replicate prior research. Typically, researchers do not need to identify individual behavior, and even trusted parties probably shouldn’t have access to data about individual people. At the same time, there is a need for information about aggregate behavior in different contexts, say for different subgroups of people in different geographies. This kind of data should allow outsiders to determine which areas of the platform to focus on, <a href="http://proceedings.mlr.press/v81/ekstrand18b/ekstrand18b.pdf">consider</a> who is benefitting from the system, and assess which user groups are underserved.</p><p>Despite the potential benefits of increased transparency, in many ways platforms have been releasing less data, not more. In the last two years, Facebook has <a href="https://www.bloomberg.com/news/articles/2022-06-23/meta-pulls-support-for-tool-used-to-keep-misinformation-in-check">hamstrung CrowdTangle’s development</a> with an eye to scrapping the project entirely and moving towards even more selective disclosures. Researchers and journalists have been forced to develop their own transparency tools, such as <a href="https://themarkup.org/newsletter/citizen-browser/launching-citizen-browser">browser extensions</a> that allow users to donate their data, or <a href="https://www.wsj.com/story/tiktok-floods-teens-with-eating-disorder-videos-b20c2c73">bot-based simulations</a> that attempt to reverse-engineer algorithmic behavior. Platforms can <a href="https://themarkup.org/citizen-browser/2021/09/21/facebook-rolls-out-news-feed-change-that-blocks-watchdogs-from-gathering-data">block these third-party tools</a> or claim that they <a href="https://www.protocol.com/nyu-facebook-researchers-scraping">violate their policies</a> and shut them down, which leaves the public worse off. Companies have also <a href="https://themarkup.org/citizen-browser/2021/03/04/official-information-about-covid-19-is-reaching-fewer-black-people-on-facebook">dismissed</a><a href="https://onezero.medium.com/the-battle-over-facebooks-top-10-list-dc3fca3d799"> findings</a> by arguing that outside researchers relied on incomplete datasets — which is often true, and is exactly the problem that greater transparency aims to solve.</p><h3>Code</h3><p>Open-sourcing recommender code could be useful, <a href="https://www.washingtonpost.com/technology/2022/04/16/elon-musk-twitter-algorithm/">but not in isolation</a>. Real recommender <a href="https://stratechery.com/2022/back-to-the-future-of-twitter/">systems are complex</a>, and there isn’t a single algorithm that underpins the whole system. Rather, different machine learning models trained on different data sets perform a variety of functions within the system. Even the core ranking code (perhaps the <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">value model</a> that is used to compute a score for each item) will not explain very much without detailed data on actual user behavior on the platform, because recommender outcomes are always a <a href="https://www.commerce.senate.gov/services/files/62102355-DC26-4909-BF90-8FB068145F18">combination of human and machine behavior</a>. For example, a high weight on content of a particular type doesn’t necessarily mean a user will see a lot of it, as exposure depends on many other factors including how much of that type of content is produced by other users. Open-sourcing the code might also help bad actors figure out how they can better manipulate the system.</p><p>Few platforms offer much in the way of source code transparency. Reddit was open-sourced back in 2008 when it first launched, but eventually <a href="https://www.reddit.com/r/changelog/comments/6xfyfg/an_update_on_the_state_of_the_redditreddit_and/">stopped updating</a> its public repository. Twitter’s Birdwatch, a crowd-sourced fact checking initiative, has <a href="https://twitter.github.io/birdwatch/ranking-notes/">open-sourced its “note ranking” code</a>, but this is a minor, experimental initiative. In the absence of such transparency measures, outsiders must resort to combing through blog posts or public academic papers, which may provide high-level descriptions of algorithms.</p><h3>Research</h3><p>Platforms are constantly running a slew of internal research projects and testing different approaches to their recommendation systems. The results of such experiments are not typically announced in press releases, but sometimes do appear in papers submitted to conferences (or leaks, as in this <a href="https://s.wsj.net/public/resources/documents/social-comparison-topics-celebrities-like-counts-selfies.pdf">100,000 user Instagram study of social comparison</a>). These studies can provide greater insight into what a <a href="https://blog.youtube/inside-youtube/on-youtubes-recommendation-system/">platform is working on</a>, the <a href="https://s.wwsj.net/public/resources/documents/social-comparison-topics-celebrities-like-counts-selfies.pdf">psychological effects</a> of platform use, <a href="https://arxiv.org/abs/2110.11010">unintended bias</a> observed in content ranking and <a href="https://dl.acm.org/doi/10.1145/3292500.3330745">mitigation methods</a> to reduce it. In the absence of publicly-available data, it’s impossible to verify or replicate these findings.</p><p>More recently, there have been conscious efforts in building collaborative efforts between industry and academia. A <a href="https://blog.openmined.org/announcing-our-partnership-with-twitter-to-advance-algorithmic-transparency/">Twitter partnership with OpenMined</a> aims to advance algorithmic transparency by releasing privacy-protected datasets that would allow anyone to replicate certain internal research. Another example is <a href="https://medium.com/@2020_election_research_project/a-proposal-for-understanding-social-medias-impact-on-elections-4ca5b7aae10">Facebook’s partnership with academics</a> to better understand discourse during the 2020 election. These projects are still nascent, and the extent to which they promote real transparency is yet to be seen. In principle, these sorts of collaborations could go well beyond data access by providing external researchers the ability to design and conduct experiments on a platform. This is essential because many of the more pressing questions are causal in nature and <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">cannot be answered by data alone</a>.</p><p>Finally, audits can be considered a particular kind of structured “research” conducted by outside parties. At the moment there are no good examples of completed recommender system audits (Facebook recently released an <a href="https://about.fb.com/news/2022/05/community-standards-enforcement-report-assessment-results/">independent assessment</a> of its community standards enforcement, but this is not a recommender audit). However, the proposed text of the recently-passed <a href="https://eur-lex.europa.eu/legal-content/en/TXT/?uri=COM:2020:825:FIN">Digital Services Act</a> requires large platforms to allow external audits of “content moderation systems, recommender systems and online advertising.” Audits could be initiated by the company themselves or as part of <a href="https://cdt.org/insights/overview-of-transparency-obligations-for-digital-services-in-the-dsa/">regulatory oversight</a>, and auditors would have more <a href="https://social.cs.uiuc.edu/papers/pdfs/ICA2014-Sandvig.pdf">extensive access</a> than outside researchers and journalists. This could include access to internal code and configurations, <a href="https://edmo.eu/wp-content/uploads/2022/02/Report-of-the-European-Digital-Media-Observatorys-Working-Group-on-Platform-to-Researcher-Data-Access-2022.pdf">clean room access</a> to user data, and separate APIs that allow monitoring system behavior over time.</p><p>There are many possible goals of recommender system audits. The Digital Services Act requires audits to assess the “protection of public health, minors, civic discourse, or actual or foreseeable effects related to electoral processes and public security” but does not say how this is to be accomplished or what standards to apply. There are more detailed proposals for auditing <a href="https://social.cs.uiuc.edu/papers/pdfs/ICA2014-Sandvig.pdf">discrimination</a>, <a href="https://dl.acm.org/doi/abs/10.1145/3460231.3478851">polarization</a>, and <a href="https://blog.twitter.com/en_us/topics/company/2021/rml-politicalcontent">amplification</a> in recommender systems.</p><p>In principle, one could assess any of the potential harms discussed above, though many harms would <a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">require experiments</a> to assess, stretching the meaning of “audit.” More broadly, audits have the potential to verify whether the design and operation of a recommendation system adheres to best practices and does what a company says it does. Performed well, audits could provide most of the benefits of open-source code and user data access, without requiring either to be made public. However, legally required audits are unlikely to address all areas of concern to external stakeholders.</p><h3>A Range of Options</h3><p>Researchers, policy-makers, and journalists have long implored platform companies to be more transparent, including around the core algorithms used to select and personalize content. In the absence of substantive improvements, regulatory authorities globally are putting forward legislative proposals with broadly-worded requirements for algorithmic transparency. Yet it’s not obvious what “transparency” actually means for such large and complex systems where outcomes are the result of interactions between algorithms and people. Four categories of transparency measures — documentation, data, code, and research — can each provide unique insights into what the algorithms we interact with every day are actually doing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b1379f12dda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/a-menu-of-recommender-transparency-options-7b1379f12dda">A Menu of Recommender Transparency Options</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Measure the Causal Effects of Recommenders]]></title>
            <link>https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/5e89b7363d57</guid>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Luke Thorburn]]></dc:creator>
            <pubDate>Wed, 20 Jul 2022 10:23:14 GMT</pubDate>
            <atom:updated>2022-11-23T17:50:25.240Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Measure the Effects of Recommenders</h3><p>— <a href="https://lukethorburn.com/">Luke Thorburn</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vbcrv2NvbeLeI0PxiRnoXw.png" /></figure><p>Social media platforms and their recommender systems have been claimed to increase <a href="https://bhr.stern.nyu.edu/polarization-report-page">political polarization</a>, reduce <a href="https://www.theguardian.com/science/2022/jan/02/attention-span-focus-screens-apps-smartphones-social-media">attention spans</a>, worsen <a href="https://www.theatlantic.com/magazine/archive/2017/09/has-the-smartphone-destroyed-a-generation/534198/">mental health</a>, and to have many <a href="https://doi.org/10.4018/JGIM.20211101.oa34">other effects</a>. Often there are compelling stories, anecdotes, or correlations that support these claims. For example, The Wall Street Journal recently <a href="https://archive.ph/H3S2j">published</a> quotes from several teenage girls who all attribute eating disorders to content recommendations they received on TikTok. It’s hard to deny such personal testimony, but what is the scale and impact of this problem? Evidence that would support wider, more generalizable claims of causality is very hard to come by. For most of the harms above, the extent to which they are caused by algorithmic effects remains a difficult open question.</p><p>In part, this ongoing uncertainty is due to the limited access that researchers have to platform data and experimentation. This constrains the types of studies that can be conducted and, hence, the quality of evidence available. Consequently the findings that do get published are usually non-causal, and can be inconclusive, inconsistent, or in the worst cases, simply false.</p><p>The risk of low-quality evidence is that we waste time and effort worrying about non-issues, at the expense of those that are real. This is not an abstract worry. Concerns about the effects of screen time on youth, based largely on observational studies, are now thought to have been <a href="https://themarkup.org/newsletter/hello-world/rethinking-the-effects-of-screen-time-on-youth">largely</a> <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7277381/">overblown</a>. Concerns that access to search engines causes our memories to deteriorate were prompted by a <a href="https://doi.org/10.1126/science.1207745">single 2011 paper</a> studying 28 Ivy League undergraduate students that later <a href="https://doi.org/10.1038/s41562-018-0399-z">failed to replicate</a>. Concerns about a “backfire effect” — in which fact checks cause people to believe false information more strongly — were prompted by a <a href="https://doi.org/10.1007/s11109-010-9112-2">single 2010 study</a> that used mock newspaper articles and had under 200 participants. Subsequent studies have <a href="https://doi.org/10.1016%2Fj.jarmac.2020.06.006">failed to observe</a> the effect, even under “<a href="https://doi.org/10.1016%2Fj.jarmac.2020.06.006">theoretically favorable conditions</a>.” There have been similar panics that TV causes aggression or violent video games lead to real world crime, but these concerns are generally <a href="https://behavioralscientist.org/history-panic-entertainment-technology/">not supported</a>.</p><blockquote>There are genuine questions that are really important, but there’s a kind of opportunity cost that is missed here. There’s so much focus on sweeping claims that aren’t actionable, or unfounded claims we can contradict with data, that are crowding out the harms we can demonstrate, and the things we can test, that could make social media better. … We’re years into this, and we’re still having an uninformed conversation about social media.</blockquote><blockquote><em>— </em><strong><em>Brendan Nyhan</em></strong><em>, quoted in </em><a href="https://www.newyorker.com/culture/annals-of-inquiry/we-know-less-about-social-media-than-we-think"><em>How Harmful Is Social Media?</em></a><em> (2022)</em></blockquote><p>But we know how to collect better evidence. In this post we step through the methodological options available to researchers at three levels: (1) without access to platforms, (2) with access to platform data, and (3) with the ability to conduct on-platform experiments. For each level, we describe the types of studies available to researchers, and the limits of what they can teach us. Ultimately, we argue that researchers need to be able to conduct experiments natively on social media platforms if we are to understand the causal effects of recommender systems on society.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*0Vy-SkOq_g_4_d8zYs4sgA.png" /><figcaption>A summary of the study types available to researchers at different levels of platform access, along with their major limitations.</figcaption></figure><h3>Without access</h3><p>Without access to platforms, researchers have two broad study types available to them: simulations and off-platform experiments.</p><p>In <strong>simulation studies,</strong> researchers assume a formal model of user behavior and use that to explore what outcomes we can expect across multiple counterfactual scenarios. Simulations are frequently used to investigate the social dynamics of <a href="https://ojs.aaai.org/index.php/ICWSM/article/view/19275">polarization</a> and <a href="https://doi.org/10.1111/bjso.12286">filter bubbles</a>, as well as phenomena like <a href="https://doi.org/10.1016/j.physa.2015.10.026">information cascades</a> and the effectiveness of <a href="http://workshop-proceedings.icwsm.org/pdf/2021_63.pdf">platform interventions</a> such as user bans or “<a href="https://doi.org/10.1080/10463283.2021.1876983">innoculation</a>” against misinformation. By using simulations, researchers can have perfect control over which scenarios they are comparing, and can compare variables (such as the true internal opinions of each person) that would not be measurable in the real world.</p><p>The other option available to researchers without platform access is <strong>off-platform experiments</strong>. One variant is a <em>deprivation study</em>, where participants are incentivized not to use social media. A good example is <a href="https://www.aeaweb.org/articles?id=10.1257/aer.20190658">this experiment</a> in which participants were paid to stop using Facebook for four weeks in the lead up to the 2018 US midterm election, and were found to have reduced factual news knowledge but also be less polarized and feel better. Other strategies that don’t require access to the platform include asking people to change their behavior, such as by <a href="https://dx.doi.org/10.2139/ssrn.3653388">subscribing to a news source from the other side of politics</a>, or building <em>toy recommenders</em> — artificial, <a href="https://themisinformationgame.github.io/">self-contained social media platforms</a> in which to conduct research. One of the more ambitious projects of this sort aims to <a href="http://ceur-ws.org/Vol-3143/paper4.pdf">build an actual news delivery app</a> to provide researchers with a research environment.</p><h4>Limited Ecological Validity</h4><p>Every simulation study, and many off-platform experiments, lack <em>ecological validity</em> — their findings may not reflect the real world. All simulations depend on simple models of human behavior, and we cannot be certain that these models accurately reflect what real people would do. At most, simulations can tell us what <em>might be</em>, not what <em>is</em>. Similarly, in most cases off-platform experiments either aren’t conducted in a platform-like environment, or artificial environments that differ in crucial ways from real-world social media platforms. One <a href="https://doi.org/10.1037/tmb0000005">review</a> found that</p><blockquote>…only a small proportion of the studies … about 18.1% … attempted to incorporate (parts of) the experience of social media use in their studies. In some of these cases, study design elements were less than ideal from an ecological validity point of view. For example, in some studies, participants were overtly restricted in the ways they could behave, for instance, by prohibiting participants to share posts on Facebook … Other studies clearly manipulated expectations, for instance, by telling participants to expect comments on their posts from coparticipants (thus rendering the fact that participants felt bad when these comments remained absent not particularly surprising)</blockquote><p>To generate findings that have a stronger case for ecological validity, researchers need access to platform data.</p><h3>With platform data</h3><p>With access to platform data researchers can do all of the above, plus conduct observational studies.</p><p><strong>Observational studies</strong> involve statistical analysis of real world platform data, but do not involve any intervention or experimentation on the part of the researchers. If the relevant data is accessible, observational studies can answer descriptive questions like “How common are diet-related posts on Instagram?” or “Are conspiracy videos on YouTube becoming more popular?”. They can also reveal correlations, such as that certain aspects of Instagram use <a href="https://doi.org/10.1016/j.chbr.2021.100121">correlate</a> with poor mental health, or that the number of moral-emotional words in Tweets <a href="https://doi.org/10.31219/osf.io/s4w2x">correlates</a> with how far they spread (the so-called “‘moral contagion” phenomenon).</p><p>Correlation is not causation though, because of the risk of <em>confounders</em> — variables that affect both the “treatment” and the “outcome”. Does social media use cause depression, or does depression cause social media use, or does some third confounding variable (say, other life events) cause both? Correlative studies cannot tell us if recommender systems are causing harm, or merely reflecting harm that would have existed anyway. Further, there is a risk of spurious correlations, as demonstrated in the debate over the <a href="https://osf.io/s4w2x">moral contagion</a> phenomenon: one analysis suggested that the number of moral-emotional words in Tweets was no more predictive of spread than the number of <a href="https://doi.org/10.1038/s41562-021-01133-5">Xs, Ys and Zs</a>.</p><p>In some cases, it <em>is</em> possible to detect causal relationships in observational datasets, even though no experiment has been conducted. For example, say we are interested in the effect that a change in the recommender algorithm had on depression rates. We would like to estimate this by looking at the difference in depression rates among users immediately before and after the new algorithm was introduced, but something else could have happened at the same time — perhaps some depressing news event. To account for this, we can compare the change in depression rates for folks who used the product versus those who didn’t. This is called the <a href="https://mixtape.scunning.com/09-difference_in_differences">differences-in-differences</a> method.</p><p>Other methods for learning from such “natural experiments” include <a href="https://mixtape.scunning.com/06-regression_discontinuity">regression discontinuity</a> methods and <a href="https://mixtape.scunning.com/07-instrumental_variables">instrumental variable</a> designs. If all possible confounders are known and measurable, then “controlling” for them by using a <a href="https://mixtape.scunning.com/05-matching_and_subclassification#subclassification">subclassification</a>, <a href="https://mixtape.scunning.com/05-matching_and_subclassification#exact-matching">matching</a>, or <a href="https://mixtape.scunning.com/05-matching_and_subclassification#propensity-score-methods">propensity score</a> method, or by including them as covariates in a model, can be sufficient to identify causal effects. If some confounders are unknown or unmeasurable — usually the case — then it may be possible to create a simulated control group (a <a href="https://mixtape.scunning.com/10-synthetic_control">synthetic control</a>), to estimate causal effects from longitudinal data (specifically, <a href="https://mixtape.scunning.com/08-panel_data">panel data</a>). <a href="https://osf.io/preprints/socarxiv/p3z9v/">Several studies</a> have investigated the effects of digital media using these techniques.</p><h4>Limited Data Scope &amp; Quality</h4><p>In practice, the data available for observational studies differs considerably depending on the platform. Some data just can’t be accessed, some data is only accessible ephemerally, and some data is — out of necessity — self-reported rather than objectively measured. For example, one <a href="https://doi.org/10.1037/tmb0000005">review</a> found 81.9% of studies into social media use relied on self-reported data. But we know that self-reported use of social media generally does not <a href="https://doi.org/10.1111/jcc4.12021">reflect</a> <a href="https://doi.org/10.1016/j.ijhcs.2019.05.004">actual</a> <a href="https://doi.org/10.1016/j.chb.2019.03.006">use</a>. Some self reporting methods are better than others: three that have recently been <a href="https://doi.org/10.1037/tmb0000005">advocated</a> for are <a href="https://www.jstor.org/stable/27795429">stimulated recall</a>, <a href="https://doi.org/10.1097/00005053-198709000-00004">experience sampling</a>, and <a href="https://oxford.universitypressscholarship.com/view/10.1093/oso/9780190256692.001.0001/oso-9780190256692">diary studies</a>. Still, none are as accurate as the objectively measured behavioral data that platforms collect.</p><p>Platform data that tracks individuals over time is rare but especially important because it gives researchers the ability to correlate on-platform behavior with off-platform outcomes, which might be measured using surveys.</p><blockquote>In a complex and rapidly changing world, social science needs as many time series as possible. As someone who has spent a lot of time studying Twitter, there is a glaring hole at the center of this literature. … I would trade *almost all of the research ever published about Twitter* for a high-quality representative panel survey of Twitter users with trace data from their accounts matched with their survey responses.</blockquote><blockquote><strong><em>— Kevin Munger</em></strong><em>, </em><a href="https://kevinmunger.substack.com/p/in-favor-of-quantitative-description"><em>In Favor of Quantitative Description</em></a><em> (2020)</em></blockquote><p>The best example we currently have of a study linking on-platform behavior with other data was conducted internally by Instagram and leaked among the Facebook papers in September 2021. Not the widely covered “one in three girls” focus group study, which <a href="https://web.archive.org/web/20220509211627/https://www.nytimes.com/2021/10/10/opinion/instagram-facebook-mental-health-study.html">isn’t much help</a> for inferring cause, but a different study. This study <a href="https://s.wsj.net/public/resources/documents/social-comparison-topics-celebrities-like-counts-selfies.pdf">surveyed 100,000 Instagram users</a> across 9 countries on the degree to which Instagram contributes to “social comparison” — how viewing other peoples’ posts on Instagram made people feel about themselves, whether positive (inspired) or negative (demoralized). These survey responses were linked with individual-level platform data about what posts each user had seen. This allowed measurements of how social comparison is correlated with factors including different categories of post, the presence of like counts, celebrity content and the use of filters. There was only one controlled experiment (which involved hiding like counts), but even mere correlations help with formulating and testing causal hypotheses. Giving independent researchers the ability to connect on- and off-platform data in this way would be a significant step towards better understanding the effects of recommenders.</p><p>There are other limitations on what granting researchers access to platform data can achieve. There need to be appropriate privacy constraints on what data anyone (both platform employees and external researchers) can look at. In particular, the content of private messages will in most cases be off limits. There is also some data that no one (including platforms) has access to. For example, evaluating the accuracy of statements (whether they are true or false, or somewhere in between) is very labor intensive, and <a href="https://twitter.com/dkroy/status/974251282071474177?lang=en-GB">difficult to do at scale</a>. Studies that require truth assessments for large numbers of items are likely impractical.</p><h4>Limited Causal Inference</h4><p>Above, we mentioned several strategies that can be used to estimate causal effects from observational data. However, often this evidence is not as strong, or requires more assumptions, than would be required to estimate those effects via experimental data.</p><p>For example, say we want to know whether diet videos on TikTok are harming the mental health of teenage girls, and have access to all the (observational) platform data that exists. This dataset probably doesn’t include all variables that affect mental health, so we can’t rely on subclassification, matching, or propensity scoring. The effect of these unobserved variables is likely to vary with time, so we can’t rely on longitudinal panel data. Natural experiments arise less frequently on platforms like TikTok because the algorithm is pretty much the same across jurisdictions, and is very responsive to user behavior, so it can be difficult to find so-called “exogenous” sources of variation that are not influenced by the choices of users.</p><p>Thus, to assess causality, we really need to conduct an experiment: that is, to randomize the degree to which TikTok users are exposed to diet videos, and measure how this affects their mental health. Such on-platform experiments are required to answer many of the most important questions we have about the effects of recommenders.</p><h3>With on-platform experiments</h3><p>At this level of access, researchers could do all of the above, plus run <strong>on-platform experiments</strong> capable of detecting many causal relationships that are of interest.</p><p>For example, researchers could assign users to different recommender algorithms and assess the degree to which each causes or mitigates affective political polarization. They could randomly show different users various political ads or interventions like fact checks, and see whether they cause changes in future media consumption patterns. Like in the <a href="https://s.wsj.net/public/resources/documents/social-comparison-topics-celebrities-like-counts-selfies.pdf">Instagram study</a>, they could understand what types of content cause improvements or deteriorations in the mental health of users. Experiments are the gold standard for accumulating causal knowledge, and it is important that at least some independent researchers have input into what on-platform experiments are run, <a href="https://misinforeview.hks.harvard.edu/article/tackling-misinformation-what-researchers-could-do-with-social-media-data/">without being subject to veto</a> by commercially-motivated platforms.</p><h4>Limited Ability to Generalize</h4><p>That said, even with the cooperation of platforms, experiments may not always be appropriate. Well-designed large scale experiments, or <em>randomized control trials</em> (RCTs), can provide causal information, but are not invulnerable to confounders and must be <a href="https://doi.org/10.1016/j.socscimed.2017.12.005">designed and interpreted carefully</a>. Compared to observational studies, they are also relatively costly, and the results can be highly context specific. Taking the results of a well-designed study of Twitter in the US and assuming they hold for Facebook in India would be like “<a href="https://kevinmunger.substack.com/p/meta-science-is-political-methodology?s=r">designing and executing a moon landing … then sending the same ship to Mars with triple the fuel and assuming things will work out.</a>”</p><p>In practice, this means that results from one platform may not generalize to others, that results from one jurisdiction may not generalize to other places, and that results from one point in time may not hold in future — “<a href="https://kevinmunger.substack.com/p/temporal-validity-is-distinct-from?s=r">the internet changes everything, very quickly</a>.” The cost of running experiments may not be justified if the resulting knowledge is brittle.</p><blockquote>Quantitative description is cheap, and much of the cost is fixed. In contrast, causal knowledge is expensive and much of the cost is marginal. The marginal cost of updating [a political] database and the [some modeled] scores for each session of Congress is much lower than the fixed cost of creating those models in the first place. In contrast, the marginal cost of re-running a Twitter RCT every time Twitter’s userbase or platform policies change is very high.</blockquote><blockquote><strong><em>— Kevin Munger</em></strong><em>, </em><a href="https://kevinmunger.substack.com/p/in-favor-of-quantitative-description"><em>In Favor of Quantitative Description</em></a><em> (2020)</em></blockquote><h4>Limited Isolation of Cause/Effect</h4><p>The social media context presents some additional subtle issues that complicate experimental design and data interpretation.</p><p>One is the question of which component of a social media ecosystem is responsible for a particular harm. If we were able to prove that social media use increases the risk of radicalisation, how could we tell if that harm is due to the recommendation algorithms, the user interface, or the <a href="https://kevinmunger.substack.com/p/susan-wojcicki-wants-you-to-think?s=r">behavior of other users</a>? It is very difficult to manipulate these variables in isolation. When you change the algorithm or the interface, user behavior <a href="https://www.commerce.senate.gov/services/files/62102355-DC26-4909-BF90-8FB068145F18">also changes</a>, often <a href="https://doi.org/10.1177/14614448211027174">strategically</a>, in ways that are not anticipated. An A/B test might capture the immediate effects, but not the long term changes to “equilibrium” behavior that would be observable after, say, 6 months.</p><p>Network effects also complicate experiments. Say a platform wants to evaluate the impact of upranking comment threads with long back-and-forth conversations. You might be one of the users selected to be part of the trial, but most of your friends won’t be. The new conversations you are involved in may peter out because your friends are less likely to be prompted to continue them. This moderates the effect of the new ranking algorithm, underestimating its impact. Such second-order network effects are difficult to avoid without changing things for everyone in an interconnected social network, which is one reason why platforms often roll out changes <a href="https://www.engadget.com/facebook-is-testing-its-less-political-news-feed-in-75-new-countries-220040366.html">one country at a time</a>.</p><h4>Limited by Ethics Considerations</h4><p>There are also ethical considerations when conducting experiments with human research subjects. Online platforms routinely conduct A/B tests with no external oversight, but academic studies are subject to ethics review by university ethics committees, which may prevent some large scale studies from being conducted if there is a reasonable expectation that they may <a href="https://psyarxiv.com/j9n3q/">disadvantage or harm</a> some users. Based on such concerns, one <a href="https://doi.org/10.1073/pnas.1820701116">study</a> found that “people often approve of untested policies or treatments (A or B) being universally implemented but disapprove of randomized experiments (A/B tests) to determine which of those policies or treatments is superior” — even if conducting the experiment would help make everyone better off in the long run.</p><p>The current <a href="https://osf.io/preprints/socarxiv/p3z9v/">scarcity</a> of causal evidence does not mean we shouldn’t take action to mitigate the risks that platforms cause harms. There is a <a href="https://en.wikipedia.org/wiki/Agnotology">history</a> of industries, from tobacco to fossil fuels, using uncertainty to deflect responsibility and avoid taking action. The <a href="https://en.wikipedia.org/wiki/Precautionary_principle">precautionary principle</a> applies: in many cases it is probably reasonable to make policy changes based on correlational studies while still evaluating causality. Yet we don’t really understand if and how recommender algorithms harm teen mental health, increase the prevalence of false beliefs, or exacerbate polarization. If we are to understand the effects of recommenders then — be it through platform-initiated transparency measures or <a href="https://cyber.fsi.stanford.edu/news/platform-transparency-and-accountability-act-new-legislation-addresses-platform-data-secrecy">regulation</a> — researchers will need access to platform data, and the ability to run on-platform experiments.</p><p><em>Luke Thorburn was supported in part by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (safeandtrustedai.org), King’s College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e89b7363d57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/how-to-measure-the-causal-effects-of-recommenders-5e89b7363d57">How to Measure the Causal Effects of Recommenders</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Will “Amplification” Mean in Court?]]></title>
            <link>https://medium.com/understanding-recommenders/what-will-amplification-mean-in-court-a6b94bad6354?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/a6b94bad6354</guid>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[regulation]]></category>
            <category><![CDATA[amplification]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <dc:creator><![CDATA[Luke Thorburn]]></dc:creator>
            <pubDate>Mon, 23 May 2022 06:50:56 GMT</pubDate>
            <atom:updated>2022-11-23T17:50:07.188Z</atom:updated>
            <content:encoded><![CDATA[<p>— <a href="https://lukethorburn.com/">Luke Thorburn</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*ZulrdiJFXjfAB7xV7ZkdNQ.png" /></figure><p>“Amplification” is a common term in discussions about how best to manage and regulate social media platforms, and is starting to appear in proposed laws. We’re not convinced this is a good idea, because the meaning of this word is not always clear. Social media surely does “amplify” user content<em> </em>in the sense that anyone, anywhere can “go viral” and find themselves with a massive audience, but this intuitive notion hides a range of very complex questions.</p><p>The democratization of reach has profoundly benefited many people, but viral <a href="https://eprints.soton.ac.uk/415943/1/InternetScience.pdf">information cascades</a> can also do harm in a <a href="https://www.gov.uk/government/consultations/online-harms-white-paper/online-harms-white-paper#the-harms-in-scope">variety of ways</a>. Humans amplify each other: information can spread rapidly through messaging apps such as WhatsApp and Telegram, and even journalists are understood to “amplify” certain issues or ideas <a href="https://www.bbc.co.uk/beyondfakenews/hownottoamplifybadideas">just by reporting</a> on them. So amplification doesn’t require any sort of algorithmic process, but many platforms incorporate recommender systems and other algorithms that filter and personalize information, including social media, news apps, streaming video, and online shopping. In these systems, information spreads through <a href="https://www.commerce.senate.gov/services/files/62102355-DC26-4909-BF90-8FB068145F18">complex interactions</a> between people and software. While it can be hard to say if an algorithm “caused” the distribution of some item, it is certainly worth asking if algorithmic changes might lead to a healthier information ecosystem.</p><p>Lawmakers are increasingly focused on this question. There are several proposed laws that use the term “amplification” when considering harms from algorithmic information filtering, such as the <a href="https://www.congress.gov/bill/117th-congress/house-bill/5596/text">Justice Against Malicious Algorithms</a> and <a href="https://www.congress.gov/bill/117th-congress/senate-bill/2228/text">DISCOURSE</a> Acts in the US, and the <a href="https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52020PC0825&amp;from=en">Digital Services Act</a> in the EU. However, none of these drafts provide a precise definition for amplification. Further, most discussions of amplification concern social media platforms, and make much less sense for other types of recommender-driven platforms such as news aggregators, online shopping, or podcast discovery.</p><p>Definitions for key terms in laws are sometimes deliberately omitted, so that judges can exercise reasonable discretion in how they are applied, while respecting the intention behind the law. But in the case of “amplification,” the intent behind the term is often unclear, and attempts to define amplification have emphasized different effects. These include the absolute reach of an item of content, its reach relative to some counterfactual scenario, and a lack of transparency or control over whether or not an item is presented to a user.</p><p>We argue that the term “amplification” may be confusing rather than clarifying in discussions of recommender systems, and is certainly not precise or unambiguous enough to be used in law — especially law that may relate to fundamental speech rights. Reviewing the three most common interpretations, we speculate on how a judge might decide whether amplification has occurred, describing the definitional and methodological challenges they would face. These turn out to be significant, so we suggest several alternative ways of thinking about algorithmic information spread that might help people more easily say what they mean, or perhaps discover the implications of what they are already saying.</p><h3>Three Interpretations</h3><p>Put yourself in the shoes of a judge who is presented with a case that hinges on whether particular content has been “amplified” on a particular platform. For example, the content might be defamatory, or consist of illegal hate speech or incitement to violence. As the judge, you have access to internal platform data. How do you decide if amplification has occurred?</p><p>We reviewed a variety of proposed definitions of amplification, and asked a number of people to explain their use of the term. There seem to be more or less three different ideas that are commonly held.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Uw7wSo-2jiVLkdzwImdzTg.png" /></figure><h4>Amplification as counterfactual</h4><p>The first interpretation is prompted by an obvious question: amplified <em>relative to what</em>? To say that something was “algorithmically amplified” means that if the platform had been designed differently, the content would not have been disseminated so widely. How widely would that have been? This notion of amplification requires us to articulate a counterfactual scenario, then compare the real and the imagined reach.</p><p>Sometimes this is plausible. When Twitter <a href="https://www.buzzfeednews.com/article/alexkantrowitz/twitter-to-introduce-algorithmic-timeline-as-soon-as-next-we">switched</a> to an algorithmic “home” timeline in 2016, they kept 1% of their users on the chronological timeline. This provided a natural point of comparison for a recent study of the <a href="https://www.pnas.org/doi/pdf/10.1073/pnas.2025334119">amplification of politics</a> on Twitter. The researchers defined an “amplification ratio,” which quantifies the reach of content in one system relative to another:</p><blockquote>We define the amplification ratio of set <em>T</em> of tweets in an audience <em>U</em> as the ratio of the reach of <em>T</em> in <em>U</em> intersected with the treatment [algorithmic] group and the reach of <em>T</em> in <em>U</em> intersected with the control [chronological] group.</blockquote><p>This is a good example of the level of detail required to unambiguously define “amplification,” and is a reasonable definition in the context of Twitter. However, this definition doesn’t generalize to other cases and platforms.</p><p>In this study the “set <em>T</em> of tweets” came from accounts belonging to politicians in seven countries. The proposed <a href="https://www.congress.gov/bill/117th-congress/senate-bill/2448/text">Health Misinformation Act</a> addresses a much more complicated class of items. Categories such as “health misinformation,” “violent extremism,” or “defamatory material” are hard to define precisely, but the measurement of amplification would first require classification of every item on the platform. There is an additional challenge of deciding how to aggregate outcomes for all such items. The authors of the <a href="https://www.pnas.org/doi/pdf/10.1073/pnas.2025334119">Twitter study</a> are upfront about the fact that methodological choices can influence findings, writing that</p><blockquote>comparing political parties on the basis of aggregate amplification of the entire party … or on the basis of individual amplification of their members … leads to seemingly different conclusions: While individual amplification is not associated with party membership, the aggregate group amplification may be different for each party.</blockquote><p>A deeper problem is that there isn’t always a clear baseline scenario or control group. While some have <a href="https://accountabletech.org/campaign/facebook-turn-off-the-algorithms/">advocated</a> for social media with “<a href="https://twitter.com/joinmastodon/status/1146113459953606658">no algorithm</a>,” what they usually mean is a simple reverse-chronological list of posts. This makes the most sense on platforms like Twitter, Instagram, or TikTok where there is a central, algorithmically-ranked feed, and a clear subset of content relevant to the user (the accounts they follow). This idea is less clear on platforms like Facebook where there are many different types of content and the user interface is less oriented around “following.” Designing a single “chronological” feed requires many detailed decisions such as whether it should include posts from groups the user has joined, or posts their friends have commented on. A chronological baseline is even less meaningful for platforms like Spotify, YouTube, or Google News, where a true chronological feed of every new item would be essentially useless.</p><p>Even where purely reverse-chronological rankings are possible, they are <a href="https://knightcolumbia.org/content/amplification-and-its-discontents">not neutral</a> — they reward accounts that post more frequently, have no defense against spam, and can have higher proportions of “borderline” content compared to rankings generated by more sophisticated algorithms. In some contexts there may be other plausible baselines, such as a standard recommendation algorithm from a relevant <a href="https://link.springer.com/book/10.1007/978-3-319-29659-3">textbook</a>, or a previous version of the algorithm used by the platform in question. None of these options is an obviously natural or impartial point of comparison.</p><p>Then there is the problem of actually tallying the counterfactual. In the Twitter study, the authors had access to real data from the baseline scenario because Twitter kept 1% of users on a reverse-chronological timeline. But this is the exception. In general, the counterfactual scenario would need to be simulated to generate a quantitative estimate of the amount of amplification. The process would be further complicated by questions of who performs the simulation (the platform? a third party?), how to model user behavior in the counterfactual (people <a href="https://www.commerce.senate.gov/services/files/62102355-DC26-4909-BF90-8FB068145F18">change their posting behavior when algorithms change</a>), and how to assess the credibility and statistical significance of the calculated difference.</p><p>A judge would also need to decide how to measure distribution or exposure. Should they count people seeing the headline? Clicking a link to the article? Sharing the article? Should they consider whether or not it was seen over a wide geographic area? What if the same number of people saw an item as compared to baseline, but it was positioned more saliently? Or they saw it more frequently? Whatever the metric, there would also need to be a threshold for what counts as amplification. For example, you might decide “the defamatory article will be said to be amplified if it is seen by &gt;10% more people relative to the baseline scenario.”</p><p>In short, there are a huge number of <a href="https://en.wikipedia.org/wiki/Researcher_degrees_of_freedom">degrees of freedom</a> in defining amplification relative to “what might have been,” and actually estimating an amplification figure can be a complicated exercise in counterfactual simulation. Perhaps with access to relevant expertise and resources, courts could rule on “amplification” decisions despite these subtleties. Unfortunately, researchers who conduct such studies often work for a social media platform or collaborate closely with people who do, which may limit their independence as expert witnesses. These ambiguities and difficulties could create confusion and undermine confidence in legal decisions involving speech and expression.</p><h4>Amplification as (mere) distribution</h4><p>Sometimes there doesn’t seem to be any semantic difference between the claim that certain content was “amplified” and the claim that the content was “shown.” For example, this <a href="https://firstdraftnews.org/wp-content/uploads/2018/07/infoDisorder_glossary.pdf?x32994">glossary</a> compiled by misinformation researchers states that</p><blockquote>Manufactured Amplification occurs when the reach or spread of information is boosted through artificial means.</blockquote><p>If “manufactured amplification” is “reach or spread” by “artificial” means, then “amplification” seems to be just “reach or spread”. More complex definitions also frequently turn out to be equivalent to mere distribution. For example, this <a href="https://policyreview.info/articles/analysis/recommender-systems-and-amplification-extremist-content">policy paper</a> defines amplification as</p><blockquote>the promotion of certain types of content … at the expense of more moderate viewpoints.</blockquote><p>while the proposed <a href="https://www.congress.gov/bill/117th-congress/house-bill/5596/text#HC6A49E2B481E4493AA136C6236BC8742">JAMA Act</a> considers</p><blockquote>the material enhancement, using a personalized algorithm, of the prominence of such information with respect to other information.</blockquote><p>The phrases “at the expense of more moderate viewpoints” and “with respect to other information” are redundant because there are a limited number of items on a page of recommendations, and ultimately because there are only so many minutes of human attention in a day. This zero-sum nature means that selection of any type of content will inevitably come at the expense of all other types of content. In this context, it is difficult to see how amplification can be distinguished from distribution, or for that matter from human editorial discretion.</p><p>Collapsing amplification to distribution or discretion would lead to very broad laws, because it would be equivalent to making certain types of content illegal. Would a platform be liable for “amplification” if they merely failed to suppress particular items? This could even prevent algorithmic distribution of accurate information on important topics such as public health misinformation or the incitement of political violence; the <a href="https://www.bbc.co.uk/beyondfakenews/hownottoamplifybadideas">BBC</a> already acknowledges that it unavoidably amplifies ideas by reporting on them. Legally and ethically, a law that treats amplification as distribution might be in violation of <a href="https://www.un.org/en/about-us/universal-declaration-of-human-rights">human rights</a> principles on free expression, and in the US, could be unconstitutional due to the First Amendment. Both <a href="https://www.nytimes.com/2021/12/01/technology/big-tech-amplification.html">judges</a> and <a href="https://www.acslaw.org/expertforum/liability-for-amplification-of-disinformation-a-law-of-unintended-consequences/">legal</a> <a href="https://knightcolumbia.org/content/amplification-and-its-discontents">scholars</a> are skeptical that a law seeking to restrict amplification could avoid this.</p><blockquote>Some versions of amplification law would be flatly unconstitutional in the U.S., and face serious hurdles based on human or fundamental rights law in other countries. Others might have a narrow path to constitutionality, but would require a lot more work than anyone has put into them so far. Perhaps after doing that work, we will arrive at wise and nuanced laws regulating amplification. For now, I am largely a skeptic.</blockquote><blockquote><em>— </em><strong><em>Daphne Keller</em></strong><em>, </em><a href="https://knightcolumbia.org/content/amplification-and-its-discontents"><em>Amplification and Its Discontents</em></a><em> (2021)</em></blockquote><h4>Amplification as lack of agency</h4><p>In the third main interpretation we found, amplification occurs when a user is shown an item of content that they “didn’t ask for” or when the reasons for it being shown are opaque. Amplification is linked to a lack of user agency or control over the information to which they are exposed. The proposed <a href="https://www.congress.gov/bill/117th-congress/house-bill/2154/text#H7E959E33CC7741F8A1DF6E2D13C90C3C">Protecting Americans from Dangerous Algorithms Act</a> takes this approach, exempting platforms from liability if their content is</p><blockquote>(I) … ranked, ordered, promoted, recommended, amplified, or similarly altered in a way that is obvious, understandable, and transparent to a reasonable user based only on the delivery or display of the information (without the need to reference the terms of service or any other agreement), including sorting information —</blockquote><blockquote>- (a) chronologically or reverse chronologically;</blockquote><blockquote>- (b) by average user rating or number of user reviews;</blockquote><blockquote>- (c) alphabetically;</blockquote><blockquote>- (d) randomly; and</blockquote><blockquote>- (e) by views, downloads, or a similar usage metric; or</blockquote><blockquote>(II) the algorithm, model, or other computational process is used for information a user specifically searches for.</blockquote><p>This has intuitive appeal: many recommenders are black boxes, and ranking decisions can seem arbitrary. In contrast, reverse-chronological feeds or Reddit-style voting algorithms are “explainable”. But there are problems.</p><p>It’s not always clear what user intent was, or what it means to “specifically search for” something. If a user searches for “politics” and is subsequently offered some items that discuss popular conspiracy theories about politicians, did they intend to see those? Also, recommender systems are designed to allow users to delegate curatorial responsibility so as to avoid having to explicitly ask for each item. Some recommenders (such as those for news articles) are valuable precisely because they show you things you didn’t know to ask for. How would a judge draw the line between items that were “asked for” and those that were not?</p><p>The second problem relates to what counts as “obvious, understandable, and transparent.” While some algorithms are certainly easier to understand than others, explaining platform algorithms doesn’t necessarily explain platform outcomes. These are the result of complex interactions between automated systems, editorial or managerial decisions, and the online behavior of large numbers of people, and may not be explainable without detailed traces of user activity. Further, users are <a href="https://www.commerce.senate.gov/services/files/62102355-DC26-4909-BF90-8FB068145F18">strategic</a>: they will quickly learn, adapt, and attempt to exploit any distribution algorithm. User behavior thus has large effects on information distribution that are difficult to separate from algorithm design. For example, it is possible that the <a href="https://www.science.org/doi/10.1126/science.abb2428">different social media strategies</a> of major political parties might explain disparities in how left- and right-leaning content is <a href="https://www.pnas.org/doi/pdf/10.1073/pnas.2025334119">amplified on Twitter</a>.</p><p>There is also the possibility that agency might be undermined when items are recommended because of inferences made from the user’s personal data, which may have been collected or used without their fully informed consent. The <a href="https://www.congress.gov/bill/117th-congress/house-bill/5596/text">JAMA Act</a> would make platforms liable for harm from amplification that “relies on information specific to an individual”. As worded, this would appear to include almost all recommendation algorithms. Even a simple subscription-based, reverse chronological timeline relies on information about which accounts each user followed. If this bill was not intended to penalize explicit following, then judges will have to invent some criteria for determining when user data becomes “specific to an individual.”</p><h3>Alternatives to “Amplification”</h3><p>The term amplification is ambiguous, but information <em>can</em> spread through online networks in a way that appears different in degree, if not in kind, from what was possible prior to the Internet. Most distinctively, content from otherwise inconspicuous citizens can “go viral”, as a burst of engagement triggers large scale information cascades. There is a case for updating the way platforms, media, and online communications are regulated to keep pace with changes in information and societal dynamics.</p><p>Ultimately, we hope that those concerned with these matters will be clearer about what their goals are when they talk about regulating the amplification of content, crafting language and policy options that clearly reflect that intent. We are not lawyers, so we are not going to try to parse which goals and formulations might be permissible under any particular constitutional regime — others have <a href="https://knightcolumbia.org/content/amplification-and-its-discontents">attempted that</a>. Instead we’ll propose plain-language interpretations of the above ideas that we hope are forthright and clarifying.</p><p>For instance, is the goal in diminishing amplification to suppress certain kinds of information or content? Stated this way, the potential threat to fundamental rights is obvious. Instead of <em>X is amplified</em>, perhaps you mean <em>X is harmful when viewed, to a degree that </em><a href="https://columbialawreview.org/content/governing-online-speech-from-posts-as-trumps-to-proportionality-and-probability/"><em>outweighs</em></a><em> certain speech rights, </em>and additionally something like</p><ul><li><em>X has reach Y as measured by metric Z</em></li><li><em>X had Y% greater reach in system U than it would have had under system V</em></li><li><em>Y users liked X so it was algorithmically selected for Z additional users</em></li></ul><p>Policy options that promote this goal include mandating “<a href="https://slate.com/technology/2020/08/facebook-twitter-youtube-misinformation-virality-speed-bump.html">circuit breakers</a>” that <a href="https://www.platformer.news/p/twitter-adds-friction">add friction to sharing</a> functionality or applying additional moderation scrutiny to content that spreads beyond certain numerical limits. Circuit breakers could be content-neutral or narrowly tailored to target particular types of speech. A more <a href="https://www.lawfareblog.com/reckless-associations-new-tort-new-information-ecosystem">unusual proposal</a> focuses on people rather than algorithms, by making superspreaders of harmful content liable for the actions of their followers.</p><p>Is the goal to give users more control? Instead of <em>X is amplified</em>, you might say</p><ul><li><em>the user did not intend/want/ask to see X</em></li><li><em>the user did not have control over whether X was shown to them</em></li></ul><p>These require some sort of definition of “ask” and “control” before they can be clear. Regulatory models include mandating a reverse-chronological ranking option, <a href="https://techpolicy.press/why-social-media-needs-mandatory-interoperability/">platform interoperability</a>, enabling third party content ranking <a href="https://cyber.fsi.stanford.edu/publication/report-working-group-platform-scale">middleware</a> (a <a href="https://knightcolumbia.org/content/amplification-and-its-discontents">competition-based</a> approach), or user control over how their personal data is used to target content (a <a href="https://knightcolumbia.org/content/amplification-and-its-discontents">privacy-based</a> approach).</p><p>Is the goal to make the reasons content is shown more transparent? Instead of <em>X is amplified</em>, you could mean</p><ul><li><em>the user doesn’t understand why X was shown to them</em></li><li><em>the reasons why X was shown are not disclosed</em></li></ul><p>In this case, regulatory models include <a href="https://www.congress.gov/bill/116th-congress/senate-bill/1356">mandating truth and transparency in advertising</a>, requiring users to use their true identities, requiring that recommendation algorithms provide explanations, or <a href="https://www.coons.senate.gov/imo/media/doc/text_pata_117.pdf">establishing a new regulatory body</a> to set transparency standards for online platforms (all <a href="https://knightcolumbia.org/content/amplification-and-its-discontents">consumer protection-based</a> approaches).</p><p>There is much at stake when regulating human expression, and the algorithmic systems that propagate it. To avoid unintended consequences, we need to think through the language we use when discussing and responding to the challenges of algorithmic media, especially when that language becomes law. Because it can mean so many different things, the word “amplification” may ultimately confuse more than it clarifies.</p><p><em>This piece was originally published on </em><a href="https://techpolicy.press/what-will-amplification-mean-in-court/"><em>Tech Policy Press</em></a><em>. Thank you to Justin Hendrix for useful feedback.</em></p><p><em>Luke Thorburn was supported in part by UK Research and Innovation [grant number EP/S023356/1], in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (safeandtrustedai.org), King’s College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6b94bad6354" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/what-will-amplification-mean-in-court-a6b94bad6354">What Will “Amplification” Mean in Court?</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s Right and What’s Wrong with Optimizing for Engagement]]></title>
            <link>https://medium.com/understanding-recommenders/whats-right-and-what-s-wrong-with-optimizing-for-engagement-5abaac021851?source=rss----47b079b44caf---4</link>
            <guid isPermaLink="false">https://medium.com/p/5abaac021851</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[social-media]]></category>
            <dc:creator><![CDATA[Priyanjana Bengani]]></dc:creator>
            <pubDate>Wed, 27 Apr 2022 12:44:47 GMT</pubDate>
            <atom:updated>2022-04-27T21:56:30.488Z</atom:updated>
            <content:encoded><![CDATA[<p>— <a href="https://twitter.com/acookiecrumbles">Priyanjana Bengani</a>, <a href="https://twitter.com/jonathanstray">Jonathan Stray</a>, <a href="https://lukethorburn.com/">Luke Thorburn</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5r3EiSfKRdvozsIfNzgc9A.png" /></figure><p>“Engagement” is a complicated, loaded word. Everyone from platforms to journalists to advertisers seeks it — even praises it — but the pursuit of engagement has been blamed for many social ills including <a href="https://time.com/6127981/addictive-algorithms-2022-facebook-instagram/">addiction</a>, <a href="https://royalsociety.org/-/media/policy/projects/online-information-environment/oie-echo-chambers.pdf">polarization</a>, <a href="https://www.nytimes.com/2019/03/29/technology/youtube-online-extremism.html">radicalization</a>, <a href="https://www.cnet.com/culture/internet/instagram-and-tiktok-are-failing-users-with-eating-disorders/">eating disorders</a>, and <a href="https://www.wired.com/story/right-wing-fake-news-more-engagement-facebook/">misinformation</a>. For recommender systems, optimizing for engagement means choosing content that users are likely to respond to, and most large recommender systems are <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">built this way</a>. While this can be a problematic strategy, user engagement often does signal genuine value, because people tend to give more attention to things that matter to them. It’s up to platform operators to balance this usefulness with the negative consequences of chasing engagement.</p><p>In this post we explore engagement in the context of recommender systems, the personalized content selection algorithms that power a wide variety of products including social media, news aggregators, music and video streaming services, and online shopping. We’ll try to unpack the advantages and disadvantages of using engagement in different ways, while also evaluating the imperfect alternatives. As we are likely to continue seeing engagement used in personalization for some time, we believe the most promising approach is to get better at differentiating between “good” and “bad” engagement.</p><h3>Defining Engagement</h3><p>“Engagement” has been used in software engineering, and interactive systems in general, <a href="http://www.dcs.gla.ac.uk/~mounia/Papers/MUE.pdf">since at least the 1980s</a>. One <a href="https://open.library.ubc.ca/handle/2429/38785/OBrien_Toms_2008_What_is_user_engagement.pdf?sequence=1">2008</a> review says that “successful technologies are not just usable; they engage users,” noting that an engaging system “push[es] the boundaries of user experience from merely perfunctory to pleasurable and memorable.” This is why product designers see engagement as positive, and why it is attractive for many types of systems including recommenders.</p><p>On the other hand, recommender systems are a type of media, and media organizations also strive for engagement. For <a href="https://medium.com/we-are-hearken/what-we-mean-when-we-talk-about-engagement-a4816f22902f">journalists</a>, “engagement happens when members of the public are responsive to newsrooms, and newsrooms are in turn responsive to members of the public.” Similarly, recommender systems that select engaging content have the advantage of being responsive to users. Measuring audience responsiveness is another matter, and a <a href="https://blog.parse.ly/define-jargon-audience-engagement/">wide variety</a> of metrics are used to represent engagement, including views, shares, time on site, comments, reactions, and bounce rate.</p><p>Engagement may be a sophisticated, holistic concept, but in practice it is usually monitored through online behavioral data. Engagement metrics can be signals of value to multiple stakeholders. Ideally, when users engage with something — an app, an article, a video — it’s because they get genuine value out of it. At the same time, businesses need engaged consumers to survive regardless of their business model: ad-supported products need to capture user attention, subscription products need to provide services that keep people coming back, and even non-profit organizations need to be able to tangibly illustrate their impact to members and funders (even the BBC <a href="https://medium.com/partnership-on-ai/how-news-organizations-use-algorithms-d2441f3482a4">optimizes for engagement</a>). As critics from <a href="http://partnershiponai.org/beyond-engagement-aligning-algorithmic-recommendations-with-prosocial-goals/">academics</a> to <a href="https://time.com/6127981/addictive-algorithms-2022-facebook-instagram/">journalists</a> to<a href="https://www.commerce.senate.gov/2019/6/optimizing-for-engagement-understanding-the-use-of-persuasive-technology-on-internet-platforms"> governments</a> have noted, engagement is not equivalent to value. Yet most recommender systems continue to be <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">based on</a> optimizing for engagement because it is a plausible proxy for user values in the absence of other easily observable data.</p><p>Before we can discuss what is good and bad about algorithmically selecting the most engaging content, we need a definition in the context of recommender systems. Putting all the above pieces together, we propose:</p><blockquote>Engagement is a set of user behaviors, generated in the normal course of interaction with the platform, which are thought to correlate with value to the user, the platform, or other stakeholders.</blockquote><p>This definition reflects the fact that engagement signals are chosen to be indicators of value, but aren’t necessarily going to be fully aligned with all values of all stakeholders in all situations. It says that engagement is something users normally do, as opposed to less common actions such as adjusting settings or taking surveys. It also suggests that there are some signals of value that can only be derived from non-behavior data, which parallels well-known arguments from economics that a <a href="https://medium.com/understanding-recommenders/what-does-it-mean-to-give-someone-what-they-want-the-nature-of-preferences-in-recommender-systems-82b5a1559157">person’s preferences</a> cannot be inferred solely from the choices they make. The definition is also a <a href="https://www.researchgate.net/publication/338516177_Multistakeholder_recommendation_Survey_and_research_directions">multistakeholder perspective</a>, as recommenders simultaneously serve different groups.</p><h3>Good vs. Bad Engagement</h3><p>Engagement is a double-edged sword. It is necessary for economic sustainability and often indicates value, but blindly seeking engagement can also have negative consequences. This situation is somewhat analogous to “profit”: ideally, a company is profitable because customers are buying something they find genuinely valuable, but there are also deceptive or harmful ways to make money.</p><p>Optimizing for engagement can lead to showing users addictive content, which can cause self-control problems impairing other areas of one’s life. A <a href="https://web.stanford.edu/~gentzkow/research/DigitalAddiction.pdf">recent study</a> paid people not to use various apps, including recommender-driven social media, and observed long term drops in subsequent use, implying social media use is habit-forming for some people. One theory is that recommenders might be designed to choose items that provide a “<a href="https://www.axios.com/sean-parker-unloads-on-facebook-2508036343.html">little dopamine hit</a>” like gambling or drugs, resulting in a kind of <a href="https://www.nature.com/articles/s41467-020-19607-x.pdf">behavioral conditioning</a>. Activities like <a href="https://www.sciencedirect.com/science/article/pii/S2352853220301401">binge-watching</a> or<a href="https://www.bbc.com/worklife/article/20210226-the-darkly-soothing-compulsion-of-doomscrolling"> doom-scrolling</a> can mimic addictive behavior by creating unhealthy dependencies.</p><p>Optimizing for engagement might also lead to incentivizing divisive, extreme, or outrageous material. There is evidence that more extreme content leads to more engagement, and some researchers consider recommender-driven platforms <a href="https://www.nature.com/articles/s41599-020-00550-7">angry by design</a>. One <a href="https://osf.io/s4w2x/">large review</a> of “<a href="https://journals.sagepub.com/doi/full/10.1177/1745691620917336">moral contagion</a>” found “each message is 12% more likely to be shared for each additional moral-emotional word.” Other studies have found that <a href="https://www.pnas.org/content/118/26/e2024292118">divisive</a> and <a href="https://datasociety.net/library/alternative-influence/">extreme</a> material is more likely to drive engagement. <a href="https://www.facebook.com/notes/751449002072082/">Internal research</a> at Facebook has found that “no matter where we draw the lines for what is allowed, as a piece of content gets close to that line, people will engage with it more on average — even when they tell us afterwards they don’t like the content.” However, platforms have <a href="http://arxiv.org/abs/2003.03318">intervened</a> and implemented policies to<a href="https://gizmodo.com/youtube-is-going-to-bury-borderline-content-it-wont-te-1832162383"> reduce the reach</a> of various kinds of engaging but objectionable content. While Twitter <a href="http://www.pnas.org/lookup/doi/10.1073/pnas.2025334119">found</a> that the mainstream political right wing “enjoyed higher amplification” than the left in six of the seven countries they studied, they also found that “far-left” and “far-right” political parties and elected officials were not algorithmically preferred (relative to a reverse-chronological feed).</p><p>A related problem is that optimizing for outrage-heavy and extreme content can leave the algorithm open to abuse by users who have a deep understanding of how the system functions and therefore can take steps to manipulate it through inauthentic activity. This type of activity has been adopted by a wide array of user types, from <a href="https://www.bbc.co.uk/news/world-asia-china-58062630">state actors</a> to Instagram <a href="https://www.researchgate.net/publication/337456796_Weapons_of_the_Chic_Instagram_Influencer_Engagement_Pods_as_Practices_of_Resistance_to_Instagram_Platform_Labor">influencers</a>.</p><p>If users are more likely to click on a particular type of content, most recommenders will serve them more of it. If the user then starts to prefer whatever they frequently see, it can lead to feedback loops causing “filter bubble” effects, polarization, or even radicalization ultimately leading to violence. This possibility has been demonstrated in <a href="https://dl.acm.org/doi/10.1145/3460231.3478849">simulations</a>, but observing it with real users on real platforms is difficult. For example, TikTok bots programmed to watch only sad videos were eventually recommended <a href="https://www.wsj.com/articles/tiktok-algorithm-video-investigation-11626877477">mostly depressive content</a>, but it’s difficult to know what happens to users in the real world from this type of experiment. The best real-world evidence we have comes from case studies linking recommender-driven platforms with <a href="https://www.wsj.com/story/tiktok-floods-teens-with-eating-disorder-videos-b20c2c73">eating disorders</a> and <a href="https://gifct.org/wp-content/uploads/2021/07/GIFCT-CAPI1-2021.pdf">violent radicalization</a>. It has also been <a href="https://static1.squarespace.com/static/5b6df958f8370af3217d4178/t/613a4d4cc86b9d3810eb35aa/1631210832122/NYU+CBHR+Fueling+The+Fire_FINAL+ONLINE+REVISED+Sep7.pdf">argued</a> that recommender systems cause society-wide polarization — different from radicalization which involves only a small number of people — however the evidence for this is <a href="https://arxiv.org/abs/2107.04953">mixed</a>.</p><p>It is natural for users to instinctively react to the outrageous, the extreme, and the sensational. However, this can lead to recommendation engines misinterpreting user activity by assuming this is the content users value. <a href="https://par.nsf.gov/biblio/10066038-returning-believing-optimizing-long-term-user-engagement-recommender-systems">Experiments</a> have found metrics based purely on immediate feedback signals are insufficient and do not represent long term value to users. This problem is closely related to the difficulty of <a href="https://medium.com/understanding-recommenders/what-does-it-mean-to-give-someone-what-they-want-the-nature-of-preferences-in-recommender-systems-82b5a1559157?postPublishedType=repub">inferring preferences from behavior</a>.</p><h3>If not engagement, then…</h3><p>Many <a href="https://www.theringer.com/tech/2018/2/27/17056320/snapchat-redesign-chronological-internet-twitter-instagram-facebook">users</a>, <a href="https://journals.sagepub.com/doi/pdf/10.1177/2056305119879671">creators</a>, and <a href="http://webfoundation.org/docs/2018/04/WF_InvisibleCurationContent_Screen_AW.pdf">analysts</a> advocate for alternatives to engagement-based ranking, most notably a reverse chronological timeline. Both <a href="https://www.buzzfeednews.com/article/alexkantrowitz/twitter-to-introduce-algorithmic-timeline-as-soon-as-next-we">Twitter</a> and <a href="https://business.instagram.com/blog/see-the-moments-you-care-about-first/">Instagram</a> moved away from a chronological feed to an algorithmic feed in 2016, which Instagram <a href="https://business.instagram.com/blog/see-the-moments-you-care-about-first/">said</a> was to prioritize “the moments we believe people will care about the most”. Because of user feedback both re-introduced the chronological timeline (<a href="https://news.sky.com/story/twitter-to-bring-back-chronological-timeline-heres-how-to-get-it-11501468">Twitter in 2018</a>, <a href="https://www.theverge.com/2022/1/5/22868889/new-instagram-chronological-feed-algorithm-tests">Instagram in 2022</a>), though the default home feeds remain algorithmic.</p><p>Yet chronological feeds are not a reasonable replacement for recommender systems. First of all, they only work for platforms designed around “following” a small set of sources. A feed of all new items would be near useless for Spotify, Netflix, YouTube, Google News or Amazon. Even where chronological feeds make sense, they come with their own issues. Frequent, spammy posters have an advantage, and posts are quickly lost to time. Instagram <a href="https://www.vox.com/2018/6/2/17418476/instagram-feed-posts-algorithm-explained-reverse-chronological">claims</a> that with the algorithmic feed, “people see 90 percent of posts from their ‘friends and family,’ compared to around 50 percent with the reverse chronological feed.” Instagram <a href="https://www.vox.com/2018/6/2/17418476/instagram-feed-posts-algorithm-explained-reverse-chronological">also found</a> that users spend more time on the app with the algorithmic feed compared to the chronological feed, which is likely more profitable, and may or may not represent genuine value for the user. On the other hand, a <a href="https://bigtechnology.substack.com/p/facebook-removed-the-news-feed-algorithm">Facebook experiment</a> found that users spent more time on a chronological feed than an algorithmic feed, but clicks and shares went down and users complained of low quality content. This illustrates that there are many different kinds of engagement, and they won’t all go up or down together.</p><p>Meanwhile, Reddit relies on a slew of <a href="https://web.archive.org/web/20160312050315/https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9#.gar13jms4">different ranking algorithms</a> — “New”, “Hot”, “Top”, and “Rising” — each of which orders items slightly differently. “New” is reverse chronological, while “Hot” is the difference between upvotes and downvotes on a logarithmic scale (the first ten votes are weighed the same as the next hundred) plus a decaying bonus for recency. Voting is also a behavioral response under our definition, which means even this relatively straightforward approach is an example of engagement-driven algorithmic ranking. However, some types of engagement, such as votes and likes, come from controls designed to solicit feedback, while other types of engagement, such as clicks and watch time, are more implicit. In general <a href="https://dl.acm.org/doi/10.1145/1869446.1869453">both implicit and explicit</a> engagement signals provide distinct and useful information to recommenders.</p><p>Implicit signals can be ethically dubious when users have little control or understanding of the data being collected and how their interactions on a platform are interpreted. Many users are <a href="https://www.tandfonline.com/doi/full/10.1080/1369118X.2020.1736124">not aware</a> that an algorithm exists at all. Despite the possible benefits of maximizing engagement, users might be uneasy about the <a href="https://www.wsj.com/articles/how-does-my-phone-know-my-thoughts-1531680602">specificity</a> of recommendations, or they might be uncomfortable with the <a href="https://www.pewresearch.org/internet/2018/11/16/algorithms-in-action-the-content-people-see-on-social-media/">data collected</a><a href="https://www.pewresearch.org/internet/2019/01/16/facebook-algorithms-and-personal-data/"> or profiles created</a>, which are not necessarily accurate representations. Instead of asking whether or not a platform optimizes for engagement, it can be more helpful to ask how explicit or implicit these engagement signals are, and provide users with more controls to allow them to tweak their online experience.</p><p>Finally, it is possible to engineer a content selection algorithm which evaluates each item based on pre-trained or hand coded proxies for important characteristics such as credibility, informativeness, and perspective, without using data collected from users. The news aggregator <a href="https://www.thefactual.com/how-it-works/">The Factual</a> ranks content without any sort of likes, clicks, or voting signals, though not using engagement signals at all is unlikely to work for other domains like social media and music recommendations. Still, not all optimization is algorithmic. The Factual must still tune its content selection algorithm to create a product that keeps users coming back.</p><h3>Building On Engagement</h3><p>Despite these issues, optimizing for engagement still <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">forms the core</a> of content recommendation on most platforms. While it can be important to offer alternatives, engagement is just too useful a signal of multi-stakeholder value to give up. Instead, it may be possible to get better at distinguishing “good” from “bad” engagement.</p><p>No real platform optimizes solely for engagement. Other factors play into content selection, including removing items that violate platform policies (often called <a href="https://dl.acm.org/doi/10.1145/3394486.3409548">integrity</a> work), <a href="https://transparency.fb.com/features/approach-to-ranking/types-of-content-we-demote/">down-ranking</a> certain kinds of objectionable content to minimize its spread, and implementing <a href="https://www.getrevue.co/profile/caseynewton/issues/new-ideas-for-fighting-covid-19-misinformation-272134">circuit-breakers</a> to curtail the spread of viral content until a human reviewer can weigh in. It is also widely understood that prioritizing short-term engagement can <a href="https://par.nsf.gov/biblio/10066038-returning-believing-optimizing-long-term-user-engagement-recommender-systems">reduce user retention</a> in the long term. Sensationalized headlines, explicit images, or clickbait conspiracies might capture user attention, but can damage user trust or lead to unhealthy relationships with the system. The healthy use of engagement signals requires continuous product iteration, driven by ongoing assessments of how well the recommender is serving users and other stakeholders.</p><p>Fundamentally, engagement is behavioral and can only imperfectly capture the emotional, cognitive, and life experiences of the user. Continuous user research including <a href="https://about.fb.com/news/2019/05/more-personalized-experiences/">surveys</a>, interviews, focus groups, and A/B tests can illuminate existing issues and reveal new ones. For example, Facebook’s data scientists <a href="https://www.washingtonpost.com/technology/2021/10/26/facebook-angry-emoji-algorithm/">found</a> that the “angry” reaction was used more frequently on “problematic posts,” and was “being weaponized” by political figures. It may be useful to focus on specific subsets of users, such as users whose engagement levels have dropped off, or users exposed to a disproportionate amount of content later <a href="https://www.niemanlab.org/2021/06/shadow-bans-fact-checks-info-hubs-the-big-guide-to-how-platforms-are-handling-misinformation-in-2021/">demoted or labeled</a>. The insights from such research can be used to tune the content ranking <a href="https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a">value model</a> to prioritize more reliable indicators of quality, whether behavioral or not.</p><p>Engagement, for all its flaws, will be with us for the foreseeable future. If a system has no engagement — no users, no new posts, no activity whatsoever — it might as well not exist. But there are choices other than “optimizing for engagement” and “not optimizing for engagement.” Careful use of engagement signals, including ongoing evaluation of what each type of engagement represents, can create better user experiences and greater value for all stakeholders, including content consumers, content creators, advertisers, the platform itself, and society at large.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5abaac021851" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-recommenders/whats-right-and-what-s-wrong-with-optimizing-for-engagement-5abaac021851">What’s Right and What’s Wrong with Optimizing for Engagement</a> was originally published in <a href="https://medium.com/understanding-recommenders">Understanding Recommenders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>