In the previous posts, we described our initial observations about Google Chrome’s FLoC Origin Trial, notably its low volume that made the experiment not very representative. Moreover, we highlighted a method to understand what’s behind the opaque FLoC ID, which is essential for marketers to engage audiences based on floc_ids. Beyond this interpretation layer, we wish to outline our point of view on the performance and privacy guarantees brought by FLoC, as well as the key actions Chrome needs to consider in order to make FLoC a viable solution for marketers to have effective digital advertising across the open web.

TL;DR

• It is still unclear which marketer needs and use cases FLoC intends to serve, and how well or poorly it might meet those needs. The experiments conducted so far did not bring clear proof of performance anywhere close to comparing with third-party-cookies-based solutions.
• Despite its promises, FLoC left major user concerns untackled: fingerprinting, cross-sites tracking and sensitive category targeting/discrimination still persist with FLoC, and for some of these concerns it may actually be worse than third-party cookies.
• The future of FLoC remains blurry, and Google must bring answers on key topics, among which supporting verifiable evidence on FLoC’s performance; if renewed, a more global and representative public trial; and finally, explanations regarding the sustainability of FLoC’s economic model, given the difference in value perceived and given to the system by the different stakeholders.

What are the performances marketers can expect from FLoC?

When looking at the official ‘What is FLoC?’ post, three distinct use cases covered by FLoC are presented, that are currently covered by third-party cookies, in a way that supposedly better protects user privacy. FLoC IDs could be used as features in a bidding model to optimize conversion rates; as recommendation features, to recommend specific content to users based on their floc_id; or as audience segmentation tools, to specifically target users belonging to a (set of) floc_id(s). Typically, the two first use cases need to be tested in a real-world environment, for example with an AB-test framework. However, the volume of the Origin Trial was so low that we found it impossible to run an AB test and reach conclusive results. As a consequence, we rather discuss a few performance-related statements shared by Google to assess the validity of FLoC to allow ad selection.

First, in the previous post, we raised doubts about Google’s announcement stating that “advertisers can expect to see at least 95% of the conversions per dollar spent when compared to cookie-based advertising” using FLoC. While such great performance in the first test seems dubious in itself, we mainly express reservations here because of the lack of transparency provided on the experiment and the data used. As highlighted by many actors in the industry, including ourselves, we find concerning that such a widely publicized 95% figure is based on no reproducible methodology or data to independently justify the claims, while peer reviews actually provide the bedrock of trust, just like in the research community. Critical details such as the volume and the length of the experiment, the actual role third-party cookies continued to play in the test population (to inform frequency capping or bidding optimization), or granular data about conversions, spend and cohorts’ size remain, at this stage, unknown. Google pinpointed the whitepaper published in October 2020 to explain this simulation of FLoC. Yet, nothing in the paper refers to this ‘95%’ performance equivalence.

In a recent paper published in mid-August 2021, which looks like an updated version of the previously mentioned whitepaper, the Google Ads team proposes several clustering approaches for Private Interest-based Advertising, including the SimHash algorithm used by the FLoC API during the OT. The paper notably compares the FLoC API model with the current cookie-based system, by measuring their ability to ‘predict’ a future conversion. The idea is to associate to each cohort an interest-based profile, based on the top 10 publisher domain categories (Sports, Electronics, Home, etc.) its users have visited over a period of 7 days, and determine if these users actually converted on a domain tagged with one of these categories in the following days. Precision and Recall rates are then computed for all cohorts to determine the relative performances of each cohort model (random cohort assignment, FLoC API, and cookie-based singleton cluster). The paper comes up with two takeaways we wish to challenge:

• “The fully decentralized SimHash algorithm achieves a 4x improvement over a random baseline (in terms of precision). These results indicate that interests are preserved through clustering.”

We believe these results should not be overinterpreted in their meaning: As SimHash uses the historical domains browsed by users to assign their cohorts, it is logical that the FLoC API achieves a better precision than random assignment. The contrary would suggest that the domain or domain category on which a user has converted is totally uncorrelated to the previous domains or domain categories the user has visited in the 7 previous days, which sounds unreasonable. Moreover, marketers do not spend their money at random. Hence this baseline is irrelevant for evaluating the utility of FLoC.

Besides, we consider that the methodology described above has major drawbacks: the conversions considered are conversions either led by the current advertising solutions — which are cookie-based — or by no advertising at all; not by cohort-based advertising. The goal of advertising, either in a prospecting scenario or a consideration scenario, is to encourage users to convert on a marketer’s website while they would not have converted without seeing ads. This is also known as incrementality. However, in the experiment conducted in the paper, Google did not compare the number of sales by users that saw FLoC-based ads vs non-exposed users; they just looked at the category of domains on which users converted, an compare to the previous categories seen in the respective cohort versus completely random categories. As such, the demonstration does not tell the concrete performance, or the ‘predictive power’ to employ Google’s own words, brought by FLoC to marketers so they can optimize their conversion-per-dollar-spent ratio. It just shows that the conversion history of a user is linked to its historical browsing history.

• “By grouping users with similar interests together we can achieve a better recall than the singleton cluster baseline. This suggests that the FLoC API could be used as a way to expand the reach of campaigns that are trying to target a specific audience.”

We must again take into account the methodology used to deliver these results. As stated above, the cohort interest profiles were built using the top 10 publisher domain categories browsed by the users within the cohort. Even though we do not have extended details about the granularity defining these categories, one could argue that a set of 10 distinct publisher categories is too broad to build relevant audiences for marketers. By restricting to only the top 1 or top 2 domain categories, the results might have been different.

In order to compare these results, we have conducted a similar analysis, by weighting the advertiser verticals (~100 distinct verticals ranging from FLOWERS/GIFTS/GREET to ONLINE GAMING) browsed by users using 7 days of browsing history. Just like in the paper, we compared the recall rate of conversions happening in the next 7 days, between the top categories of each cohort’s centroid versus the top categories of each singleton cluster, i.e. the individual shopping intent of users on the advertiser websites where we are directly integrated (~20,000 marketers). Looking at advertiser verticals rather than publisher categories seems more relevant to us since a buying intent and a conversion are actually linked to advertisers’ websites, not publishers.

We made two observations:

• By considering the top 10 categories, we cannot confirm the positive difference in recall rates between a cohort-based approach and a singleton cluster
• When restricting the number of categories to define a user interest, the difference in recall rate becomes strongly negative for FLoC cohorts versus individual shopping intent history.

These results show that the cohort-based system proposed by FLoC actually is less efficient at representing user interest, especially when restricting the user interest profile to only a few categories. In contrast, marketers would rather use a few categories only to build performing interest-based campaigns and optimize ROI.

What about privacy?

One key goal of FLoC is to propose a viable framework for interest-based advertising while preserving the privacy of individual users and thus should meet some of the use-cases covered today by third-party cookies. In that respect, we must specify that Google, with its FLoC proposal, actually tackles privacy concerns related exclusively to the advertising market. FLoC does nothing to address true privacy risks on the web, like the leak of sensitive information linked to user identity (e.g. credit status, sexual orientation, medical records) or the use of data to discriminate people (e.g. in a job selection process). Instead, by only looking at the advertising market, FLoC tackles a case where the identifiers (pseudonymous IDs), the data at stake (shopping data) and the ultimate goal (deliver a marketing message) do not present risks of anything approaching the same criticality as those outlined previously. Beyond these initial considerations, there are also major privacy-related questions in the use cases FLoC pretends to solve directly:

First, one must admit that FLoC does not entirely prevent web actors to individually identify shopping histories, even though they are grouped in cohorts with others. Although there should be some k-anonymity mechanism enforcing a minimum cohort size in FLoC (we could not verify the 2000 Chrome users threshold announced by Google during the Origin Trial), we need to take into account the users’ device properties, that can provide a much thinner granularity when combined with FLoC IDs. As stated by Mozilla in their Technical Comments on FLoC Privacy paper, “It is possible that some fingerprinting inputs are strongly correlated with FLoC cohorts. For instance, people with the same interests might also tend to buy the same computers, use the same browsers, and apply the same customizations.” By crossing floc_id information with IP, device, environment or any user agent’s properties, the cardinality of unidentifiable users reduces drastically. Google may answer these concerns with their Privacy Budget initiative; however it is still in design, and we have very little information about its specifications so far.

Second, FLoC can still be used as a tool for cross-site tracking. Imagine a user changes floc_id over a 2-week period, for example going from cohort 125 to 456 and then from 456 to 6798. This cohort pattern can be observed on a given website A, and then on another website B. It is very likely that such a pattern describes the exact same user, and thus be used as a cross-site tracking ID. This mechanism is known as the longitudinal privacy question. In this GitHub issue, Google proposes several mitigations, like adding noise to each browsing history, or making the floc_id sticky for a given website, meaning the cohort ID won’t be updated once it has been called on a given website, or at least with a reduced refresh rate. All of these solutions introduce a utility/scale tradeoff and still need to be detailed by Chrome.

Third, a common criticism associated with FLoC is the sensitive category targeting. Since FLoC groups users based on their browsing history, we can imagine cohorts regrouping users from a clear demographic group like minorities based on the content they browse. By analysing the domains visited by these cohorts and interpreting them, web actors can use the resulting connections to make discriminatory decisions (e.g. refuse a consumption credit based on FLoC ID). In a specific whitepaper, Google Chrome intends to tackle these concerns by looking at all domains visited by each cohort, and block cohorts containing a large ratio of users visiting one or more domains from a considered ‘sensitive’. While it’s a good sign Google acknowledges the issue, we believe there are some major limitations to this approach:

• First, such an approach implies that Google would have access to each individual browsing history of users within a cohort ID, to make a performing audit of sensitive categories browsed, which goes against the privacy-preserving principles of FLoC, not to mention the computation resources needed.
• Second, the method considered in the whitepaper oversimplifies the classification of sensitive, by looking at the sensitive domains considered as such (drugs.com, or guns.com for example). However, a sensitive user pattern cannot necessarily be identified through the lens of the domains visited: persons suffering mental illness may not visit such explicit domains like ‘mental-illness.org’”. Moreover, no automated classification, nor any manual annotation will ever be perfect and allow to cover the diversity of understanding of what sensitive means across time and geographies. There will be cracks and they will be exploited. What will be Google’s answer in the face of a PR storm caused by FLoC?

The economics of FLoC

If the industry is to build on top of FLoC, it must also believe in its economic viability for all actors. The value exchange is a complex and vital aspect of such a project that we think has been largely ignored until now.

We believe that the balance between the value received and given to FLoC greatly depends on the type and size of the stakeholders considered. As a small specialized publisher (e.g. organic-food-blog.com), my domain contributes significantly to FLoC, because it brings very specific information about the people visiting my site. In contrast, visiting facebook.com or youtube.com brings close to zero value to FLoC, because the vast majority of Chrome users visit those domains anyway. Yet, the value extracted from the “coop” does not follow the same pattern; quite the opposite. Big publishers, with higher average CPM, will extract more value than small ones.

While this effect may be mitigated was the FLoC assignation algorithm to take into account more granular signals than the top-level domain +1 (TLD+1), reaching a fair balance will prove challenging.

Besides, advertisers have little incentive to continuously opt-in FLoC: if FLoC scales up, opting-out of computation for 90% of marketers’ traffic would still allow visibility on which cohorts visit their websites. For the rest of their visitors, they might not participate in FLoC (because they have no clear incentive to do so), but still gain value from the whole proposition, thanks to publishers. And here again, a small specialized advertiser added more value to the pool compared to a giant retailer, selling a vast range of product categories.

Finally, we are concerned that the fixed data-cooperation/opt-in policy for FLoC will not fit the wide range of actors who will benefit and serve FLoC in very various scales. Advertisers and publishers currently enter data cooperation models under many different ins and ours that can suit each individual situation. For example, an adtech provider can let an advertiser or a publisher share its data to benefit from the whole network, but with specific rules, like not sharing data with its direct competitors. We fear that the lack of flexibility of FLoC in that regard will make it unacceptable or irrelevant to many.

One could think of solutions to measure the value brought by each actor and redistribute the total value created by FLoC accordingly. However, no trivial solution exists and we believe that this should be a focal point in future discussions about FLoC.

Conclusion

Google already announced that the Origin Trial is over for now and that Google’s teams would work on “improving FLoC to incorporate the feedback we’ve heard from the community before advancing to further ecosystem testing”. The Origin Trial was supposed to be an opportunity for the industry to familiarize with the real feature in a real-world environment. While we were able to technically collect FLoC IDs, the test was vastly different from what we can expect as a final product, because of a very low volume (we only saw 0.02% of users with a floc_id on Chrome 91+). Overall, we can list a few critical actions that still need to be tackled by Google Chrome’s teams following this first public experiment:

• Google should be more transparent in the next changes to be expected in a future version of FLoC.
• FLoC suffers from the lack of transparency and clarity on the expected performance in addressing specific use-cases. The recent consideration to switching FLoC to a topic-based approach is clear proof of that.
• A future public experiment of FLoC should be more representative of a hypothetical production version. A wider and more representative experiment also entails the extension to all geographies (As a reminder, the FLoC Origin Trial was not opened to EU countries, presumably because of GDPR law obstacles).
• Finally, Chrome should ponder FLoC’s relevancy in its current form to answer marketers’ needs for interest-based ads. FLoC is built using top-level domains +1 that don’t easily align with commerce-related audiences (e.g. shoe lovers) or socio-demographic characteristics the marketers use. In addition, a person’s interest is rarely unidimensional across contexts, times of day, etc. Trying to assign a single attribute to a browser is not likely to be very useful for marketers trying to engage the right audience in the right context at the right time.

In this series of posts, and despite the major drawbacks inherent to the volume available during the Origin Trial, we have listed a few characteristics of FLoC cohorts so the ad tech industry could better understand what this proposal means in practice. While we recognize this public experiment was a good opportunity for the market to test the solution and give feedback to Chrome, we consider that it was not close enough to the final product to draw any meaningful conclusion. On September 30, 2021, Google announced that both FLoC and FLEDGE will be available for wider testing by sometime in Q1 2022. Like for other proposals, including FLEDGE, the industry now expects more collaboration and transparency in order to build cookie-resilient and privacy-preserving solutions, that could be the foundations of future advertising.

Article written by Pl Mrcy and Antoine Rouzaud

If you missed our previous articles on the topic, head to the below summary:

• FLOC Origin Trial: What We Observed (1/4)
• FLoC Origin Trial: How Do FLoC Cohorts Behave? (2/4)
• FLoC Origin Trial: Understand FLoC Cohorts (3/4)

Interested in what we are working on? Check out our open roles:

Careers at Criteo | Criteo jobs

FLoC Origin Trial: Questions and Perspectives (4/4) was originally published in Criteo Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the two previous posts, we saw how challenging it was to draw meaningful conclusions on FLoC cohorts and to interpret them, because of the low volume and representativeness of the experiment, as well as the important inconsistency of users’ membership in cohorts. Still, we noted that it may be possible to analyze FLoC IDs through the lens of the domains they have been flagged in. Even though a floc_id does not have any meaning per se, one way marketers might use the opaque IDs provided by FLoC is to use them as a form of contextual targeting, by looking at what domains certain IDs visit. These domains can provide some understanding of what types of user activity are represented by each cohort, and whether the same cohort ID represents the same interest over time.

We will thus try to give an interpretation to FLoC cohorts, and determine how well can FLoC cohorts serve the marketers’ needs for matching ads to people and optimizing their return on ad spend.

TL;DR

• FLoC cohorts are built on top of different domains that change over time, and thus these domains cannot be sufficient to exhaustively characterize a FLoC cohort in terms of user interest over time.
• FLoC algorithm hardly regroups similar users in terms of browsing history; only the biggest cohorts show good enough similarity, but do not represent the majority of cohorts observed during the Origin Trial.
• Regrouping the domains browsed by topic categories may enable one to temporarily interpret the opaque FLoC numerical IDs.

FLoC cohorts and domains

As explained in our first blog post, the FLoC algorithm uses the domains visited by Chrome instances to cluster similar users in the same cohort. Looking at the relationship between floc_id and domains, we can answer several questions:

• Is a domain visited by the same (set of) floc(s)? If so, websites’ owners may be concerned about some FLoC IDs ‘stealing’ their audiences. But they can also gain knowledge about what kind of users land on their webpage, to tailor the content.
• Does FLoC group visitors from the same domains? If so, one could very easily decrypt FLoC cohorts’ meaning, by looking at the websites they visit. It may also raise concerns in terms of privacy and competition.

First, by computing the number of distinct floc_ids visiting a domain per week, we observed the data was vastly skewed by the volume of the Origin Trial: we could only find a few users with a floc_id on the vast majority of publisher and advertiser domains, thus a small number of distinct cohorts for most sites across the open web. In contrast, some rare domains drove an enormous pool of flocs. As an example, weather.com was visited by ~4,000 cohorts per week; washingtonpost.com approximately ~1,600 per week, while the overall median was only 2 cohorts per week for all domains. This can be expected as visiting these vastly popular websites does not hint at any specific intent/profile (“everyone” is checking the weather and weather.com is one of the top providers of weather forecasts).

Still, if we only restrict to the domains visited by at least 100 users with a floc_id per week, these domains were visited by 122 FLoC cohorts per week in median. We compute this figure on a weekly basis because FLoC cohorts are recomputed every 7 days, and the vast majority of users have changed cohorts after this period, according to our second blog post. The cumulative curve proposed in the graph below suggests that the FLoC IDs visiting a given domain tend to change, and thus a domain is not constantly visited by the same cohorts. As a consequence, it may be harder for big domains’ owners to get insights about the user profile browsing their site, as the amount of FLoC to interpret becomes bigger and bigger.

Second, by looking at all the domains visited by users within a given floc_id, we can interpret what kind of user interest does the cohort represent. We can also determine whether a FLoC cohort correlates to one or a few websites’ desired actions, such as an eCommerce conversion. We have observed that the median FLoC cohort has visited 7 distinct domains per week, advertisers (57%) and publishers’ websites (43%) equally represented. Again, in a scaled production version of FLoC, this figure should be higher.

Interestingly, the domains visited by FLoC cohorts are not consistent from one week to another: on the whole period (June 1st — July 13th), the median number of distinct domains a FLoC cohort has visited is 43 (in contrast with the 7 weekly distinct domains), which means that the domains representing a FLoC cohort tend to change over time. This confirms Google’s statement “a cohort represents a cluster of browsing activity, not a collection of people”. As such, the cohort id does not represent a predefined set of domains, or set of people, that would be fixed across time. Since users’ browsing habits evolve over time, it makes sense that the set of domains visited by a floc_id varies. But it also suggests that looking at the domains visited by a cohort on a given week is not enough to characterize the general browsing pattern of its users. Moreover, it means that marketers will not be able to use cohorts seen on their own domain to identify other domains to attract new customers, given the cohorts are not consistent over time.

Now that we have understood that the domains visited by cohorts change over time, we need to verify whether these domains are similar, meaning they provide similar content and thus could be used to interpret the cohorts in terms of user interest profile.

Do FLoC cohorts group similar users?

As described previously, the number of distinct domains visited by each FLoC is pretty low per week (the median is 7) but tends to change over time, and thus could not be used as absolute common characteristics to define what a cohort represents in essence. But then, how can we assess the similarity between users grouped in the same FLoC cluster? Are we able to find common and durable patterns that help us to define a given cohort?

In a whitepaper written by Google Ads and Google Research teams, Google presents different ways for implementing and evaluating cohort assignment algorithms, like PreFixLSH, the solution used during the FLoC Origin Trial. Google states that the experiments conducted by Google Ads and Research in this paper performs rather well for some types of ad targeting: “Affinity Audiences” (like “Cooking Enthusiasts”) and “In-Market Audiences” (like “people actively researching Consumer Electronics”). Google’s ad blog goes even further, stating marketers should expect to see “95% the effectiveness in performance advertising using FLoCs relative to any tactic supported by “cookie-based advertising”. Assessing the robustness of these audiences in practice with our logs is essential to determine whether these audiences can really bring performing campaigns.

In the same paper, Google proposes several representations of users’ browsing history, that can then fuel the FLoC algorithm and group users together:

• Domain One-hot encoding
• Domain TF-IDF encoding
• Topic categories

Out of the 3 methods proposed, “topic categories” brings the closest user similarity within the cohorts. Concretely, the idea is to categorize the websites browsed by the user into topics. For instance, expedia.com is a travel website. Similarly at Criteo, we have several categorisations layers used to segment both advertisers and publishers, depending on the products they sell and the content they show respectively. As such, just like Google, we can use our own categorisation to represent each browsing history of a user as a sparse vector where each entry in the vector corresponds to the average weight of a topic category across all URLs visited by a user during its time in a floc_id. Using a different categorisation tool, Criteo is able to assess from an external point of view the relevancy of the clusters generated.

Thanks to this embedding, it is possible to compare users of the same cohort and determine how close they are to each other. In order to do so and measure the quality of a cluster, as proposed in the whitepaper, we can use ‘the average cosine similarity (or normalized dot product) between all users in the cluster and the centroid of the cluster’. The metric is neatly bounded between 0 and 1: 0 if users have nothing in common, 1 if they are identical.

For the maths enthusiasts, cosine similarity is measured as the cosine of the angle between two vectors and determines whether two vectors are pointing in roughly the same direction. If x and y are two vectors, like in our case the vector of user X and the centroid of its floc_id, then the similarity between these two vectors will be computed as:

We computed this similarity score for all FLoC IDs, by looking at the browsing history of the users flagged within this cohort, and categorise the visited domains using a list of around 40 categories, ranging from Books & Literature to Automotive. The cosine similarity is calculated week after week for each FLoC in a cumulative way (e.g. on week 3, we compare all the browsing histories of users when they were flagged in the cohort, from its creation to week 3). As such, we can estimate how stable a FLoC’s meaning is over time, while the number of users having passed in this cohort grows. We observe that FLoC cohorts’ similarity score hardly reaches 0.5 after a few weeks, which is considered as a minimum bar of usefulness in the industry.

However, the results are more encouraging when filtering on the cohorts with a lot of users, which may look closer to a production version of FLoC, if Google decides to scale its solution. Indeed, if we only keep the cohorts for which we have flagged at least 100 Chrome users during the whole period analysed (~8% of FLoC cohorts), the cosine similarity stabilises above 0.75.

We can conclude that the similarity between users proposed by FLoC cohorts shows promising results only when considering the biggest cohorts; but this statement cannot be generalized to the whole pool of floc_ids created during the OT, because of the low volume observed.

Can FLoC cohorts be interpreted easily?

Despite the mixed results observed in the similarity score, we attempted to use these topic category representations to show the qualitative interpretation of FLoC cohorts, using word clouds. These word clouds are semantic interpretations of the main categories browsed by cohorts and could be used to infer the user interest profile contained in them. They also echo recent discussions around the possibility of assigning topic categories to domains and people rather than assigning opaque FLoC IDs to them.

Below is an example of a cohort for which we manage to interpret a sort of user profile hidden in the floc_id: the cohort seems to mainly contain gaming enthusiasts, interested in video games and electronics. Its interpretation seems fairly consistent over time, by keeping the same main topics across time. Still, one can argue that these interpretations layers are too broad to deliver actionable insights to advertisers, that aim to target users interested in a specific set of products rather than domain categories, while FLoC is actually built on top of top-level domains.

However, the task is not that easy for smaller cohorts that represent the majority of FLoC audiences during the OT — the median number of Chrome users flagged on publisher and advertiser domains during the period analysed is only 23, which is way below Google’s threshold of thousands of users. In the example below, we see the words cloud drawn does not provide meaningful interpretation of the cohort’s user interest and tends to change over time.

Conclusion

If the Adtech industry is to use FLoC to build audiences to target, it becomes essential for marketers to understand what’s behind the opaque cohort ID assigned to each group of users. We showcased one method to so, that proved successful for some of the biggest cohorts. However, the low volume was once again a blocker to validate that it worked for a majority of cohorts. Besides, the granularity provided by such understanding calls into question the utility of this approach for real world marketer needs, that would rather use more granular product-based interest profiles to build relevant audiences.
The performances brought by these interest-based audiences compared to third-party cookies remain highly debatable. In the next and final blog post, we will discuss our remaining questions and doubts about the use cases covered by FLoC, as well as the metrics used to assess its performance.

Article written by Pl Mrcy and Antoine Rouzaud.

Read more from the series here:

• FLOC Origin Trial: What We Observed (1/4)
• FLoC Origin Trial: How Do FLoC Cohorts Behave? (2/4)

Interested in joining the fun? Check out our open positions:

Careers at Criteo | Criteo jobs

FLoC Origin Trial: Understand FLoC Cohorts (3/4) was originally published in Criteo Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In our first post, we introduced Google’s FLoC Origin Trial, the first experiment of Google’s FLoC proposal to solve the marketers’ need for audience-based advertising by providing an alternative to marketer-defined audiences. We presented results about the coverage of the Origin Trial, as well as the volume of users in FLoC cohorts. We observed that the FLoC Origin Trial was conducted on a very small portion of the Chrome traffic, which may not be representative of the global Chrome traffic. However, thanks to our direct integration with a wide number of advertisers and publishers, we were able to observe some interesting properties of how FLoC cohorts change over time, as well as first insights on how they represent a user’s intent.

TL;DR

• FLoC cohorts are recomputed quite frequently, once every 7 days on average.
• FLoC cohorts are unstable in terms of users composition: only 12% of Chrome users are still in the same cohort after 7 days.
• We did not observe a ‘closeness’ pattern between FLoC IDs, thus we could not confirm that FLoC was able to capture an user’s behaviour.

How often do FLoC cohorts change?

We wish to determine whether FloC cohorts are stable over time by containing the same users. If so, FLoC cohorts could serve marketers that try to expose a consistent audience of users.

Refresh rate

A first way to determine if FLoC cohorts are stable is to know how often does the FLoC algorithm recompute the cohorts assigned to Chrome users. As shared by Google, “the user’s browser is associated with one interest cohort at a time and recalculates its cohort periodically (currently once every seven days during this initial origin trial) on the user’s device”. We want to verify this statement by estimating the recomputation cycle length with our logs. We cannot know the precise computation period on all the users with a floc_id; but we can estimate it thanks to the users that changed cohort during the period we analysed.

In practice, we could find users that change cohorts with a higher or lower frequency than 7 days. This discrepancy can be explained by several factors:

• Users may not use Chrome during several days and thus are not flagged the day they change cohort, so we overestimate or underestimate the actual cycle length
• Users may visit domains that opt-out the FLoC OT, or domains where we do not have tags firing, so we overestimate or underestimate the actual cycle length

To mitigate these effects, we can filter these users out of the analysis. We only keep the users that did change cohort, and that we saw continuously between this change date and the previous cohort change.

We see that, out of all these users that changed cohort during the period we analysed, 91% of them did it on the 7th day. However, we do observe some users that change cohort with a higher or lower frequency, like 10 days, or 1 day.

As an example, below we show how a user’s cohort evolves over time. Even though the user was seen in the same cohort from June 2nd to June 11th with the floc_id ‘9306’; he then changed cohort on June 12th, after 10 days in the same cohort.

Besides, even though we observed cohort changes happening every day, we see higher frequency of them happening on Saturdays. Moreover, there seems to be a particular ‘reset’ of cohorts on Saturday June 26th, when almost all users with a change shorter than 7 days have changed cohort on this day. In contrast, there was no clear date pattern in the way users with a longer period change cohort.

Finally, the refresh frequency does not seem to be defined at the user level: If we only consider the users with a cycle length different from 7 days, the average standard deviation of days elapsed per user is 2.4 days. This tells us that a given user may have different cycle lengths during its FLoC journey.

Churn rate

Based on these observations, we have computed the average churn rate of FloC cohorts. The churn rate is computed every day as the percentage of users that are seen with a different floc_id on day D+d, compared to the first floc_id they were flagged in on the first day D, whatever for how much time they have been in the cohort before. The churn rate enables us to understand how stable is a cohort over time, in terms of user composition. Note that this is different compared to the refresh cycle of 7 days, which estimates how often a user’s cohort is recomputed: for the refresh rate computation, we did not take into account all the users that did not change cohort, nor the ones that we did not see continuously in our logs, since we could not compute a precise cycle length with those users.

We can observe that cohorts are unstable: after 7 days, only 12% of the users are still in the same cohort. Note that it makes sense that the curve’s slope decreases after 7 days, since users’ cohorts are recomputed every 7 days. But it also shows that very few users actually stay in the same cohort for more than one cycle of 7 days.

This constant fluctuation makes FLoC cohorts less efficient for marketers aiming to expose a consistent audience of users.

Are ‘close’ FLoCs similar?

One interesting property of the unsupervised clustering algorithm used by Google in the Origin Trial, namely SimHash, is that the SimHash’s results that are close enough from each other will have close browsing histories: in other words, there is supposedly a ‘closeness’ property of FLoC, where 2 close FLoC IDs will likely be similar in their users’ composition. Accordingly, while it makes sense that users change cohort_id, we expect that the cohort_ids they are assigned to should be in a similar range, if FLoC indeed captures some basic user interest. As a consequence, a given user ought to fluctuate in the relative vicinity when changing cohort.

Assuming this last statement is correct, then it would show that FLoC audiences capture an user behaviour, that we assume constant, or slowly changing over time. However, we observe that the average difference in floc_id during a cohort change is 9600, whereas the vast majority of floc_ids ranges between 0 and 34000. While we have no doubt the SimHash algorithm indeed has this ‘closeness’ property, then it would mean that FLoC cohorts do not represent efficiently an user’s behaviour with some stability. It also makes the clustering of consecutive floc_ids irrelevant for now to target similar interest groups.

Conclusion

Despite the low volume seen during the FLoC Origin Trial, we were able to determine a few characteristics of FLoC cohorts. We saw that the cohorts tend to refresh quite frequently (7 days) and change in their composition, since users do not stay long in the same floc_id. Moreover, we did not observe a ‘closeness’ pattern in FLoC cohorts’ identifiers, thus we could not confirm that FLoC was able to generate a meaningful audience interest segment. This last observation also shows that the clustering of consecutive floc_ids may be irrelevant for now to target similar interest groups. Still, we need to understand how FLoC cohorts can be used as a viable method for matching relevant ads to users. In other words, how can a marketer choose which audience cohorts they should buy to achieve a positive return on their media spend?

In the next article, we will dig deeper into this challenge of interpreting FLoC cohorts, and understand to what extent a floc_id can be used as a predictive information to build interest-based ads.

Article written by Pl Mrcy and Antoine Rouzaud

Missed the first part? Check it out:

FLOC Origin Trial: What We Observed (1/4)

Want to join in on the fun? Check out our open positions:

Careers at Criteo | Criteo jobs

FLoC Origin Trial: How Do FLoC Cohorts Behave? (2/4) was originally published in Criteo Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

On July 13, 2021, Google ended the first Origin Trial of its FLoC proposal, a key proposal of Chrome’s Privacy Sandbox initiative. The Privacy Sandbox aims at creating a viable framework for advertising across the open web that both respects the users’ privacy while enabling marketers to continue to provide ad-funded access to digital properties. Google hopes FLoC could solve the marketers’ need for audience-based advertising by providing an alternative to marketer-defined audiences. During this Origin Trial, Criteo leveraged its unique assets to understand what these FLoC audience cohorts mean in practice and evaluate how well they work for the AdTech industry to support performance-based advertising campaigns.

In a series of blog posts, we will share what we saw and our main findings with regards to this Origin Trial, starting here!

TL;DR

• Google Chrome is not the only browser that generates FLoC cohorts; Yandex browser represents 88% of our advertiser events logged with a floc_id, predominantly from Russia.
• The volume of the Origin Trial is very low (0.02% of Chrome users) and hardly representative of the global Chrome traffic, since it mainly runs on Beta and Dev Chrome versions.
• Even if we observed more than 33,000 FLoC cohorts, the volume of users within these cohorts is very small, making their interpretation challenging.

Context: what is FLoC and the Origin Trial according to Google?

FLoC stands for Federated Learning of Cohorts and is one of the cohort-based advertising mechanisms proposed in Google’s Privacy Sandbox, among others like FLEDGE. Google hopes FLoC could solve the marketers’ need for audience-based advertising by providing an alternative to marketer-defined audiences.

In practice, a floc_id is assigned to the user’s browser, based on its browsing history. Several users are grouped together in the same floc_id expressing related browsing history, supposedly expressing proximity of interest. Please refer to the official FLoC specifications for more information.

Origin Trial

The Origin Trial (OT) was launched end of March 2021 via a FLoC API, available in Chrome 89 and above and ended on July 13. According to Google, “the initial testing of FLoC is taking place with a small percentage of users in Australia, Brazil, Canada, India, Indonesia, Japan, Mexico, New Zealand, Philippines and the U.S”. More precisely, even within these regions, the FLoC audiences were only available for a small fraction of qualifying Chrome users: 0.5%.

FLoC, as tested during the OT, is only one implementation of the many implementations possible for the same general idea. This implementation is subject to changes in the future. Some of the points that we highlight in this series of posts address this specific implementation, while others relate to the broader concept of FLoC.

The FLoC version used during the OT is built on top of an unsupervised clustering technique called PreFixLSH (note that there is currently no Federated Learning in FLoC). As explained by Google, with this clustering technique, people are more likely to end up in the same cohort if they browse the same websites — meaning the top-level domains. Neither the precise URL nor the content of the pages, nor the frequency of the visits, etc. is used to build these cohorts.

A domain is able to declare that it does not want to be included in the user’s list of sites for cohort calculation by availing itself of the interest-cohort permissions policy. Indeed several big names in the industry expressed vocally their decision to opt-out of the trial (such as Amazon and WordPress). However, the policy is to allow by default.

What did Criteo analyze during the Origin Trial?

From June 1, 2021, to the end of the test we collected two pieces of information related to the FLoC Origin Trial, thanks to our direct integration to advertisers (demand) and publishers (supply):

• floc_id, indicating which cluster the user’s browser is in
• floc_version, the label that identifies the algorithm used to compute that ID.

With a month and a half of data based on 16k+ advertiser websites and 2k publisher websites, we aimed to

1. understand what are the cohorts built by FloC
2. determine to what extent our customers could use them as a potentially viable alternative to third-party cookies.

What we saw

First, we wish to compare what the OT looked like in practice compared to Chrome’s announcements.

“The initial testing of FLoC is taking place with a small percentage of users in Australia, Brazil, Canada, India, Indonesia, Japan, Mexico, New Zealand, Philippines and the U.S”

We have stored all online events on our advertiser websites where the FLoC API was activated and where users were flagged with a floc_id.

By looking at all these events, we actually observed that Google Chrome is not the only browser generating FLoC cohorts; Yandex, the leading browser in Russia — also based on the open-source Chromium project, is also creating FLoC cohorts, using the method described above. Google stated that the “string representation of the interest cohort version (floc_version) is implementation-defined”, meaning that each browser may have its own floc_version. In Google’s own words, “It’s recommended that the browser vendor name is part of the version (e.g. “chrome.2.1”, “v21/mozilla”), so that when exposed to the Web, there won’t be naming collisions across browser vendors”.

As such, many FloC cohorts are generated by Yandex for users from other countries, i.e. not part of the OT list. In total, only 12% of our advertiser events with a floc_id come from countries within the OT list and Chrome. The rest, represented by approximately 300,000 FLoC cohorts, come mostly from Russia and were generated with a floc_version named “chrome 1.0”, which is not the algorithm version tested by Google Chrome (version chrome 2.1). Even if they mention ‘chrome’ in their floc_version, all these cohorts were built by “browsers not running the actual FLoC algorithm that Google has described”, according to Google: “Anything that says “chrome.1.0” is based on someone starting up Chrome and activating the API with their own collection of command-line flags”.

In a nutshell, Criteo ignored these ‘chrome 1.0’ / Yandex FLoC versions when assessing the potential viability of the official Chrome solution. Please note also that while the official Chrome Origin Trial ended, we still receive floc_ids from the Yandex browser.

“The standard usage limit for a trial is 0.5% of all Chrome page loads”

To observe Google’s statements in practice, we restricted our FloC datasets to Chrome-only browsers (desktop and mobile) in the 10 OT countries, based on the country declared by the browser user agent. We looked at what percent of all advertiser events contained any FLoC audience cohorts by each Chrome browser version within the OT countries.

Between June 1 and July 13, 85% of advertiser events that contained a FloC come from Chrome version 92 and 93, i.e. what Google calls Beta Chrome versions. This would be pretty consistent with what was announced in this Github issue, stating that “the OT has not yet started on Chrome’s Stable channel, only for some people on the Beta and Dev channels”. So, the Original Trial did not run using a sample of all Chrome instances, but a subsegment of it being the Beta/Dev versions, which may not be representative of the global Chrome traffic. Indeed, Chrome 91 represents more than 99.8% of the Chrome 91+ traffic.

Besides, the percentage of Chrome users with a FLoC ID varies a lot from one Chrome version to another but is only 0.02% globally on Chrome 91+ (not 0.5% as announced). The very low volume is another reason to say we will hardly get representative results with the Origin Trial.

“The observed number of cohorts created by this clustering algorithm, based on data from qualifying Chrome users was 33,872”

On Chrome instances in the 10 OT countries, we have observed 33,105 distinct FloC cohorts. The FLoC IDs do not contain any letter or other special character; they consist of integers, being unique identifiers, ranging for the very large majority between 0 and 34,000.

“The minimum number of qualifying Chrome users in a cohort is 2000”

The average number of distinct chrome users flagged in each FloC cohort per day is only 1.4, between June 1 and July 13, on our advertisers’ websites. As most of the FLoCs contains very few users on a given date, analyzing FloC audiences over short periods of time may not be representative of the API in general availability (the FLoC would then necessarily represent at least N=2,000 users) and results are tricky to interpret.

Since users may stay several days in a given cohort, this is not much different when taking the weekly average: we have observed only 2.8 distinct users per cohort_id per week on our advertisers’ websites during the Origin Trial.

Conclusion

The Origin Trial was supposed to be an opportunity for the industry to familiarize themselves with the real feature in a real-world environment. While we were able to technically collect FLoC IDs, the test was vastly different from what we can expect as a final product. Indeed, being only available in dev versions of Chrome, FLoC’s volume remained extremely low and not representative of the general population, to the point that drawing any meaningful conclusion is a challenge: a challenge that we will take on in our next post (stay tuned!)

Article written by Pl Mrcy and Antoine Rouzaud.

Stay tuned for the next articles in this series and check out some of our recent publications on Medium!

• Beware of Lambda Captures!
• Privacy-preserving ML Challenge at AdKDD’21 on Criteo’s data

Want to become a writer? Join the team!

Careers at Criteo | Criteo jobs

FLOC Origin Trial: What We Observed (1/4) was originally published in Criteo Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.