Podcast: Quantifying the advertising value of cookies (with Garrett Johnson and Shunto Kobayashi)

On this week’s episode of the podcast, I am joined by Garrett Johnson and Shunto Kobayashi, professors at Boston University who have co-authored a paper titled “Can privacy technologies replace cookies? Ad revenue in a field experiment.” The paper quantifies the impact of Google’s Privacy Sandbox on publisher revenue, user experience, and market competition through a field experiment conducted in collaboration with Raptive, a digital advertising management platform.

Our conversation highlights the specific challenges faced by the digital advertising ecosystem as it moves away from traditional identifiers and toward a new, more restrictive privacy regime. Among other things, we discuss:

• How the loss of third-party cookies specifically affects regional markets with stricter privacy regulations like the European Union
• If the adoption of privacy-enhancing technologies could truly bridge the revenue gap created by the deprecation of cookies
• What role browser vendors should play in establishing technical guarantees that satisfy both regulators and advertising stakeholders
• Why the industry tends to respond to identity restrictions by developing more complex identifiers instead of contextual alternatives
• When the concentration of traffic toward larger platforms will reach a tipping point for small-scale content creators
• How the removal of personalization features on platforms like YouTube serves as a blueprint for future privacy-driven market shifts

We also cover another paper by Garrett Johnson, “COPPAcalypse? The Youtube Settlement’s Impact on Kids Content.”

Thanks to the sponsors of this week’s episode of the Mobile Dev Memo podcast:

• ⁠⁠INCRMNTAL⁠⁠⁠. True attribution measures incrementality, always on.
• Xsolla⁠. With the Xsolla Web Shop, you can create a direct storefront, cut fees down to as low as 5%, and keep players engaged with bundles, rewards, and analytics.
• ⁠Branch⁠. Branch is an AI-powered MMP, connecting every paid, owned, and organic touchpoint so growth teams can see exactly where to put their dollars to bring users in the door and keep them coming back

Interested in sponsoring the Mobile Dev Memo podcast? Contact Mobile Dev Memo advertising.

The Mobile Dev Memo podcast is available on:

• Spotify
• YouTube
• Apple Podcasts

Transcript

Eric Seufert: Welcome to the Mobile Dev Memo podcast. I’m your host, Eric Seufert, and I am joined today by Garrett Johnson and Shunto Kobayashi. Shunto and Garrett, welcome to the podcast.

Garrett Johnson: Hey there, Harvard man.

Eric Seufert: Oh, well, hey, hey.

GJ: Congratulations on your recent graduation of Harvard’s Master’s in Data Science.

ES: Master’s of Applied Computation, and the data science was the specialization.

GJ: Well, congratulations.

Shunto Kobayashi: Congratulations.

ES: Thank you. It is funny because I follow the SEAS, the School of Engineering and Applied Sciences, on LinkedIn. They have these profiles of my classmates where someone is moving to Amazon or Palantir. I thought it would be funny if they had a profile of me stating that I graduated and will continue to write a blog and run a podcast.

GJ: The treatment effect might not be huge here, but you learned a lot.

ES: Right, the increase in wage is zero.

It is really nice to have you both on. You both co-authored a paper which was published recently titled, “Can Privacy Technologies Replace Cookies?” I want to dive into what you found and the methodology. Before we get to that, can you give a refresher about the specific topic of the Privacy Sandbox? Provide some background on the Privacy Sandbox, the actual experiment, and explain the provenance of the data you used. Also, please introduce yourselves as you do so.

SK: I am Shunto Kobayashi, an assistant professor of marketing at Boston University. I am an economist by training with a PhD in economics from Caltech. I work on research about economic and privacy issues in digital markets, especially digital advertising.

GJ: Much of what Shunto said applies to me. I am an associate professor at the Questrom School of Business at Boston University. We are in the same department and share interests in privacy in digital marketing.

This paper studies the impact of Privacy Sandbox. Many have tried to forget about Privacy Sandbox because Google ultimately retired it in October 2025. The big idea was to use privacy-enhancing technologies to have the measurement and targeting benefits of cross-site or cross-app identity in online advertising, while giving better privacy protection to users to reduce cross-site tracking.

There were a bunch of different technologies in the Privacy Sandbox. The most aggressive technology was the Protected Audience API. It inverted the industry; the criticism of the industry right now is that it takes people’s personal cookie IDs and shares them with hundreds of different companies, allowing them to track users across sites. Protected Audience put the entire industry on a user’s device and auctioned off a targeted ad on the device while not allowing that information to leave the browser.

Google retired this in October 2025, but privacy-enhancing technologies still live on. The W3C is proposing an attribution standard that would use privacy-enhancing technologies to provide attribution for the web. Google said it would support this proposal when it retired the Privacy Sandbox.

This created a lot of stir because Google is an important market participant and said they were going to re-architect the industry. The UK Competition and Markets Authority got involved and wanted to understand the impact of these technologies before Google rolled them out to the entire industry. They created an experiment that took 2% of Chrome users and randomized them into three groups: a status quo group with third-party cookies and Privacy Sandbox turned off; a Privacy Sandbox group with third-party cookies turned off and Privacy Sandbox turned on; and a cookieless group that had neither technology.

We were able to study the impact on publishers by collaborating with Raptive, the largest ad management company. They gave us data for this experiment representing over 5,000 publishers who get a couple hundred million visitors a month.

ES: I will note that Paul Bannister, the Chief Revenue Officer at Raptive, is a friend of the podcast. What did you find? What are the headline findings in the research, and how material are cookies to publisher revenues?

GJ: Raptive was an outstanding partner. We found that when you get rid of cookies, ad prices fall by 29%. A bunch of impressions already do not have cookies, so when you look just at those impressions, ad prices fall by 35%. That is significant.

The next question is how well Privacy Sandbox does in recovering those losses. Unfortunately, the news from this research was dismal: it recovered only 4% of this lost revenue. However, it is important to provide context that the adoption of Privacy Sandbox tools by the industry was modest, which puts a ceiling on how much revenue movement you expect to get. You would see more movement if more of the industry adopted it.

ES: One interesting revelation from the paper was the latency that Privacy Sandbox introduced and its impact on publisher economics. The Privacy Sandbox tools mediated the data flow, and enabling these tools introduced latency, which has its own separate effect on economics. Can you talk me through that?

SK: Auctions traditionally run on servers, like ad exchanges. In that process, users’ data can get exposed in the form of bid requests. Privacy Sandbox tries to prevent that by moving the ad auction process into your browser. Google Chrome is doing this auction process on your device, running bidding algorithms. That takes time.

In our dataset, we find that ad delivery takes twice as long under the Sandbox condition. That is roughly two seconds versus one second in the status quo condition that has third-party cookies but not Privacy Sandbox. Users may navigate away or click away from the webpage even before this ad gets rendered, which leads to zero ad revenue for these publishers. This is statistically tricky to capture because we have a dataset on impressions, and by definition, these are not impressions. We had to statistically back this out.

We compared the ratio of Sandbox impressions to non-Sandbox impressions in our dataset to see how it matches the ratio of treatment assignment rates we know from the experimental design. We found a gap coming from ad loss to latency. We estimate the missing rate is about 3%. This means 3% of the time under the Sandbox condition, impressions do not lead to ad revenue for the publisher because users navigate away. This is worse for smartphones because they typically have more limited computation power, leading to longer latency.

ES: That is meaningful. If you told someone they adopted this technology and 3% of their users essentially go to zero, that is a significant loss alongside the decrease in publisher revenue from the loss of fidelity.

GJ: Raptive was frustrated with this phenomenon. Protected Audience API, referred to as “Poppy” in ad tech circles, is good for targeting in a privacy-preserving way, but 3% of impressions going missing is bad.

ES: The paper also finds significant differences in impact between EU and non-EU users. What were those differences, and what explains them?

SK: These publishers get impressions from across the world. When we decompose the effect into different regions, there is a sharp contrast in the cookie removal effect between the US and EU. The damage from removing third-party cookies on revenue is about twice as large on EU traffic.

For impressions that carry cookies, publishers lose roughly 66% of the revenue on their EU traffic versus 33% on their US traffic. Not only that, the share of impressions that carry cookies is lower in the EU in the first place: 57% versus 81% for US users. The reason for both is GDPR. Because of its consent rules, you have to get users’ explicit consent for enabling third-party cookies and potentially using alternative IDs. We have more opt-out users in Europe, and it is harder to deploy alternative IDs there. Consequently, there are fewer alternative IDs available for European impressions.

On the flip side, Sandbox recovers more revenue from EU users. The recovery share is about 23% for the EU versus 2.4% for US users. That is almost a ten-fold difference in the ability of Sandbox to recover lost ad revenue. Why is Privacy Sandbox performing so much better for EU impressions? We think there are two reasons.

First, Sandbox and traditional ad auctions compete for the same impressions. When these publishers launch auctions through traditional channels like header bidding and through Poppy, the Poppy channel performs better or is more pivotal when the traditional side is weaker, such as when there are no third-party cookies or alternative IDs. Since we have fewer alternative IDs in the EU, Sandbox matters more.

Second, the tighter privacy environment under GDPR in the EU might mean European firms had more interest in adopting Privacy Sandbox or privacy-enhancing technologies in general to target consumers in Europe.

ES: We also saw more vocal resistance to the Privacy Sandbox from European publishers. There was a German consortium led by Axel Springer that filed a complaint with the EC. Their argument was anchored to competition, but the point was that this was going to be painful.

GJ: My recollection is they were less opposed to Privacy Sandbox per se and more opposed to getting rid of third-party cookies. Several browsers like Safari and Firefox had already blocked third-party cookies. What changed is that Chrome has such large market share that when they do it, it is even more damaging to these companies.

ES: Paul mentioned that the fact Chrome did not deprecate third-party cookies while Safari did prevented a clean comparison because there was a place for that revenue to flow. You could not look to Safari as the comparison baseline because publishers could prioritize Chrome support, and that would absorb some of those users. If you lost it in Chrome, it would be worse than what you would expect if you just looked at Safari.

GJ: The model behind that argument is that advertisers have a fixed budget. If you have policies that hurt identification for a minority of browsers, the money will switch to the larger browser. However, some of that money is not fixed. Many listeners are performance advertisers, and if you cannot measure performance, you are going to spend less on that channel. The truth is probably somewhere in between, but Chrome being the last big browser standing getting rid of third-party cookies would tip the market substantially.

ES: The experiment takes place in a world where publishers are already using hashed emails and probabilistic identifiers. If those alternatives had been removed, what would happen with revenue? How much larger would the revenue gap be?

GJ: I agree it would be larger. One criticism of research on the value of cookies is that if you get rid of them, the industry would be forced to adapt and innovate. What we are capturing is a sizable revenue effect after there has been a lot of this adaptation and innovation. Raptive’s value proposition is to maximize revenue, and since a third of browser traffic does not have cookies, they have been trying to increase revenue through alternative IDs. The effect we are showing is net of that.

We have other peer-reviewed research where we find that ad prices fell 52% about a decade ago when users opted out of behavioral targeting via the AdChoices program. Similarly, a Google experiment in 2019 found that ad prices fell 52% when it deactivated cookies. The UK regulator re-estimated that and said the loss could be as high as 70%. This is critical from a policy perspective because if the concern is not cookies per se but all forms of tracking, then any policy that maximizes privacy and restricts all IDs would create this larger revenue gap.

When the industry has been squished by getting rid of cookies or mobile identifiers, it does not innovate toward contextual advertising; it innovates toward more identifiers, whether they be probabilistic or deterministic. This undermines the argument that the industry can just innovate away from these things. It shows there is a difficult policy tradeoff: if you maximize privacy, it is going to affect publisher revenue. The UK’s privacy authority, the ICO, released a policy proposal in the last few months that would waive the consent requirement for certain advertising use cases like attribution if they are enabled through PETs. The door is opening to trying to create this compromise way for the industry to function while doing a better job on privacy.

ES: Facebook said the same thing; 50% of the impression value from the Facebook Audience Network is derived from its targeting abilities. The policy proposal from the UK privacy authority is interesting, but I am curious about the conditions on the PET. How do you quantify that to say it offers certain functionality? A regulator seems to be quantifying the privacy value by the reduction in revenue. What is the scope of the PET that satisfies a requirement?

GJ: PETs come with technical guarantees. For attribution, instead of sharing individual data, you share it through a server and put back some noise and aggregate data. This has formal mathematical privacy guarantees when you add differential privacy noise, which should be satisfying to regulators. The cost on the firm side is a big headache when you are trying to use machine learning algorithms to work with this noise data. The ICO also talked about fraud detection and referenced the private tokens approach from Privacy Sandbox, where you get a binary or two-category signal regarding whether a user is real or fraudulent. That has an intuitive privacy definition; there is no detailed personal information.

ES: Regarding the differential privacy approach, one option would be to mandate an epsilon. That was one of the complaints about Apple; they apply DP everywhere, but their epsilon value was 17, which is very large and does not provide those guarantees. Is that the mandate?

GJ: The epsilon term refers to how permissive the privacy-enhancing technology or differential privacy is. If it is very large, you are not changing the data much. It would have to be a dialogue between firms and the regulator. Somebody somehow has to decide what these parameters are, and incentives go in different directions to set them. Nobody wants to be the one to pick the number. DP specifically is very divisive; the Trump administration banned DP this month in the Census Bureau. It is a very thorny policy issue.

Is it realistic that we are going to have user-level data in marketing datasets forever? Eventually, there is going to be enough policy will to get rid of these sorts of datasets. We want to still have some of these benefits of data-driven advertising, so we may have to accept these sorts of tradeoffs. Look at Safari and Firefox; they do not allow any of this data to pass right now. If they get behind this attribution proposal at the W3C, you are actually going to start to see some data coming from these browsers, which is probably good for the industry and for monetizing those users.

ES: Privacy-enhancing technologies often assume an economic tradeoff between privacy and monetization. Based on the findings in the paper, do you think browser vendors have underestimated the magnitude of that tradeoff, or is the current generation of PETs simply not mature enough?

SK: Google definitely knew the tradeoff because Sandbox is by design tracking worse than cookies. We have aggregation and differential privacy, which lead to noise or less granular detail. What they underestimated was the severity of the adoption problem and how it would undercut the overall effectiveness of Privacy Sandbox. It requires two-sided adoption from both the demand side and supply side. Only a few ad tech firms adopted it, which held back the overall effectiveness.

What held back adoption was uncertainty and the cost of adopting Privacy Sandbox. Regulators have been ambiguous about which direction we are going. Google also kept moving the deprecation deadline, so it was hard for ad tech firms to commit. Privacy Sandbox is not just a drop-in replacement for third-party cookies; you have to change how your algorithms run and potentially your infrastructure. Because of this cost of adoption and the uncertainty in the reward, adoption was held back.

However, there is reason for some optimism about the technology itself. Garrett, Jiangrong, and I have separate work studying the advertiser-side performance of Privacy Sandbox using data from a DSP covering 3,000 advertisers. We found that incremental conversions per dollar, adjusted for spending, are pretty similar across the cookie channel and the Sandbox channel. This illustrates that if you adjust for the fact that there is not a lot of supply of impressions, the technology can perform. The problem is that the equilibrium of ad tech firms adopting it just never formed.

Google probably killed the Sandbox mandate mainly because of recent antitrust developments, like the DOJ ad tech case and investigations in the EU. I do not read the abandonment as a failure of PETs in general. Third-party cookies keep fading, and we have to choose between going to contextual advertising, allowing more alternative IDs and identity graphs, or reaching some kind of privacy-preserving compromise. The challenge is that you have to get ad tech firms to adopt it and abandon third-party cookies. That takes standard bodies like the W3C, policymakers, or big tech firms to force the move.

ES: I want to move on to the second paper on the docket, which is “Copocalypse.” Garrett, you published a paper about the Copocalypse with YouTube. Can you give me background on the paper, the methodology, and the headline findings?

GJ: In September 2019, YouTube paid the largest fine ever under the Children’s Online Privacy Protection Act, a $170 million settlement. As part of that, beginning in January 2020, YouTube identified kids’ content and eliminated all personalized ads associated with it, as well as on-site personalization features like commenting, playlists, and subscriber notifications. The title of our paper, “Copocalypse,” comes from the panic this created among made-for-kids creators. We do not know exactly what the impact was on personalized ads revenue because that data is not public, but one creator shared their data with us, and we saw that ad prices fell 73% without personalized ads.

We studied this by looking at 5,000 top US YouTube channels and compared the made-for-kids content creators to the non-made-for-kids creators using a difference-in-differences design. We showed that the made-for-kids content creators produced 18% less content. They also started to pivot away from made-for-kids content and reduced the quality of the content they were releasing, by releasing less original content, reducing manual captioning, and their video ratings started to fall. On the other side of the market, YouTube views of made-for-kids content fell by 20% and subscriptions fell by 25%.

ES: That is a substantial drop. Part of my argument in a series I have been writing, “Netflix’s YouTube Opportunity,” is that they have been poaching these made-for-kids creators. It is probably much more enticing when the YouTube revenues are deteriorating. The paper is ostensibly about COPPA, but it reads like a preview of what happens whenever platforms remove personalization. Do you view the YouTube settlement as a useful analog for things like ATT, Privacy Sandbox, or any future restrictions on behavioral advertising?

GJ: There is broad agreement that children’s privacy is important, but there are few opportunities to study the impact of restricting behavioral ads on content creation. We think this YouTube settlement is an unusually clean case because the platform eliminated online behavioral advertising and imposed compliance on other firms. ATT is a good comparable because Apple’s policy imposed the “ask app not to track” prompt, but the YouTube settlement meant no behavioral ads at all.

It is also interesting to contrast this with the GDPR. The GDPR did not force a ban on behavioral targeting in reality, though arguably that is what the law or certain regulators intended. The industry tried to maximize cookie consent for as long as possible. Now that there is greater ease for users to opt out of cookies in the EU, this would have some impact, but it is hard to see this blunt impact in the data. The DMA also wanted to eliminate behavioral advertising wholesale in the original text, but they watered that down. There is some recognition, at least in Europe, that there would be an economic harm imposed if they did that.

ES: One of the most noteworthy findings in the Copocalypse paper is that views became more concentrated among the largest channels. Should we interpret that as evidence that personalization disproportionately benefits long-tail creators?

GJ: A recurring finding of the privacy literature is that smaller firms are more affected by privacy regulation. Personalization tends to benefit long-tail creators. In this setting, personalized recommendations and search remained on, but other elements that make YouTube sticky—adding videos to playlists, commenting, end screens, subscriber notifications—were deactivated. It is not wholly clear why that would affect smaller channels more than larger ones, but the smaller channels reduced their content creation by more than the larger channels. This creates a negative feedback loop where they have less content, which gives them less views, less money, and less ability to create content. These smaller creators were likely more dependent on YouTube revenue than the Disney kid channels. It is probably harder for the smaller creators to get alternative revenue sources like sponsorships as well.

ES: Right, if you are Dude Perfect or MrBeast, you just go to influencer campaigns where you are promoting a product or selling chocolate bars versus the ads you are dependent on. It is a fascinating finding. When the new paper drops, I will have you both back on. I appreciate you coming on the podcast and sharing your wisdom.

SK: Thank you so much for having us. It was really fun to be here.

GJ: Thanks for having us, Eric, and thanks for all you do. I always enjoy the insights you and your guests have to share. Thanks for using your time in a way that benefits so many of us.

ES: I appreciate that. Thank you very much.