Podcast: Quantifying the impact of AI Overviews on outbound clicks (with Ananya Sen and Saharsh Agarwal)

On this week’s episode of the podcast, I am joined by Ananya Sen and Saharsh Agarwal to discuss their research on the impact of Google’s AI Overviews on web publisher traffic. We explore the results of their field experiment, currently in pre-print, which utilized a custom browser extension to measure how generative search results influence outbound clicks and user satisfaction.

The results of the field experiment suggest a significant reduction in traffic for certain queries, challenging the narrative that these interfaces only filter out low-quality interactions. We dive into the implications for the broader web ecosystem, including the risks posed to smaller publishers and the role of data licensing deals. Among other things, we discuss:

• How generative search interfaces might fundamentally alter the symbiotic relationship between search engines and the publishers who provide their data
• Whether the removal of low-quality clicks truly compensates publishers for the overall decline in traffic driven by AI-generated summaries
• What happens to smaller publishers when major outlets secure exclusive data licensing deals that prioritize their content in search results
• Why AI Mode represents a potential endpoint for search that could essentially eliminate outbound traffic for a majority of queries
• Whether the long-tail of the internet can survive a transition where search engines become information destinations rather than mere conduits

Thanks to the sponsors of this week’s episode of the Mobile Dev Memo podcast:

• ⁠⁠INCRMNTAL⁠⁠⁠. True attribution measures incrementality, always on.
• Xsolla⁠. With the Xsolla Web Shop, you can create a direct storefront, cut fees down to as low as 5%, and keep players engaged with bundles, rewards, and analytics.
• ⁠Branch⁠. Branch is an AI-powered MMP, connecting every paid, owned, and organic touchpoint so growth teams can see exactly where to put their dollars to bring users in the door and keep them coming back

Interested in sponsoring the Mobile Dev Memo podcast? Contact Mobile Dev Memo advertising.

The Mobile Dev Memo podcast is available on:

• Spotify
• YouTube
• Apple Podcasts

Transcript

Eric Seufert: Hello and welcome to the Mobile Dev Memo podcast. I am your host, Eric Seufert, and I am joined today by Ananya Sen and Saharsh Agarwal. Ananya, Saharsh, welcome.

Ananya Sen: Thanks, Eric, for having us.

Saharsh Agarwal: Thanks, Eric, for having us here.

ES: I asked you both to join me today to discuss your preprint paper, “The Impact of Google AI Overviews on Publisher Traffic and User Experience: Evidence from a Field Experiment.” I will have links to the paper in the show notes so anyone listening along can browse it or read it closely. It is a fascinating field study that you did. I shared it on LinkedIn and Twitter a week or two ago, and it generated a lot of conversation, so I am excited to talk through it at length today. Before we do that, why don’t you both introduce yourselves? Ananya, we can start with you.

AS: I am Ananya Sen, an Associate Professor at Carnegie Mellon University’s Heinz College of Information Systems and Public Policy. Over the past decade, my research has focused on topics at the intersection of technology, business, and society. I have particularly focused on digital and media platforms in the context of the online information ecosystem more broadly. I have been at Carnegie Mellon for about seven or eight years. Before that, I was a postdoc at MIT Sloan School of Management and got my PhD from the Toulouse School of Economics.

SA: My name is Saharsh Agarwal, and I am an Assistant Professor at the Indian School of Business. Broadly, my research focuses on technology and how it changes the way we live and interact online, as well as the offline implications of technological developments. Recently, my research has focused on issues related to digital well-being online, governance of platforms, and competition policy. The paper we are discussing today sits squarely in the space of platform governance and competition policy. I have been at ISB for the last three or four years, and before that, I was a PhD student at Carnegie Mellon’s Heinz College, where I met Ananya.

ES: This paper and this topic broadly are touching on a really important topic which is still able to be steered from a policy standpoint. My sense is a lot of the protestations about Google’s behavior in the ad tech trial and the search trial were a little bit futile. There was nothing to be done at that point, coming decades after the fact. Whereas here, there can be a policy intervention that steers things in the way that policymakers might feel is best. It is great that you have studied this empirically. I have written a series called Google’s Gambit where I talked about this. When they first introduced AI Overviews, my point was that this is where they are taking search; they want it to become an interface that doesn’t lead you away, keeps you engaged in the chatbot experience, and will deprive publishers of clicks. It is great that you both have authored this to study that empirically. Please provide an overview of the paper and its principal findings.

SA: The search market is traditionally very symbiotic. Publishers depend on search engines, and search engines depend on publishers. This has been a healthy ecosystem created over the last couple of decades. In recent years, we have increasingly seen a trend towards Google trying to capture some of that value by trying to keep users on its platform and not send traffic downstream. This has been a trend over the last few years with knowledge graphs and featured snippets, so this discussion is fairly old.

With AI Overviews, it has taken on an entirely different scale because of the prevalence of AI Overviews and the nature of the answers they generate, which reduces the need for people to click. The paper is about providing strong causal evidence on this topic. There have been people who tried to shed some light on this, but most have been correlational, comparing before and after or comparing queries with and without overviews. We know that is not causal enough, and that is the gap we wanted to fill.

To answer this question, the best way is to collaborate with the platform and run a field experiment on millions of users. Google must have done that internally, but as researchers, we did not have that kind of access. What we did was run a field experiment with live users by creating a custom browser extension. This browser extension did two things. First, it passively tracked data from people in a natural browsing setting. We got data from about a thousand people on the websites they were visiting and the searches they were making. We got to see the search results presented to them, whether an AI Overview showed up, and where they were clicking.

Besides passively tracking, the key part was that we were able to actively manipulate the search interface. In the main study, we had a control group where we just tracked and did not do anything else. We also had a treatment group where we hid the AI Overviews whenever Google chose to show them. Because this was randomized, all else was equal, and what we measured were the causal effects of showing AI Overviews on clicks.

We see that when we hide the AI Overviews, the downstream organic clicks really increase a lot. Showing AI Overviews reduces clicks by 40% conditional on a search happening, which is a fairly big number. We also find that these clicks are not necessarily higher quality. Google’s argument has been that AI Overviews only cut noisy or not-so-useful clicks. Because we have data on the amount of time that people are spending and their interaction with the downstream page, we are able to shed light on this. We find that the quality of the clicks was not different across the two groups. One of the headline results is that although we find such huge effects on downstream publishers in terms of clicks, user experience was absolutely unchanged even when we hid the AI Overviews. In terms of overall search experience, the quality of information, and ease of finding information, we found that users in both groups felt exactly the same. That is a brief overview of our paper.

ES: I like the point you made about the ideal way to do this being an RCT at the service layer, but researchers often make the perfect the enemy of the good. You found a very clever way of collecting this data. The methodology was impressive. One thing that struck me from the paper is that users didn’t report any improvement in search quality despite the 40% reduction in outbound clicks. How do we interpret that disconnect? Google says they are only removing noisy clicks, but maybe that’s not what is happening. What is the right interpretation of that finding?

SA: When users are shown AI Overviews, they are not clicking, so you would expect that must be improving the search experience. However, when we remove it, the search experience is not changing. We have thought about this a bit. This probably even relates to some of the results we had about position. Most of the time, AI Overviews are shown in the prime spot at the top. Common experience suggests that whatever people see in the prime real estate is what they give attention to. Most people don’t even scroll too much.

When an AI Overview is forced upon them at the prime real estate, that is what catches their attention, their question is answered, and they don’t feel the need to click. However, when the AI Overview is not there, they are equally happy visiting the various links. It is not as if they are having a worse search experience. If the AI Overview is salient enough, then users engage with it and find some value. But the moment it is not there, it is not as if users are looking for it or finding that there is something missing. It is one of those products where unless you really make it salient and give it the prime spot, users don’t care about it too much. That would be the indication from what we have, but I have to be careful. Our survey measures are self-reported. It is possible there are relatively small improvements in convenience which our survey measure is not able to capture.

ES: Maybe we could talk about some nuance that should be applied to the interpretation of this self-reported data.

SA: When I see the AI Overview, my search experience improves slightly because of convenience. However, it is not a very big shift. When I am asked about this at the end of two weeks—remember, the survey measure was collected at the end of two weeks, not in real time—it is possible that while you were searching, you had a slight improvement in convenience, but it is not something that you remember at the end of two weeks. We also had a third arm in the paper where we forced people to use AI Mode. There we did see a shift in the experience numbers. In the AI Mode arm, we saw a huge decline in all the experience numbers. That really stuck with people. Even though we asked that question after two weeks, we saw a reduction in experience. That indicates this measure is definitely capturing some signal, but given that it is self-reported, we have to be careful.

ES: The experiment measures the short-run effect on clicks. Do you think click behavior might change meaningfully once AI Overviews are no longer new and users have grown accustomed to them?

AS: That is a great question because this space is moving quickly. We need to zoom out from our study and think more broadly. Our experiment lasted for two weeks. We observed behavior for two weeks and then made the final payment for people who installed the extension. People were then free to uninstall the extension, but about 60% kept it installed. At the end of the two-week period, which was the main experimental period, we switched people’s interventions. People who were in the treatment group became the control group. People who were in the control group now had their AI Overviews hidden. In both these periods, we find results that are pretty consistent, which is that click-through rates on organic links increase when AI Overviews are hidden.

This behavior sets in pretty quickly. We saw this even in some of the smaller pilots that we had done. Given how AI Overviews are currently, I think this effect is pretty robust even though we measure something for two weeks. Stepping back, our experiment is in a particular point in time. These AI Overviews have been around for over a year now. I think they were first launched in May 2023. The percent of queries that have seen AI Overviews has been increasing over time. I am guessing people are getting used to it, but things can change.

I can see it changing in two opposing ways. Let’s say AI Overviews and how they appear on the Google search page remain similar to what we see. Then maybe people learn how to read the AI Overviews and scroll past them because they know it’s there but they also know that they can ignore it. They don’t always need to click on it. That would mean our treatment effects might be an upper bound, and things might actually become better for publishers down the road. But it can also work the other way. We do have a third arm, which is AI Mode. If you think about why Google might have introduced AI Overviews in the first place—to compete with the LLM chatbots out there—AI Mode might be the natural culmination of this process. In that case, we see that click-through rates reduce even more on organic links. I think the reason why we had the AI Mode arm was exactly because we were trying to think of what it might look like down the road. It’s highly plausible it might look something like AI Mode. In that case, we see downstream traffic being even lower.

ES: It’s a very fair point that this is not new. It launched in a limited beta in May 2023 and rolled out to a broader set of countries in 2024. Even so, it’s been more than a year. This isn’t a new idea. Chatbot interactions have become ubiquitous in the consumer experience. The stuff that hasn’t taken off yet probably is not going to because people seem to characterize consumer interaction with chatbots as this new phenomenon, and it’s not. It’s years old. We’re not in the early innings. One of the questions that pointed a spotlight at the paper, and which people reacted to in predictable ways because there is a lot of animosity towards Google from the traditional publisher sphere, is about licensing deals. Google in a way was responding with AI Overviews. People had written Google off and said search was going to go to zero as a result of ChatGPT. When I’ve spoken to people at Google, I’ve made the case that Google’s Gambit was a deliberate choice. Maybe the purpose wasn’t to abandon the open web or starve it of traffic, but that was going to be the outcome. People at Google have said they recognize that outcome was a consequence, but what else could they have done when their own business was under threat from ChatGPT? Everyone is going to expect to interact with a chatbot. The AI Overviews always felt like this transition phase to get people accustomed to it, but my sense has been you go to google.com and you land on Gemini at some point.

The deterioration in the click rate doesn’t account for the fact that some publishers that have done data licensing deals with Google might be preferred in AI Overviews in terms of outbound traffic. That’s not a criticism; you couldn’t parse that. Given that pretty meaningful deterioration in the click rate, could some publishers actually be seeing an even sharper decline because they are not participating in these data licensing deals? The actual outbound clicks are preferencing the people that are, so if you would exclude the companies that are doing the deals, the decrease is even more extreme.

AS: I think this is a super interesting point. Based on Google AI Overview research, about 20 major international outlets have signed licensing deals with Google. It is not a dimension that we explored in the paper because as soon as we start digging into subsamples, we lose power. We have a thousand people over a few weeks, but it is hard to detect effects as soon as you start looking at heterogeneity. One thing to keep in mind right now is that the proportion of clicks coming from AI Overviews is pretty small. In our study, it was about 7 or 8%. That is higher than what I have seen reported in the Pew Research study, but it is still only 7 or 8%.

If we take your question to its logical endpoint, let’s say that’s one of the models that comes through, where there is a licensing play. Then it’s perfectly possible that outlets that are not signing these deals don’t show up in these AI Overviews. Then that eventually spills over to search results and has an even more negative downstream effect on them. It’s not only on the publishers to decide whether they want to sign these deals or not. Who does Google think is worthy of these deals? What is important to them? The thing that I do slightly worry about is that when I think of the open web, the internet, and e-commerce, we often think about the long tail and the benefits of the long tail to the online population. I fear that here the smaller outlets and smaller publishers might lose out the most. That remains to be seen. This was our way of providing reasonably neutral evidence as a neutral party. Per se, we don’t really care which way the results go. Our main job is to pin them down rigorously and report them and then contribute to these conversations.

ES: You make a really good point about the data licensing deals. Not only is it a little bit risky for a publisher to do that licensing deal in the first place because there’s no guarantee that you’re going to get the clicks—doing the deal could ultimately result in you getting fewer clicks because your data appears in the AI Overview and may obviate the need for a click—but at the same time, Google becomes kingmaker in that way. They’ve always been kingmaker in the sense that they had an algorithm that determined which websites got clicks. Presumably, that was opaque for sure, but you could maybe presume that it was in the best interest of the consumer because Google was incentivized to drive you away. They want you clicking out in the first couple of links, and so they’re probably going to be pretty good at ranking. When ads got introduced and expanded to become the whole top of the fold, then they created a situation where everyone’s interests were kind of aligned if the advertiser that got the placement would have been the top link anyway. The auction mechanism still ensures that the best information is being placed up front. But now that’s demonstrably not the case. They want to keep you there, presumably to show ads to you, and keeping you in that engagement environment allows them to show more ads. So they have no incentive to drive you out.

Now the question becomes, who do they need for that purpose? They don’t need a very small niche long-tail publisher. Whereas before, for a non-monetizable query, that might have been the best destination, and they might have preferenced you over some much bigger website because they crawled everything. The size of your user base was irrelevant to them; they just wanted the best information. But now they don’t need you because you need a lot of data to be useful to them. If you’re some very niche publisher without a lot of content, you’re not valuable to them and they’re not going to do a licensing deal with you. Then you’re left completely out in the cold. You get to this Faustian bargain situation where you say, “I know if I give you my data, I can’t control whether you’re going to drive clicks to my website anymore, which would then enable me to monetize as a publisher through ads, but if I don’t do the deal with you, I’m pretty sure you won’t drive any clicks to me, and I am therefore starved of the ads revenue while starving myself of any potential data licensing revenue.” It’s a very interesting situation that publishers have been put in, but it also probably disadvantages the smaller publishers the most.

You talked about the intervention that removes AI Overviews entirely. Could a version of AI search exist that preserves most of the user benefit while restoring publisher traffic? Is there a version of AI Overviews that hits a sweet spot in terms of driving clicks out but also fulfilling that need now to give consumers that natural text search opportunity?

SA: That’s a great question, and it also helps us appropriately clarify the scope of our study. What we are basically studying is as per the current design, which we are seeing the status quo, what is that doing to downstream traffic and user experience. Our intervention is basically studying the impact of the current design. It’s not saying that AI search in itself is always going to lead to the same results. Therefore, I do agree that a lot of what we have measured is very typical to the design of the interface as well. It’s quite possible that there might be a different version of AI search where there are AI capabilities, but at the same time, publisher traffic is also not lost.

Google is also exploring some of this. In the recent core updates and recent Google I/O, they did mention some of these changes, for instance, trying to make the links more clickable and having a carousel of links in the AI Overview. They are basically trying to make the links a little more prominent. The impact of these is not yet known, so it’s a fairly open question. While our paper looks at what is happening right now, the question that you pose is very different but very interesting: is there a version of search at all where we can get the best of both worlds? I think there would be a lot of platform design elements that would go in here.

There are a few other things that you can also think of. If you don’t want to go down the fancy route of playing around with the font size of the AI Overviews or the prominence of how much of the snippet you’re showing, one simple thing could be just to reduce the prominence. Rather than let them have prime real estate, maybe just move them to a less prominent position, or maybe give users a choice. Say an AI Overview is available; do you want to view it? There you still have some element of AI search, but because it’s not being forced upon people, maybe publisher loss will not be that heavy. I think these are all very interesting scenarios to examine, but something which our paper doesn’t look at currently.

ES: Do you have any sense of what that might look like?

SA: Google will definitely not do some of these on their own. They are making some interface changes on their own, like making the links more prominent and having a carousel of links. But Google is not really letting go of its prominent position. If we talk about reducing the prominence of the AI Overviews, I don’t think it’s going to happen on its own because incentives are very differently aligned here. An intervention to reduce the prominence of AI Overviews, maybe not have it as a default, maybe give users a choice to unroll it if they want to—I don’t think those are going to happen without regulation. I don’t think we can get to that without regulation.

ES: Is there a way to have both? You could have a different colored background on the info box that contains the source links just to draw the user’s attention to the actual source of the information. Something as simple as that might work.

SA: There has been decades of research on nudging people towards specific elements on the web page. All of these possibilities are there. For instance, right now in the AI Overview, not only are they at the top position, but you also get to see a significant amount of the AI Overview, and then you have to click on the ‘show more’ button to see more. What if you just reduced that size so you only get to see one or two lines? Maybe users don’t really care about it if it’s not that prominent. Some of these interventions can actually be fairly simple. You just have to make it less prominent and let the attention go organically to the other links, but as I said, it’s not something which you can expect Google to do without some external pressure.

ES: The result that surprised me the most was that the additional clicks weren’t lower quality. How do I reconcile that with what Google has said publicly? I don’t think that they’re mismeasuring this and I don’t think that they’re prevaricating. I’m just wondering if there’s a different interpretation or if the way that they’ve talked about these kind of false clicks or the lower quality clicks is just a definition that might not be intuitive to somebody who doesn’t work at Google. Is there an explanation for that that allows me to reconcile both of these things?

AS: We have also seen some of those statements and we also mention that in the paper. The point of the exercise was to try and understand the impact of such an AI product on different stakeholders within this ecosystem. One of the things that we’ve heard is that the quality of clicks that’s coming through AI Overviews is better, but when we analyze it, conditional on a click, we don’t find any difference based on the dimensions of quality that we can measure. We don’t find any difference in terms of bounce rates or downstream time spent or the probability of clicking on the go back button. On the face of it, it does seem to go against what Google is saying, and you’re completely correct in speculating that maybe we’re just talking past each other.

Maybe Google is measuring something else. Our objective is to just try and pin down the most rigorous evidence that we can, and we’re happy to take into account any data that Google provides. I haven’t seen anything yet. We have these measures as part of our experimental setup, but we can’t look further down the funnel. We don’t know whether people are purchasing more because that’s not something that we can observe. The proportion of clicks that are coming from within AI Overviews for us is 8%, but that 8% constitutes a reasonably small number just given our sample size. Maybe Google has a significantly larger sample size and they are measuring something differently or based off of a larger sample size, or they might be looking at certain types of segments which we can’t do. We don’t have enough power. This is not to say that the publishers are all right, Google is all wrong, or vice versa. It’s just that this is what we find and we want to contribute to the conversation around the online information ecosystem, whether that’s policy, whether that’s related to publishers or Google.

ES: It’s possible that they just have some very proprietary metric that would be impossible to intuit from the outside that is a bad click, but it does seem like it’s difficult to reconcile.

AS: At the very least, we would have to reconcile the fact that maybe something is happening down the funnel for certain brands or for certain types of purchase, but at the top of the funnel, we are still seeing no differences across treatment and control—AI Overviews versus no AI Overviews. Any explanation has to overcome that at least because these are effects that we can pin down reasonably well.

ES: How much have you thought about the right policy response to this? I do think it’s important to bang the drum on the idea that this is when you can actually influence things. If you wait a decade, you can’t. This is the moment to actually be paying attention. I don’t know that this FTC or this DOJ is interested in intervening on these types of issues. Assuming they were, what would you say? What’s the least interventionist policy you’d recommend? Is this an antitrust problem, a copyright problem, a transparency problem, or something totally new? What do you think the timeline is for addressing it? I do feel like these moments are very fleeting when they arise; maybe we’re still in that window, but I doubt it’s going to last much longer. What would be your policy prescription?

AS: In the paper, we don’t go into policy proposals because the main goal of the paper is to provide evidence. Based on the evidence, I don’t know if we have an optimal legal remedy. But based on some of the feedback on the paper and this conversation, we are thinking of writing up a separate piece trying to speculate about what this evidence might imply for different types of intervention. Let me take a crack at that speculation. The DOJ or the FTC haven’t done much right now and they might not have the appetite. But in England, the CMA has introduced an opt-out policy where news publishers can opt out of being featured in the AI Overviews, but that doesn’t automatically lead to their exclusion from the organic search results. Based on our evidence, I don’t know if that’s the right approach because at the end of the day, the downstream publishers might still not be getting access to the traffic.

On top of that, if we head in the direction of Google signing licensing deals, then news outlets lose out on that revenue as well. I think transparency matters in general; choice to an extent matters. But we need to come up with a solution that potentially compensates publishers for providing content which forms the basis of these AI Overviews. But you might get compensated, and if you don’t get access to the traffic, then it’s very hard to build a brand, to convert users into subscribers. Going back to the platform design conversation, is there a way to make these links more salient such that some of the traffic still does go through to publishers so that they can build their brand, they have data on the traffic so that they can optimize their content offerings, while at the same time they get compensated for providing the content that powers these AI Overviews? I think maybe a combination of that would be the ideal scenario in my mind.

This is not the first time we’ve seen a legal back-and-forth between Google and downstream publishers. We saw this with Google News about a decade ago. At that time, I don’t think that Google really wanted to pay news outlets for the tiny excerpts that they had along with the links on Google News. But I think this context is fundamentally different and I think everyone is recognizing that given the fact that we already have licensing deals between Google and different publishers. The main point that we highlight through the paper is that there is a diversion of traffic and that could have implications for antitrust and copyright, but we don’t have an exact legal remedy. We do hope to speculate about this in a separate piece.

ES: Please ping me when that is live. I would love to read it. Ananya, Saharsh, this was fantastic. I really appreciate you taking the time and going through the effort to collect this data. This is not an easy thing to do, but this is the time to be interrogating these questions because if you wait five or six years, as is often the case, it is too late. I appreciate that you’ve published this now. Thank you very much for sharing your wisdom and your insight.

AS: Thank you for having us on.