Executive summary
The first part of this paper focuses on competition between search engines that match user queries with webpages. User welfare, as measured by click-through rates on top-ranked pages, increases when network effects attract more users and generate economies of scale in data aggregation. However, network effects trigger welfare concerns when a search engine reaches a dominant market position. The EU Digital Markets Act (DMA) imposes asymmetric data sharing obligations on very large search engines to facilitate competition from smaller competitors. We conclude from the available empirical literature on search-engine efficiency that asymmetric data sharing may increase competition but may also reduce scale and user welfare, depending on the slope of the search-data learning curve. We propose policy recommendations to reduce tension between competition and welfare, including (a) symmetric data sharing between all search engines irrespective of size, and (b) facilitate user real-time search history and profile-data portability to competing search engines. The second part of the paper focuses on the impact of recent generative AI models, such as Large Language Models (LLMs), chatbots and answer engines, on competition in search markets. LLMs are pre-trained on very large text datasets, prior to usage. They do not depend on user-driven network effects. That avoids winner-takes-all markets. However, high fixed algorithmic learning costs and input markets bottlenecks (webpage indexes, copyright-protected data and hyperscale cloud infrastructure) make entry more difficult. LLMs produce semantic responses (rather than web pages) in response to a query. That reduces cognitive processing costs for users but may also increase expost uncertainty about the quality of the output. User responses to this trade-off will determine the degree of substitution or complementarity between search and chatbots. We conclude that, under certain conditions, a competitive chatbot markets could crowd out a monopolistic search engine market and may make DMA-style regulatory intervention in search engines redundant. The paper concludes with some policy recommendations.
Introduction
Search engines are a crucial gateway to access online services in modern digital economies. When a single search engine – Google Search – reaches a dominant market position, covering about 90 percent of all searches1, the lack of competition in search may distort downstream services markets that depend on referrals from search engines.This has already led to several competition cases against Google Search, for example the Google Shopping case in the European Union (Deutscher, 2021), and reports on Google’s alleged abuse of dominance (US House of Representatives, 2020; Scott-Morton and Dinielli, 2020). Researchers have attributed Google’s dominance to data-driven network effects: more users generate more data, which improves the quality of search and therefore attracts even more users to the search engine (Prüfer and Schotmuller, 2022). The economic impact of network effects is ambiguous.They may increase user welfare through better services but may also reduce user welfare because of reduced competition in downstream services markets.
EU competition policymakers focused on the negative competition effect and concluded that breaking the network-effects feedback loop could solve this.To that end, the EU Digital Markets Act (DMA, 2022) imposes obligations on very large ‘gatekeeper’ search engines, requiring them to share user query and click data with other smaller search engines. DMA Art 6(11) states that gatekeepers “shall provide to any third-party undertaking providing online search engines, at its request, with access on fair, reasonable and non-discriminatory terms to ranking, query, click and view data in relation to free and paid search generated by end users on its online search engines”. Giving competing search engines access to data collected by the dominant incumbent will facilitate market entry. Competitors will no longer depend on network effects to accumulate the necessary user data to run an efficient search engine.That should eliminate monopolistic search market problems.
However, policymakers may face tension between competition policy and user welfare objectives. ‘Classic’ competition policy seeks to maximise consumer welfare by promoting competition in markets, avoiding or suppressing the emergence of dominant players with large market shares and replacing them with many competing players with smaller market shares.This competition policy objective should remain valid in the digital platform economy (Digital Competition Expert Panel, 2019). It is reflected in the stated policy objective of the DMA: “to ensure contestability and fairness for the markets in the digital sector” (DMA, Recital 7).
In the case of search engines, this view would run into problems if more competition between search engines decreased the quality of search and user satisfaction with search results, because competition fragments user data across many search engines. Our first research question in this paper is to examine how likely these negative effects on consumer welfare are. We do this by reviewing several recent empirical papers on search-engine efficiency.They reveal the importance of user data collection, especially for rare keywords that represent a majority of all searches, and confirm that smaller market shares reduce search engine access to rare keyword data. We then explore whether an appropriate design of search-data-sharing governance mechanisms could reduce the tension between competition and welfare. We conclude that symmetric search-data sharing between all search engines, rather than the asymmetric sharing mechanism foreseen in the DMA, may achieve this.
Apart from data sharing, there may be other ways to instil competition in search markets. In the second part of the paper, we explore a second research question: can recent substantial changes in searchengine technology lead to more competition, and do they make the DMA search-data-sharing obligations redundant? The recent arrival of generative AI with Large Language Models (LLMs) and chatbots or answer engines, such as Chat-GPT2, and their ability to produce a more elaborate and indepth semantic reply to user queries, compared to search engines, represents a substantial technological innovation and a natural experiment to detect the impact of technology on searchmarket positions. Rather than ranking webpages and letting users search for the desired answer in these pages, LLMs give users a reasoned natural language answer to a query. Hence the label ‘answer engines’, as distinct from ‘search engines’. Answer engine LLMs are pre-trained on very large text databases, prior to usage. Unlike search engines, they do not rely on user network effects to climb a learning curve with increasing market share. On the one hand, the absence of network effects may facilitate market entry for smaller players. On the other hand, LLMs require access to oligopolistic input markets, including a global index of web pages, an inventory that only Google Search and Microsoft Bing have compiled, and hyperscale computing infrastructure. Moreover, pre-training LLM models comes at a high fixed cost for new market entrants.The net effect of these two opposing forces on market entry remains an open question.
In practice, search and answer engines are partial substitutes. For semantically simple queries, consumers may prefer search. For semantically complex queries, answer engines reduce user transaction costs because they do the semantic processing that search engines cannot do (Wu et al, 2020). At the same time, answer engines may not be entirely reliable and produce poor quality or even erroneous responses.They may ‘hallucinate’. Producers of search services may combine search and answer engines in a single service to capture users that prefer one or the other. That positions search and answer engines as at least partial substitutes, depending on user preferences regarding the tradeoff between semantic transaction costs and ex-post uncertainty about the outcome. We conclude that input market bottlenecks and user behaviour will determine the degree of competition between competitive chatbots or answer engine services and more monopolistic search engine services.