Allow Google’s AI scraping or lose search visibility

As the US government weighs its options following a landmark “one man” ruling against Google last week, online publishing is increasingly facing a bleak future. (And this time, it’s not just because of drastically reduced ad revenue.) Bloomberg reports that their choice now hinges on allowing Google to use its published content to generate “answers” for internal AI-generated searches or losing visibility on the company’s search engine.

At the heart of the problem is Googlebot, a search engine that crawls and indexes the live web to produce the results you see when you enter search terms. If publishers block Google from using their content to find the AI-generated answers you now see at the top of many search results, they also lose the right to include their web pages in general web results.

Catch-22 has led publications, rival search engines and AI startups to pin their hopes on the Justice Department. on tuesday, The New York Times reported that the DOJ is considering asking a federal judge to split up parts of the company (to investigate divisions such as Chrome or Android). Other options it is reportedly weighing include forcing Google to share search data with competitors or abandoning its default search engine deals, such as the $18 billion one it struck with Apple.

Google uses a different search engine from the Gemini (formerly Bard) chatbot. But its main search engine covers AI Overview and general search, leaving web publishers with little (if any) scope. If you let Google crawl your content for AI responses, readers may think that’s the end of the story without bothering to visit your site (which means no revenue from potential readers). But if you block Googlebot, you lose search visibility, which means very little short-term revenue and a huge loss of long-term competitive standing.

Fixit CEO Kyle Wiens told Bloomberg“I can stop ClaudeBot [Anthropic’s crawler for its Claude chatbot] in listing us without harming our business. But if I block Googlebot, we lose traffic and customers.”

Another problem with combining the two is that it gives Google an incalculable advantage over small AI startups. The company receives tons of free training data from publishers who are determined to stay visible in search. In contrast, AI companies are forced to pay publishers to access their data – and, even then, it won’t include the motherlode that Google gets (literally) for free.

From that point of view, it is not surprising to read that, according to BloombergGoogle rejects publishers who try to negotiate content deals. (Reddit was one exception.) Why waste money on content deals when they get all the training data they need to get the search results most publishers need to survive?

“Now you have a lot of tech companies paying for content, paying for access to that because they need to be able to compete in any serious way,” Alex Rosenberg, CEO of AI startup Tako Inc. , you have been told Bloomberg. “Although at Google, they don’t have to do that.”

It comes down to optimization, Google using desperate publishers. Despite the industry’s current financial woes (online ad revenue has fallen sharply over the past eight years), AdWeek it was reported in March that Google’s AI-generated search answers could lead to a 20 to 60 percent reduction in organic searches.

The ball is now in the Justice Department’s court to figure out where Google — and, to some extent, the entire web — goes from here. BloombergThe full story is a must read.