As anyone who’s been online over the past few weeks has probably noticed, Google’s major search artificial intelligence update has had a rocky start. In the days since the company released its responses to AI-generated queries, called AI Reviews, the feature has been widely ridiculed for generating erroneous and sometimes crazy answers, such as recommendations on how to eat rocks or shop. pizza with glue.
Novel data from search engine optimization firm BrightEdge suggests that Google has significantly reduced how often it shows people its AI overviews since introducing the feature, and in fact significantly reduced the feature before the wave of criticism. The company has been tracking how Google’s AI responses appear in search results based on a list of tens of thousands of sample searches since the feature was first offered in beta last year.
When AI overviews were made available to logged-in U.S. users in English on May 14 following Google’s I/O conference, BrightEdge saw AI-generated responses for nearly 27 percent of the queries it tracked. However, their presence dropped sharply a few days later, a week before screenshots of AI Reviews errors went viral on the Internet. At the end of last week, when Google published a blog post acknowledging bugs with its AI feature, BrightEdge saw AI overviews appear on just 11 percent of search results pages. On Monday, their incidence was essentially the same.
Jim Yu, founder and executive chairman of BrightEdge, says the cancellation suggests Google has decided to take an increasingly cautious approach to the rollout. “There are obviously some risks that they try to manage closely,” he says. However, Yu adds that he is generally positive about Google’s approach to AI overhauls and views these early issues as a “blip” rather than a feature.
“We continue to refine when and how we display AI reviews to make them as useful as possible, including last week rolling out a number of technical updates aimed at improving the quality of responses,” says Google spokesman Ned Adriance. Google declined to share its internal statistics on how often AI reviews appear in search results, but Adriance says BrightEdge’s numbers don’t reflect what the company sees internally.
It’s unclear why Google may have decided to significantly reduce the visibility of AI reviews soon after its launch, but last week the company’s blog pointed out that millions of people using the feature provided up-to-date data on its performance and bugs. The company’s head of search, Liz Reid, said Google had made “a dozen or so technical improvements,” such as limiting the appearance of satirical content in search results. In her post, she noted that these changes will create restrictions on when AI reviews will be offered, but did not detail how these restrictions will affect how often AI scores appear.
BrightEdge began tracking AI reviews through a list of sample queries after Google allowed users to opt-in to beta testing the feature behind schedule last year. The test queries covered nine categories, including e-commerce, insurance and education, and were designed to cover popular but also less common searches. They were tested repeatedly, in some cases several times a day.
In December 2023, BrightEdge found that summaries appeared in 84% of searches, but this number decreased over time. Google’s Adriance said it didn’t automatically trigger an AI review on 84 percent of searches, but didn’t explain its internal measurements. After Google made the AI previews available to everyone, BrightEdge continued to track their appearance using various accounts that had previously signed up for the beta testing and others that had not, but did not notice a significant difference between what both groups saw.
Google wouldn’t reveal how exactly it changed how many AI overviews it showed the general public compared to people signed up for the beta test, but Adriance said people who agreed to take part in the test were shown AI overviews for a wider range of queries.
BrightEdge data also sheds lithe on topics where Google believes AI reviews can be most helpful. AI responses have appeared in the majority of healthcare keyword searches, most recently at a frequency of 63 percent. Example queries included in the BrightEdge data included “foot infection,” “intestinal bleeding,” and “telehealth urgent care.” By comparison, e-commerce queries return AI reviews at a rate of around 23 percent, while restaurants or travel very rarely trigger AI review responses.