The introduction of Google-Extended, a standalone product token launched by Google to allow users to block Bard, Vertex AI generative APIs, and future generations of models from accessing their content, has sparked a significant response from various websites. According to research shared with Search Engine Land by the Detailed.com team, more than 250 websites have now implemented blocks against Google-Extended. This trend highlights the ongoing debate surrounding the decision of whether brands and businesses should block bots that crawl content used to train language models. The growing number of sites choosing to block Google-Extended is indicative of their reluctance to contribute to AI companies’ profitability and competitiveness at their own expense.
The numbers speak volumes about the increasing skepticism towards Google-Extended. As of November 19, 252 out of 3,000 popular websites have taken measures to block Google-Extended, representing a substantial increase from just over a month earlier when only 89 sites had done so. This signifies a notable 180% jump in the number of sites implementing the block in the past month.
Furthermore, the list of websites blocking Google-Extended includes prominent names such as Ziff Davis properties (e.g., PC Mag, Mashable), Vox properties (e.g., The Verge and NYMag), The New York Times, Condé Nast (22 sites, including GQ, Vogue, Wired), and Yelp, a frequent critic and legal opponent of Google. This significant boycott emphasizes the growing unease among websites about the implications of allowing their content to be used by AI companies without their consent.
It is important to note that while blocking Google-Extended in robots.txt prevents access to content, it does not prevent Google from using the content for its Search Generative Experience. To fully opt-out, websites would need to block Googlebot, which would also remove them from Google Search. However, they can opt-out of SGE overviews by using the “nosnippet” tag.
In conclusion, the surge in websites implementing blocks against Google-Extended underscores the escalating concerns regarding the use of their content by AI companies without proper consent. This growing trend signifies a shift in the attitudes of website owners and businesses, reflecting their desire to protect their content and avoid potential exploitation by large tech companies like Google. As the debate continues, it remains to be seen how Google and other AI companies will address these growing concerns and adapt to the changing landscape of content accessibility and usage.
Read Full Article