Google has introduced a new product token called Google-Extended that allows web publishers to control whether Bard and Vertex AI have access to the content on their sites. This feature is the result of a public discussion initiated by Google in July regarding choice and control over web content. Bard is Google’s conversational AI tool, while Vertex AI is its machine learning platform for generative AI-powered search and chat applications. With Google-Extended, website administrators can choose whether to help improve these AI models. The new crawler has been added to the Google Search Central documentation on web crawlers. Publishers can use robots.txt to block Google-Extended from accessing their content. This offers transparency and control, but managing different uses at scale may become more complex as AI applications expand. While some popular websites have already blocked OpenAI’s web crawler, GPTBot, you can now decide whether to opt out of helping Google improve its AI products. This announcement aligns with an argument made by a contributor to Search Engine Land that using robots.txt for managing data usage in generative AI models is not the right approach.

source: Google introduces Google-Extended to let you block Bard, Vertex AI via robots.txt