Google is deprecating the crawl rate limiter legacy tool within Google Search Console on January 8, 2024, citing the tool’s diminished usefulness due to improved crawling logic and other available tools for publishers. The crawl rate limiter is a tool within the legacy version of Google Search Console that allows users to request that Google crawls their site less frequently. However, Google has historically advised against using this tool unless server load problems are directly caused by Googlebot activity. The tool will remain accessible until it is removed.
Google is removing the crawl rate limiter tool due to improvements in crawling logic and other available tools for publishers. Googlebot now automatically adjusts crawl rates based on server response times and HTTP status codes, which reduces the need for users to manually adjust crawling speeds. Google’s improvements ensure that crawling rates are more efficiently managed, making the crawl rate limiter tool less relevant.
Gary Illyes from Google explained that the crawl rate limiter tool had a slow effect and was rarely used by site owners. Google will be setting the minimum crawling speed to a lower rate comparable to the old crawl rate limits, effectively honoring past settings if the Search interest is low and ensuring that crawlers do not waste a site’s bandwidth.
In the event of crawling issues, site owners can refer to Google’s help document and use the report form to address any concerns. Site owners who have been using the crawl rate limiter tool should be aware that the feature will be going away, and they should monitor the impact on their servers when the feature is turned off.
In summary, Google is deprecating the crawl rate limiter tool within Google Search Console due to its decreased usefulness, improved crawling logic, and other available tools for publishers. The tool allowed users to request slower crawling rates, but was rarely used and is now being removed from Google Search Console. Site owners should take note of the removal of this feature and monitor any potential impact on their servers. If any issues arise, they can refer to Google’s support resources to address and resolve them.