Google announced the deprecation of the crawl rate limiter legacy tool within Google Search Console, with January 8, 2024, set as the date for its removal. This feature is considered obsolete due to Google’s enhanced crawling logic and other available tools for publishers.
The crawl rate limiter is a tool within the legacy version of Google Search Console that allows communication to Google to crawl a website less frequently. However, it is not recommended by Google unless there are server load issues specifically caused by Googlebot’s crawling activity. The tool is still accessible until its removal. The reason for removing the tool is that it had a slower effect and was rarely utilized by site owners.
Google will set the minimum crawling speed at a lower rate comparable to the old crawl rate limits, effectively continuing to honor the settings previously established by site owners for low search interest and to prevent the site’s bandwidth from wasting. If site owners experience crawling issues, they are encouraged to refer to a help document and use a report form to alert Google.
Gary Illyes from Google explained that Googlebot’s behavior is responsive to how a server handles Googlebot’s HTTP requests. With the continued enhancements to crawling logic and other tools available to publishers, the crawl rate limiter’s usefulness has dissipated. The tool’s effects were slower and rarely used, mainly set to the lowest crawling speed by site owners. Illyes emphasized that the rate limiter tool often took over a day to apply new limits on crawling and had little impact.
For those who have been using this crawl rate tool, it is important to note that it will be removed, so it is advisable to set a reminder to monitor any potential impact on server performance when the feature is turned off.
In summary, Google is discontinuing the crawl rate limiter legacy tool within Google Search Console due to its obsolescence. Googlebot’s crawling logic and available tools for publishers have been enhanced, diminishing the need for this tool. The minimum crawling speed will be set at a lower rate, and site owners experiencing crawling issues are advised to refer to a help document and utilize a report form to inform Google. It is essential for users of this tool to keep track of its removal and its potential impact on server performance.
Read Full Article