Google has officially removed the crawl rate setting tool from the Google Search Console, as previously announced in November. The tool, which allowed site owners to communicate to Google to crawl their site less frequently, is no longer useful according to Google. The company stated that they have improved their crawling logic and other available tools for publishers, rendering the crawl rate limiter obsolete. As of January 8, 2024, the tool is inaccessible within Google Search Console and site owners are redirected to a notice of its unavailability.
The crawl rate limiter was a feature within the legacy version of Google Search Console, providing site owners with the ability to reduce Google’s crawling of their websites. However, this tool was only recommended for use in cases where server load problems were caused by excessive Googlebot activity. Google emphasized that limiting crawl rates has typically been discouraged and should only be used in specific instances.
Gary Illyes from Google explained that the decision to remove the tool was based on the improvements made to Google’s crawling logic and other available tools for publishers. He noted that Googlebot’s response to a site’s server handling has been the primary factor in adjusting crawling frequencies. The crawl rate limiter tool had a relatively slow effect and was rarely utilized by site owners. In instances where it was used, the crawling speed was often set to the bare minimum. Consequently, Google has set the minimum crawling speed to a lower rate following the deprecation of the crawl limiter tool.
While this change will impact site owners who have been using the crawl rate tool, Google has assured that they will continue honoring previous crawl rate settings if the search interest is low. Site owners who experience crawling issues can make use of a help document and report form provided by Google. It is important for site owners to consider how this change will impact their crawl rate in the coming days and weeks.
In conclusion, site owners are advised to adjust their strategies for managing their site’s crawl rates following the removal of the crawl rate setting tool. It is crucial to consider the potential impacts on crawl rate and to seek assistance from Google in the event of any crawling issues.
Read Full Article