Google has announced that they will be removing the search console crawl rate limiter tool from their search console, with a scheduled removal date of January 8, 2024. This decision comes as a result of improvements to their crawling algorithms which have essentially made the tool unnecessary.
The crawl rate limiter tool was first introduced to the search console fifteen years ago in 2008, with the purpose of allowing publishers to control Googlebot crawling in order to prevent overwhelming their servers. At the time, some publishers were experiencing an excessive amount of crawling, which could lead to their servers being unable to serve webpages to users. After receiving complaints, Google released the tool within the search console.
However, the impact of the tool was primarily to provide Google with data. According to the company, requests to limit crawling usually took about a day to go into effect and remained in effect for 90 days.
The decision to remove the rate limiter tool is based on the fact that crawling algorithms have advanced to the point where Googlebot can now automatically sense when a server is reaching capacity and take immediate action to slow down the crawl rate. Additionally, Google stated that the tool was rarely used, and when it was, the crawl rate was generally set to the lowest setting. Moving forward, the minimum crawl rate will be set to a lower rate by default.
By deprecating the crawl limiter tool, Google aims to make search console less cluttered with tools that are rarely used, ultimately improving the user experience. Publishers who still encounter problems with Googlebot’s crawl rate can still use the Googlebot report form to provide feedback to Google.
The overall goal of this decision is to streamline the search console experience and continue honoring the settings that some site owners have previously set. The removal of the crawl rate limiter tool is part of Google’s effort to improve their crawling algorithms and simplify their search console platform for users.
Read Full Article