1. Knowledge Base
  2. >
  3. Results
  4. >
  5. Telling Google How Often To Crawl Your Website

Telling Google How Often To Crawl Your Website

Google uses sophisticated algorithms for determining how often to crawl your site. Their goal is to crawl as many pages as possible from your site on each visit without overwhelming your server's bandwidth.

Too many requests on your site in a very short period can cause your site to load slowly and even cause load issues on the server.  This is especially true of very busy websites, or websites that are poorly or inefficiently coded. If you are experiencing bandwidth-related server load issues (i.e. too many requests too quickly), you may want to reduce Google's crawl rate (the time used by Googlebot to crawl your website for an entire domain or subdomain).

If you are not experiencing any bandwidth-related load issues on your server, it is recommended that you allow Google to determine the optimal crawl rate for your website.

Changing Google's Crawl Rate

Google allows you to adjust the crawl rate, but you cannot specify different crawl rates for your site's section, such as specific folders or subdirectories. For example, you can specify a custom crawl rate for www.yoursitesdomain.com and subdomain.yoursitesdomain.com, but you cannot specify a custom crawl rate for www.yoursitesdomain.com/subfolder.

Changing Google's crawl rate only changes the speed of Googlebot's requests during the crawl process. It does not affect how often Google crawls your site or how deeply the URL structure is crawled.

To change Google's crawl rate on your website:

  1. Access the Crawl Rate Settings page.
  2. The next page will display two options — Let Google optimize for my site (recommended) and Limit Google's maximum crawl rate.

    Google Crawl Rate Options

  3. Depending on the needs of your website, select your preferred action.
Note: These instructions are up-to-date as of the day we posted them. However, as the Google Webmaster Tools may change, please expect variations from the instructions provided. As always, we recommend that you check out Google Webmaster Tools documentation directly.