HostMonster Web Hosting Help

Telling Google How Often To Crawl Your Website

Google uses sophisticated algorithms for determining how often to crawl your site.  Their goal is to crawl as many pages as possible from your site on each visit without overwhelming your server's bandwidth.

If you are experiencing bandwidth related server load issues (i.e. too many requests too quickly), you may want to reduce how fast Google and other search engines crawl your site.   Too many requests on your site in a very short period can cause your site to load slowly and even cause load issues on the server.  This is especially true of very busy websites, or websites that are poorly or inefficiently coded.

If you are not experiencing any bandwidth or bandwidth related load issues on your server, it is recommended that you allow Google to determine the optimal crawl rate for your website.

Changing Google's Crawl Rate

Google allows you to adjust the crawl rate (the time used by Googlebot to crawl your website) for an entire domain or subdomain.  You cannot specify different crawl rates for sections of your site (e.g. specific folders or subdirectories).

For example, you can specify a custom crawl rate for and, but you cannot specify a custom crawl rate for

Changing the crawl rate only changes the speed of Googlebot's requests during the crawl process.  It does not have any effect on how often Google crawls your site, nor how deeply they crawl your URL structure.

To change Google's crawl rate:

  1. Login to Google Webmaster Tools
  2. Add your site to Google Webmaster Tools, if you have not done so already.
  3. On the Webmaster Tools Home page, click on the site you want.
  4. Under Site Configuration, click Settings.
  5. In the Crawl Rate section, select the option you want.

The new crawl rate will be valid for only 90 days.

Knowledgebase Article 75,619 views bookmark tags: google seo (updated 944 days ago)

Was this resource helpful?

Did this resolve your issue?

Please add any other comments or suggestions about this content:

Recommended Help Content

A guide to configuring bing's crawl rate for your site. (updated 948 days ago)
Knowledgebase Article 92,482 views tags: bing seo

What is SEO and how can it help you (updated 1273 days ago)
Knowledgebase Article 38,968 views tags: google rank seo site

Does HostMonster allow me to create a Google Site Map and what do I need to do to create one? (updated 783 days ago)
Knowledgebase Article 92,058 views tags: google python site xml

Related Help Content

Why is my site not the first result on Google or an other search engine? (updated 1476 days ago)
Knowledgebase Article 106,397 views tags: bing engine google search seo yahoo

How do I use Google AdWords? (updated 145 days ago)
Knowledgebase Article 71,269 views tags: adwords campaign google marketing promotion seo

How to remove the "This site may harm your computer" warning in Google search results. (updated 1301 days ago)
Knowledgebase Article 179,027 views tags: block blocking google malware

How do I upload my web site? (updated 559 days ago)
Knowledgebase Article 1,246,969 views tags: ftp site upload uploading web website

The website is taking a long time to load. It just seems slower today than normal. (updated 123 days ago)
Knowledgebase Article 207,721 views tags: site slow web website

Clients in China are unable to view my website. My website's content doesn't contain any objectionable material. (updated 2119 days ago)
Knowledgebase Article 104,760 views tags: access block blocking domain name people site website

Why don't my images display on my site? (updated 2136 days ago)
Knowledgebase Article 95,036 views tags: display image images link site website

I created a website using BlueVoda and now I want help publishing it. (updated 2119 days ago)
Knowledgebase Article 92,037 views tags: publish site upload web website