How to increase Google crawl rate

How to increase Google crawl rate or the Google crawling rate is a common question that people ask us when they run into problems with their web sites such as web pages not being indexed for a long time. The frequency with which Googlebot visits your site can vary from hours to weeks.

Sometimes you may need to decrease or increase Google crawl rate when experiencing problems with your server.

How to increase Google crawling rate

Before we answer the How to increase Google crawling rate question, we should explain what determines how often or with what frequency Google crawls your web site.

• Relative importance of your site,
• relative importance of your site,
• and relative importance of your site.

The more important your web site is, the more often Google will crawl your web site. This is pretty obvious. A large content-rich portal needs to be crawled by Google more often because such portal is likely to add new content often while a low-quality outdated web site would be a waste of time if Google crawled it frequently.

Relative importance of your web site is measured by two metrics:

• Number of links to your web site
• Number of web pages within your web site

Links: The first measure is quite obvious. If Google does not find enough links pointing to your web site, then Google does not think your web site is very important. The more quality links you have pointing to your web site, the higher Google values your site, and crawls it with higher crawl frequency.

How do we define a "quality" link? Links from bad neighborhoods can hurt you. A quality link is the one that is from a page with relatively high page rank and yet better from a site with similar content. Links without the nofollow attribute work the best. See here for more information about the nofollow tag: nofollow.

Pages: If your site has only a few pages, Google will not crawl them very often. Google is likely to increase crawl rate or frequency when it finds large and constantly growing sites.

How can I increase Google crawl rate

Now that you know what it takes to increase Google crawl rate, you are probably going to write a few more pages and build some links. There is something more you should know.

If you have a Google Webmasters Tools account (http://www.google.com/webmasters/tools), you can increase Google crawl rate through the account in the Dashboard - Settings screen. This however applies to web sites that have been running for some time only. If your web site is brand new, Google assigns some default crawl rate to your site, and you will see the following in the Settings screen:

If your web site has been online for some time, you will be able to decrease or increase Google crawling rate in the screen.

It is important to mention that setting the Google crawl rate to "fastest" is not always the best idea. Google crawler can eat up significant resources on your server and lower your server performance.

Google crawl rate and server speed

The Google crawl rate can also be affected by your server response time. With so many new sites and pages being added to the Internet every second, the Google crawler is a busy man and does not spend much time at a web site. If your web site is experiencing slowness, the Googlebot will remember it and come back later with a decreased crawl rate.

If this is your case, you will find a message in Google Webmasters Tools. You can find this message when you go to the Dashboard - Settings screen.

Improve Google crawling rate with a sitemap

If you are still having hard time getting Google to find some of your pages in a reasonable time, you can help the Googlebot by providing a sitemap. Sitemap is a solution to make sure Googlebot crawls your new articles or pages even faster. See here for more details about Google sitemap: How to create a Google sitemap.

How can I find out when Google last crawled my site?

We have a special article dedicated just to this topic, see here: Last time Google indexed a website.

I need help with increasing Google crawl rate

If you are experiencing some difficulties with your Google crawl rate, you are welcome to ask a question in our SEO discussion forum.

You might also be interested in the two following articles:

36 tips on how to improve your AdSense revenue and make money with AdSense (AdSense make money)
How to add URL to Google
Difference between AdSense and AdWords
How to find cheap AdWords
What are the effects of AdSense bid gap on your AdSense earnings?
How to use pagerank sculpting to improve your ranking

and much more, see below.