Watch this Booster Seat episode for more updates on this topic!
After doing SEO for hundreds of websites, I realize there are many reasons to ask: “how often Google is crawling my website?” Some website owners wonder how often they should be making changes to their website and they try to base their decision off of how frequently Google visits. Others simply wonder whether Google has noticed them and their SEO efforts at all. Luckily, this blog post does double duty and answers both.
Finding the Crawl Stats for Your Website
The answer of how frequently Google crawls your site can easily be found in your Google Webmaster Tools account (Google Webmaster Tools is a free service provided by Google, so if you don’t have it, go to google.com/webmaster/tools, where Google will walk you through how to set it up). Once you’ve logged in to your Google Webmaster Tools account, navigate to the “Crawl” menu item on the left, followed by the “Crawl Stats” sub item after that. Tweet This
You should see something similar to the image below:
As you’ll notice, Google Webmaster Tools only stores historical data for up to 90 days. But this should give you all the info you need to discover Google’s crawling habits related to your site. This particular website is a moving truck rental company, and here we can see that Google visits them almost every day and crawls an average of 17 pages each day.
How Do I Get Google to Crawl My Site More Often?
This is what people really want to know when they ask “how often is Google crawling my site?” And if it is not the real question, it’s what they want to know once they find out how frequently Google visits their site. There are a few ways to encourage Google to pay your website more frequent visits.
Server Connectivity and Page Load Time
I have a confession: I didn’t choose the above website’s stats at random. Rather, I did so to make one very important point. If your website has connectivity errors (meaning that Google can’t access the site when it tries to because your site is down or its servers are down), you will not like the results. Google will stop coming to your site, your rankings will plunge, and as a result, your traffic will go down the drain. Tweet This This should make perfect sense – if Google can’t access your site for a long period of time, why should they come back? And why should they send anyone your way, if your site is down? Let’s take a look at the server connectivity and crawl errors for this client to see what’s going on:
The blue line represents the percentage of time that Google tried to visit our truck rental site, but couldn’t pull up the website because it was down or timing out. We can see that the problems started in August, where Google tried time and time again to visit this website, with a sharp spike in crawl errors.
If we compare these stats to the first stats we saw, we can see that after a few days of trying and only being partially successful, Google tried to visit less and less, until, for many days, it didn’t try at all.
Once we fixed the crawl errors (early September), you’ll notice that Google warms up to the site again.
Frequent Site Changes
If you’re constantly adding new material to your website, Google won’t want to stay away; it won’t want to miss anything new that you post for fear of not having your site fully indexed in its search engine. Adding fresh content to your site is the easiest way to entice Google to come back and come back often. Tweet This Frequently updating a news site makes sense, but small- and medium- sized business websites may have a tougher time coming up with ways to generate new content. Luckily, the addition of a blog to your site can give you the perfect platform for adding new material as often as you want. And yes, you can blog about anything, including replacement parts for hydraulic equipment (speaking from experience).
Sitemaps can do more than just tell Google what pages you have on your site. They can also tell Google how frequently you update the content and therefore, how frequently Google should come back and check in on you. There is a bit of debate on how much influence this actually has on Google.
On one hand, you’re telling them something directly. On the other hand, Google may ignore this and simply pay attention to your actual habits rather than your stated intentions. (We don’t have to look much farther than New Years’ Resolutions to see how reliable stated intentions are.)
Nevertheless, Google’s bots do read through the sitemap, and if your habits and stated intentions aren’t too dissimilar, this could give you some extra pull. Most sitemap generators, such as xml-sitemaps.com, will let you create a sitemap for free. They also have fields for how frequently you plan on changing or adding to the content on your site.
Getting others to link to your website is a great way to get Google to notice you (though you have to be careful about how you go about this or you might invite the wrath of Google’s Penguin Update, which has been known to devastate sites with poor linking habits). The bottom line is that if others are noticing you and linking to your site, Google wants to notice you too. And in a search world dominated by Google, that’s a pretty good start to growing your online presence.
Read more about Google Webmaster Tools and the SEO Importance of On-site Content in some of our other blog posts designed to help you rock the SEO world. Could you use some help making sure your website is reaching its full potential? Our online marketing consultants can help answer any questions you have, and it’s free!