Dealing with Spiders—How, When, and Why Google Crawls Your Site

Dealing with Spiders—How, When, and Why Google Crawls Your Site

Does Google even know my page exists, let alone that I’ve been working really hard on my SEO over the past year? Should I update my site this month or next—does it matter? What are Google spiders, and how do they even “crawl” sites?

For several years it seemed that exactly how Google search engines worked was an impossible mystery. But nowadays Google fills us in on its methods and provides several tips for getting higher search engine result rankings. Here are some of the basics:

 

How Does Google Crawl My Site?

It is a widely accepted concept that Google and other search engines have little bots that are crawling through the whole Internet. They look at each and every site to decide its value and gather information to help create the complex algorithms that eventually determine the site’s rank on a search engine results page. These little bots have a huge job!

These bots are not actual robots. They are little bits of code that go from site to site, reading more code. Google has dubbed these floating bits of code as bots or spiders. These Google spiders are crawling all over the web, including your site, combing through the code and relaying what they find back to the search engines..

 

How Often Does Google Crawl My Site?

Naturally, there are factors that cause some sites to be crawled more frequently than others. Google is fairly open about their spiders and how often and why they crawl sites. They have a whole page, called the Webmaster Guidelines, dedicated to helping people learn about how the spiders crawl sites.

There are so many factors at play when it comes to how, when, and why Google crawls your site that there is no hard-and-fast strategy that applies to every site. In order to see how often Google is crawling your site, you will need to access your free Google Webmaster Tools account, and find the “Crawl” menu item to see the crawl stats for your site. There you’ll be able to see how often Google has crawled your site in the past 90 days.

By accessing the free Webmaster Tools from Google, you’ll have access to metrics and other information that will help you determine how often your specific site is being crawled and what information the spiders are sending back to Google about your site.

A good tip to remember is that the frequency of the Google spiders visiting your site depends on how often you update it. For sites that are constantly adding and updating content, the Google spiders will automatically crawl more often—sometimes multiple times a minute! However, for a small site that is rarely updated, the Google bots will only crawl every few days.

 

Do I Want Google to Crawl My Site?

In order for the time you put into your site’s organization and SEO work to reflect on search engine results pages, you need the spiders to be crawling your site. You need Google to start crawling your site immediately and with as much frequency as possible. Without Google’s bots regularly crawling your site, you have no chance of being ranked high at all.

The more the bots crawl your site, the more they will trust your site and relay the information back to Google for search results pages. And the better the SEO tactics you use on your site and the more the Google bots are crawling, the higher your site will continue to rank.

Google’s Webmaster Tools Help page offers some tips for making sure that the bots crawl your site regularly:

  • Make sure your site includes plenty of links to other credible, relevant, high-authority websites.
  • Make sure your website has a clear sitemap and a straightforward organization to give Google’s bots easy access to every page on your site. You can also submit your sitemap to the Google index to ensure that the Google bots know to look for your site as they crawl the Internet.
  • If your site is consistently down when Google tries to crawl it, then Google will learn to ignore it, even when it’s functioning properly.

If you build out your site properly and incorporate basic, white hat SEO tactics, the Google bots are more likely to crawl it. And the more often spiders crawl your site, the more likely Google will be to rank it high in search engine results pages.

Mike Marsh
[email protected]

Mike has been in the online marketing industry since 2012. He has extensive experience with SEO, email marketing and linkbuilding campaigns for in-house and agency teams around the country. You can connect with him right here: Linkedin