GoogleBot

The web crawler software that Google uses to collect documents and build a searchable index. GoogleBot is characterized by two different types of web crawlers: a desktop crawler and a mobile crawler. You can use your robots.txt file to give specific instructions to the GoogleBot or other web crawlers on how to crawl your site.

As of July 1, 2019 Google announced that all new websites crawled by GoogleBot will be on a mobile-first index standard. This means that when a site is crawled that the mobile version of the site will be served to their index. Making your site “mobile friendly” is now more important than ever to avoid indexation issues.