crawl

The act of scraping and indexing content from the internet for the purpose of serving search results to a search engine and browser. There are many types of crawls but one of the most common is a website crawl. This type will start at the homepage and grab links from there to search through a whole site. Each link followed from that point is another level in depth.

Each website has what is called a “crawl budget” that is determined by the bot. A crawl budget is the maximum number of pages that the crawler will visit on a website. This typically is only an issue for larger sites but can affect the indexation of pages further down the crawl.