Robots.txt

A standardized way websites communicate with web crawlers about which areas of the website should be crawled and which shouldn’t. Not all crawlers follow the Robots.txt directives as there are some that scan for security vulnerabilities. Not to be confused with a sitemap, robots.txt file is often used in conjunction with traditional XML sitemaps.