crawler

A software application that fetches, analyzes and files information from web servers. A web crawler systematically browses the internet for the purpose of indexation. Because of the enormous size of the internet even the biggest and most complex crawler falls short of a fully complete index. A robots.txt file can request that crawlers only visit specific parts of a website or nothing at all.