On August 8th, 2023, OpenAI’s ChatGPT announced information about its new web crawler GPTBot. 

This bot’s function is a web crawler that goes into websites and collects information and data for their AI platform. By doing this, it helps provide more information for their search queries to help provide answers to questions or prompts.

If you are someone who doesn’t want this web crawler to visit and crawl your website, there is a way to protect it. You can use robots.txt to block GPTBot from accessing your website, or parts of it. 

Adding the robots.txt file disallows GPTBot from crawling your site and prevents them from using your content. This same technique can be applied to blocking other Bots such as GoogleBot, BingBot, or other crawlers. 

Share:

administrator

Ansley is the Content Marketing Manager at Boostability. Since graduating from Utah State University with two degrees in Communication Studies and Journalism: Public Relations, Ansley specializes in creating engaging and informative SEO content for readers, customers, and partners through different marketing channels. Along with creating new content, Ansley works to keep content organized and creates and executes new content strategies. When she’s not writing, she loves to travel, visit National Parks, and loves all things Disney.