On August 8th, 2023, OpenAI’s ChatGPT announced information about its new web crawler GPTBot. 

This bot’s function is a web crawler that goes into websites and collects information and data for their AI platform. By doing this, it helps provide more information for their search queries to help provide answers to questions or prompts.

If you are someone who doesn’t want this web crawler to visit and crawl your website, there is a way to protect it. You can use robots.txt to block GPTBot from accessing your website, or parts of it. 

Adding the robots.txt file disallows GPTBot from crawling your site and prevents them from using your content. This same technique can be applied to blocking other Bots such as GoogleBot, BingBot, or other crawlers.

 

Stay Informed on the Latest AI News

The search landscape and world of AI evolve quickly. Check out our free resources below to stay informed on the latest AI best practices, product advancements, and more!

 

Partner with the Global Leaders in White Label SEO

Let Boostability do the heavy lifting for you with our white label SEO services. We have been the global leaders in this industry for over 15 years, having helped thousands of businesses increase their online presence.

Get in touch with our team to learn how we can support and scale your agency while achieving your clients’ goals. Learn more about what white label SEO marketing is, what to expect, and more!

Share:

administrator

Ansley is the Content Marketing Manager at Boostability. Since graduating from Utah State University with two degrees in Communication Studies and Journalism: Public Relations, Ansley specializes in creating engaging and informative SEO content for readers, customers, and partners through different marketing channels. Along with creating new content, Ansley works to keep content organized and creates and executes new content strategies. When she’s not writing, she loves to travel, visit National Parks, and loves all things Disney.