On August 8th, 2023, OpenAI’s ChatGPT announced information about its new web crawler GPTBot.
Breaking 🚨
OpenAI just launched GPTBot, a web crawler designed to automatically scrape data from the entire internet.
This data will be used to train future AI models like GPT-4 and GPT-5!
GPTBot ensures that sources violating privacy and those behind paywalls are excluded. pic.twitter.com/oR3kY4buaU
— Shubham Saboo (@Saboo_Shubham_) August 7, 2023
This bot’s function is a web crawler that goes into websites and collects information and data for their AI platform. By doing this, it helps provide more information for their search queries to help provide answers to questions or prompts.
If you are someone who doesn’t want this web crawler to visit and crawl your website, there is a way to protect it. You can use robots.txt to block GPTBot from accessing your website, or parts of it.
PSA – If you’ve implemented an AI chatbot on your site, and you don’t want your its output to be seen as a part of your site for indexing: use a robotted iframe, a robotted JavaScript file / resource, or at maybe use data-nosnippet to block it in the snippet.
— John Mueller (official) · #MaybeABot (@JohnMu) August 7, 2023
Adding the robots.txt file disallows GPTBot from crawling your site and prevents them from using your content. This same technique can be applied to blocking other Bots such as GoogleBot, BingBot, or other crawlers.
Stay Informed on the Latest AI News
The search landscape and world of AI evolve quickly. Check out our free resources below to stay informed on the latest AI best practices, product advancements, and more!
- What is ChatGPT?
- SEO & AI: 6 Ways AI Will Affect your SEO Strategy
- Best AI Search Engines to Use
- AI Differences Between ChatGPT, Google’s Bard, and Jasper
Partner with the Global Leaders in White Label SEO
Let Boostability do the heavy lifting for you with our white label SEO services. We have been the global leaders in this industry for over 15 years, having helped thousands of businesses increase their online presence.
Get in touch with our team to learn how we can support and scale your agency while achieving your clients’ goals. Learn more about what white label SEO marketing is, what to expect, and more!
