Before deciding which OpenAI crawlers to block check what each one is doing. According to OpenAI's documentation GPTBot is the one that gathers the training data. Others are used to index pages for ChatGPT's real time answers, inform search, or check pages that are related to ads. If you block all four you might lose citations which can help drive traffic back to your source pages.
OpenAI documents each crawler's product token (the name in its User-Agent, such as GPTBot), User-Agent, and purpose. We read that alongside our own research to give each one a CrawlerUsage value, such as Train, Index, Input, Search, or Analytics, and an IsArtificialIntelligence flag.
GPTBot
GPTBot gathers content that may train OpenAI's generative models, so its usage is Train. What it takes can become part of a model and used to create answers that compete with your content and don't send traffic back to you.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
IsArtificialIntelligence returns "True".
OAI-SearchBot
Your pages can be cited in ChatGPT answers if you allow OAI-SearchBot. It builds the index used by ChatGPT's search features. OpenAI says sites that disallow it in robots.txt aren't shown in those answers, though they can still appear as navigational links. We classify its usage as Index plus Input rather than Search because the index feeds AI answers. A traditional search index gives readers a list of links and sends them to the source.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
IsArtificialIntelligence returns "True".
ChatGPT-User
OpenAI advises that rules in robots.txt may not apply to ChatGPT-User because the request is happening at the request of a person. They also advise that they don't crawl the web at scale with this crawler token. We associate this crawler with the Input usage. You can then decide how you wish to treat these requests, either specifically for OpenAI or all agents that are associated with the Input usage.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
IsArtificialIntelligence returns "True".
OAI-AdsBot
OAI-AdsBot checks the safety and relevance of pages submitted as ads on ChatGPT. OpenAI states the data is not used to train its generative AI foundation models. Its usage is Analytics, the value for crawlers that gather data for marketing analytics.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
Set your crawler rules in robots.txt
Robots.txt is a request, like a "Keep off the grass" sign, and nothing enforces it. RFC 9309, the standard that defines it, says its rules "are not a form of access authorization". 51Degrees can enforce it, and send a crawler that ignores it to the licensing terms for that content, but that's a subject for another post. So be precise about which usages you allow, and put your terms in the file the crawler actually reads. Our free generator knows 433 crawlers, OpenAI's included. Pick the usages you allow and it writes the crawler names for you, plus a Terms Document Locator line pointing at your terms, such as the Search Only Contract. Trusted Reviews, a UK publisher, told Press Gazette how it logs the bots that ignore its contract.
See them in your analytics
To understand which OpenAI crawler agents reach your pages add CrawlerName, CrawlerUsage, and IsArtificialIntelligence as custom dimensions in Google Analytics 4. Once you understand frequency and what they're consuming you can decide what to allow, block, or license.
AI treatment
For ways to handle AI crawlers beyond robots.txt, read our guide to the good, the bad, and the ugly.