What is AI crawler?
Also called: AI bots, GPTBot, OAI-SearchBot, Google-Extended
AI crawlers are programs that fetch web pages automatically so that AI providers can use them for training, search or answers to user questions.
Example
An online shop allows the search crawler OAI-SearchBot in its robots.txt but blocks GPTBot, which collects content for training models.
For pros
Many providers separate their crawlers by purpose. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Google-Extended controls whether content is used for training and grounding in other Google systems such as Gemini; regular Google Search is not affected. Control works via robots.txt under the Robots Exclusion Protocol (RFC 9309). That is a request, not a technical barrier. Sites that block search crawlers appear less often as a source in AI answers.
Written by AI Compare Ninja with AI assistance and edited by us. Every explanation is our own text.