Many site owners now ask the same question: should I let AI crawlers onto my website? The short answer is that it depends on which crawler you mean. Treating them all as one thing leads to two common mistakes: blocking everything and disappearing from AI search, or allowing everything without ever deciding whether you are comfortable with it.
What robots.txt does
robots.txt is a plain text file at the root of your site, for example omnishown.com/robots.txt. It lists crawlers by name, called user agents, and says which parts of your site each may visit. Reputable crawlers read it before they fetch anything and follow it.
It is a request, not a lock. The file is public, and a crawler that chooses to ignore it is not stopped by it. If a page must stay private, put it behind a login. robots.txt is for managing well-behaved crawlers, not for security.
Five kinds of AI crawler
The big AI providers do not run one crawler each. They run several, with different jobs. Knowing the job tells you what allowing or blocking it is likely to mean.
| Kind | Examples | What it is for |
|---|---|---|
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Finds pages that may be shown or cited in an AI product's search results. |
| On demand | ChatGPT-User, Claude-User, Perplexity-User | Fetches one page when a person asks the assistant to look at it. |
| Training | GPTBot, ClaudeBot | Collects pages that may be used to train models. |
| Control token | Google-Extended, Applebot-Extended | Not a separate crawler. A switch that lets you opt out of certain AI uses of content the provider already crawls. |
| Dataset | CCBot | Builds an open web dataset that many AI projects draw on. |
Crawler names and behaviour change, and each provider documents its own. Treat the table as a starting point and confirm against the provider's current page before you rely on it.
What blocking each kind is likely to do
Blocking a search crawler makes it harder for that product to find your pages when it builds an answer, so you are less likely to be cited there. If being named by AI is part of how customers find you, this is the one to be careful with.
Blocking an on-demand crawler means that when someone asks an assistant to read your page, it may be unable to. That can be fine for a private tool and awkward for a public service page.
Blocking a training crawler is a statement about how your content is used to build future models. Providers describe training crawlers as separate from their search crawlers, so many sites allow search and decline training. That is a choice about your content, not an SEO tactic.
Using a control token such as Google-Extended opts you out of certain AI uses. Google says it does not affect how your site appears in Search.
How to choose: three questions
- Do you want to be named by AI? If buyers might ask an assistant for businesses like yours, keep the search crawlers allowed.
- Is your content something you sell or protect? Original research, paid courses and premium articles may be reasons to decline training crawlers. A local clinic's service page usually is not.
- Who can change the file? robots.txt is code. A redesign or a template change can overwrite it without anyone noticing, so someone has to own it.
How to set it up
- Decide, crawler by crawler, which to allow. The AI-crawler robots.txt generator lists the common ones with a one-line note on each and builds the file in your browser.
- Save the result as
robots.txtat the root of your site. - Open
yoursite.com/robots.txtin a browser to confirm it is live and says what you meant. - Optionally add an llms.txt file. It is a proposal, not a standard, and no provider has committed to reading it, but it costs nothing.
- Recheck the file after every release.
Allowed is not the same as named
Letting a crawler read your site removes a barrier. It does not make an AI product recommend you. Whether an answer names your business depends on how clearly your pages state the facts, how consistently the web describes you, and the question being asked. Answers also change from one run to the next.
That is why the useful next step is to measure. Omnishown Watch asks AI your buyers' questions on a schedule and records which answers name you, so you can see whether changes make a difference. Omnishown Check reviews each push to your repository for SEO, AEO and GEO issues, including pages AI crawlers cannot read, and Omnishown Fix sends crawler-access changes in robots.txt as a pull request that you approve. For the wider picture, read the AI visibility guide and the glossary.