robots.txt and AI bots: which ones to let through?

The robots.txt file tells bots what they can crawl on your site. With AI, it carries a new decision: let your pages be used to train models, let them be used to answer people’s questions, both, or neither.

Good news: at most companies, these two uses go through different bots. So you can choose one without the other.

What robots.txt does

The protocol has been standardized since September 2022 by the IETF, in RFC 9309. The essentials:

• the file is called robots.txt, in lowercase, and sits at the root of the site: your-domain.com/robots.txt;
• it is read in groups: one or more User-agent lines, which name the bots, then Allow and Disallow lines, each followed by a path;
• its rules are requests. The standard specifies that robots.txt grants no access authorization: an honest bot respects it, a malicious bot can ignore it;
• it is public. Never list in it a folder you want to keep secret: you would be pointing it out to everyone.

Google also points out that robots.txt governs crawling. To keep a page out of its results, you need a noindex tag or a password, because a disallowed page can still appear there when other sites link to it.

Three kinds of AI bots

1. Training bots collect pages that may be used to train future models. Blocking them amounts to refusing that use.
2. Search bots index pages so the assistant can cite them, with a link. Blocking them removes you from those answers.
3. On-demand visits happen when a user asks the assistant to read a page. OpenAI, Perplexity and Meta warn that robots.txt may not apply to these visits, which a person triggered. Anthropic, on the contrary, says it respects robots.txt for all its bots.

Two names deserve an explanation. Google-Extended and Applebot-Extended are mere names to use in robots.txt: no bot by that name visits your site. They say what Google and Apple can do with the pages Googlebot and Applebot have already crawled.

• Google-Extended covers the training of Gemini models and the grounding of the Gemini app’s answers. It touches neither Google Search nor ranking. Refusing it doesn’t remove you from AI Overviews, which go through Googlebot, but can keep you out of the Gemini app.
• Applebot-Extended covers the training of Apple’s models. Refusing it leaves your pages findable by Spotlight, Siri and Safari, and Apple specifies that this choice doesn’t count in its ranking.

The rules that catch people out

• A bot that finds its name ignores the general group. The group starting with an asterisk only applies to bots without a group in their name. If you write a group for GPTBot, copy into it the general disallows that should apply to it too.
• The most specific rule wins. Between Allow and Disallow, the longest path prevails; in a tie, Allow goes through.
• A broken file blocks everything. If robots.txt answers with a server error, the standard asks bots to treat the whole site as disallowed. If it is simply missing, everything is allowed.
• A change takes up to a day. The standard lets bots keep the file in memory for up to 24 hours, and OpenAI and Perplexity both announce that delay.
• Case doesn’t matter in a bot’s name, but it does in paths: /Photos/ and /photos/ are two different paths.

Examples to copy

Allow everything, AI included. Two lines are enough:

User-agent: *
Allow: /

Refuse training, stay in the answers. One group can name several bots:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

The search bots aren’t listed: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Applebot follow the general group, or have no rule if there isn’t one. Remove Google-Extended from the list if you want to stay in the Gemini app’s answers.

Refuse a single search bot, for example to test its effect:

User-agent: PerplexityBot
Disallow: /

Save the file as plain text, encoded in UTF-8, at the root of the site.

Your host’s or CDN’s settings

An open robots.txt is useless if the service that delivers your site refuses bots before they even read it. Google advises checking that crawling is allowed in robots.txt, but also by the CDN and the host.

Cloudflare, a very widespread CDN, documents its settings precisely:

• On July 1, 2025, it started blocking by default AI bots that didn’t have the site’s permission, and asking each new domain, at sign-up, whether it wanted to let them through.
• Since September 15, 2026, its settings distinguish search, agents and training. Each can be allowed, blocked everywhere, or blocked only on pages that show ads.
• For a new domain, Cloudflare offers two presets. Without ads, everything is allowed. With ads, training is refused, agents are blocked on the pages that show them, and search stays allowed.
• Its managed robots.txt can add lines at the top of your own file when you turn on these preferences.

If your site goes through Cloudflare, open these settings and check that search is allowed. With another host, look for an option that mentions bots or AI, and ask support when in doubt.

The right setting for a local business

• Let search bots through. They are the ones that get your address, your opening hours and your prices cited in answers.
• Training is a matter of principle. At OpenAI, Anthropic, Apple and Meta, refusing it changes nothing in search. At Google, refusing Google-Extended also distances you from the Gemini app.
• Allow on-demand visits: behind them, there is a person who wants to read your page.
• Reread your robots.txt after every change of host or site, and watch in Search Console for pages blocked by mistake.

In short: separate training from search. Let OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Applebot through, refuse training bots if you wish, and check that your host isn’t blocking anything without your knowledge.