# robots.txt for AI crawlers: a directive-by-directive reference

> Every AI user agent worth naming, what each one controls, the precedence rules that decide which directive wins, and the wildcard mistake that quietly removes sites from AI answers.

- Source: https://shiftrank.ai/blog/robots-txt-for-ai-crawlers
- Published: 2026-07-11
- Updated: 2026-08-07
- Author: Shiftrank
- Category: Crawlers
- Tags: robots.txt, crawlers, gptbot, claudebot

---
`robots.txt` is thirty years old and was never designed for the question we now ask of it: not "may you index this" but "may you learn from this, and may you read it aloud to someone". Operators have adapted by issuing separate tokens per purpose. The result is a file that has to be written more carefully than it used to be.

## The agents

### OpenAI

| Token | Purpose |
|---|---|
| `GPTBot` | Bulk crawl for model training |
| `ChatGPT-User` | Live fetch during a user's conversation |
| `OAI-SearchBot` | Index for ChatGPT search |

### Anthropic

| Token | Purpose |
|---|---|
| `ClaudeBot` | Bulk crawl for model training |
| `Claude-User` | Live fetch during a user's conversation |
| `Claude-SearchBot` | Index for Claude search |

### Google

| Token | Purpose |
|---|---|
| `Googlebot` | Search index; also feeds AI Overviews |
| `Google-Extended` | Not a crawler — a control token for Gemini training use |

`Google-Extended` is the one people get wrong. It will never appear in your logs, because no request is ever made under that name. It is a switch that governs how content `Googlebot` already fetched may be used for Gemini. Disallowing it does not affect your search ranking, and it does not remove you from AI Overviews, which are served from the search index.

### Others

| Token | Operator | Purpose |
|---|---|---|
| `PerplexityBot` | Perplexity | Index |
| `Perplexity-User` | Perplexity | Live fetch during an answer |
| `Applebot` | Apple | Siri and Spotlight |
| `Applebot-Extended` | Apple | Control token for Apple model training |
| `CCBot` | Common Crawl | Public archive used by many training datasets |
| `Bytespider` | ByteDance | Training crawl; widely reported to be aggressive |
| `meta-externalagent` | Meta | Training crawl |
| `Amazonbot` | Amazon | Alexa and index |

`CCBot` deserves attention out of proportion to its traffic. Common Crawl is a nonprofit archive, and a great many training datasets are derived from it. Allowing `CCBot` while blocking `GPTBot` is a coherent position, but it is not the position most people think they are taking.

## The precedence rule that breaks files

**A crawler obeys exactly one group: the most specific user-agent match. Groups do not merge.**

This one rule causes most broken `robots.txt` files. Consider:

```
User-agent: *
Disallow: /admin/
Crawl-delay: 10

User-agent: GPTBot
Disallow: /pricing
```

`GPTBot` here is allowed to crawl `/admin/`. It matched its own group, so the wildcard group — including the `/admin/` disallow — does not apply to it at all. Anything you want a named agent to obey has to be restated inside that agent's group:

```
User-agent: GPTBot
Disallow: /admin/
Disallow: /pricing
```

Within a group, the most specific path rule wins regardless of order, and `Allow` beats `Disallow` on ties. So this permits `/blog/` while blocking everything else:

```
User-agent: GPTBot
Disallow: /
Allow: /blog/
```

Other mechanics worth knowing: `*` matches any sequence and `$` anchors the end, so `Disallow: /*.pdf$` blocks PDFs. Matching is case-sensitive on paths and case-insensitive on user-agent tokens. And `robots.txt` is per host and per scheme — `example.com` and `www.example.com` need their own files unless one redirects to the other.

## Three configurations

**Open, with the application fenced off.** The default for most B2B software. Marketing and docs are the point; the app is not.

```
User-agent: *
Allow: /
Disallow: /app/
Disallow: /api/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
```

**No training, yes retrieval.** For publishers who want citations but not corpus inclusion.

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /
```

**Training on the archive, not on premium.** For a site with both free and paid content.

```
User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /research/
Disallow: /members/
```

## The mistake that removes you from AI answers

Someone reads a headline about AI scraping, adds a broad block, and ships it:

```
User-agent: *
Disallow: /
```

That does block `GPTBot`. It also blocks `ChatGPT-User`, `Perplexity-User`, and `Claude-User` — the agents that fetch your page *because a real person asked a question you could answer*. The company has not protected an asset; it has opted out of its own category's AI answers, and because nothing in its analytics ever showed that traffic, nobody notices for two quarters.

If you take one thing from this: decide about training and about retrieval separately, and write the file so those decisions are visible.

## Where robots.txt stops

It is a request, not a fence. There is no enforcement, and compliance is a policy choice by each operator. Well-behaved crawlers honour it; scrapers impersonating well-behaved crawlers do not, and the user-agent string they send costs them nothing to fake.

Enforcement means inspecting requests where you can actually reject them: WAF rules, edge middleware, or a proxy in front of your origin, matching on verified IP ranges rather than the user-agent header alone. Shiftrank does this at the edge and — more usefully at first — runs in observe mode, so you can see exactly which agents *would* have been blocked before any rule takes effect. Getting that order right matters, because a `robots.txt` mistake is silent and a blocking-rule mistake is not.
## FAQ

### How do I block AI crawlers in robots.txt?

Add a group per user agent with a Disallow rule. For example, "User-agent: GPTBot" followed by "Disallow: /" blocks OpenAI's training crawler from the whole site. Each AI operator uses distinct tokens for training, live retrieval, and search indexing, so blocking one does not block the others — you must name each agent you intend to affect.

### Which robots.txt group applies when a crawler matches more than one?

Crawlers follow the single most specific matching user-agent group and ignore all others, including the wildcard group. If your file contains both "User-agent: *" and "User-agent: GPTBot", GPTBot obeys only the GPTBot group. Rules do not merge, so any directive you want to apply to a named agent must be repeated inside its own group.

### Does robots.txt legally prevent AI training on my content?

No. robots.txt is a voluntary protocol with no enforcement mechanism. Major operators publicly honour it, but a crawler that ignores it faces no technical barrier. If you need enforcement rather than a request, you need edge rules or WAF policies that inspect and block requests, backed by IP verification.