robots.txt for AI Crawlers: The Complete Guide to GPTBot, ClaudeBot, and PerplexityBot

You probably have a robots.txt. It's probably configured for Google. And you probably never considered whether ChatGPT or Perplexity can index you.

This gap is critical for GEO — and yet it can be fixed in under an hour.

Why robots.txt matters for GEO

For AI systems to cite you, they have to know your content. And to know it, their crawlers (indexing bots) have to visit and process it.

Every major AI system has its own crawler:

If any of these crawlers is blocked in your robots.txt — or if there is no explicit directive for it — the outcome depends on each system's default behavior. And that may not work in your favor.

How to check your current state

Open yourwebsite.com/robots.txt in a browser. Check:

1. Is there a User-agent: GPTBot directive? If not, GPTBot follows the default rule (usually User-agent: *)
2. Is the permission explicit (Allow: /) or is it a block (Disallow: /)?
3. Are GPTBot and the other AI crawlers blocked under a general User-agent: * with Disallow: /?

If your website uses a security plugin or a WAF (Web Application Firewall), check there as well — some configurations block unknown crawlers across the board.

How to configure robots.txt for AI properly

The recommended configuration for maximum AI accessibility:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /private/

# OpenAI
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Google (Gemini)
User-agent: Google-Extended
Allow: /

# Cohere
User-agent: cohere-ai
Allow: /

Sitemap: https://yourwebsite.com/sitemap.xml

The key principles:

• An explicit Allow: / for every AI crawler — don't rely on default behavior
• Block api/, admin/, and similar sections that AI shouldn't index
• Always include a link to your sitemap

What to block, what not to

The most common mistake: blocking too much. The second most common mistake: blocking too little.

Block AI crawlers from:

• /api/ — API endpoints aren't content worth citing
• /admin/ — administration interfaces
• /checkout/, /cart/ — transactional pages
• /user/, /account/ — user profiles
• URLs with dynamic parameters (?session=, ?token=)

Keep accessible:

• All content pages (blog, products, services)
• About and Contact pages
• FAQ and documentation
• Landing pages

robots.txt isn't enough — add llms.txt

robots.txt tells AI crawlers where they may go. llms.txt tells them what they'll find there and how to think about you. Both are part of a complete technical GEO configuration.

A typical mistake: a company fixes its robots.txt but doesn't deploy llms.txt. The result: the AI crawler visits the site but gets no structured context about the company — and has to piece it together on its own from mixed content.

Verification after implementation

After changing your robots.txt:

1. Google Search Console → "Test robots.txt" — syntax verification
2. Direct test: curl -A "GPTBot" https://yourwebsite.com/robots.txt — simulating a GPTBot request
3. robots.txt tester (freely available online tools) — complete validation
4. Check your server logs after 48–72 hours — to see whether the crawlers are actually showing up

Results in AI answers aren't immediate — AI systems re-index content at varying intervals, typically over weeks to months.

The ethical dimension: why block AI crawlers?

There are legitimate reasons to block AI crawlers — especially for content publishers who don't want their work training AI models without compensation.

For companies that want to be visible in AI answers, blocking is counterproductive. You can't block GPTBot and expect ChatGPT to cite you at the same time.

The decision to block or allow AI crawlers is strategic — and it should be a conscious one, not the result of an overlooked configuration.

Frequently asked questions

Which AI crawlers should I allow? For visibility in AI answers, at minimum OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot, and ClaudeBot. Training bots (GPTBot, Google-Extended) can be managed independently according to your strategy.

What is the difference between GPTBot and OAI-SearchBot? GPTBot collects data for model training; OAI-SearchBot indexes the web for search and citations in ChatGPT. You can block training while staying visible in search — they are separate rules.

Will blocking AI crawlers hurt my Google rankings? No — Googlebot is a separate crawler, and blocking AI bots doesn't affect it. You will, however, lose visibility in ChatGPT, Perplexity, and other AI systems that influence a growing share of purchase decisions.

Is allowing the crawlers enough to appear in AI answers? No. Robots.txt only opens the door — what decides citations is content citability, entity signals, and llms.txt as a complement.

---

Technical GEO configuration — robots.txt, llms.txt, schema markup — is part of every GEO Audit we do. If you're not sure about your current state, we'll be happy to check it for you.