AI Crawler Firewall & Robots.txt Generator

The DirJournal AI Crawler Firewall is a security utility that generates precise `robots.txt` directives to block known large language models (LLMs) and AI web scrapers from harvesting site content.

Modern AI systems train on the open web. Bots like GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Google-Extended (Gemini and AI Overviews) crawl your pages to build training datasets and to answer user questions directly with your content. Blocking them at robots.txt protects your intellectual property, saves the server bandwidth these bots waste on aggressive re-fetches, and prevents unauthorized ingestion of your writing into commercial LLM training runs.

AI & LLM Crawlers

Block the bots training on your content

Each row maps to one or more User-agent: lines in the output.

OpenAI · GPTBot + ChatGPT-UserGPTBot, ChatGPT-User, OAI-SearchBot
Trains ChatGPT, indexes pages for SearchGPT, and fetches links on user prompts.
Block
Anthropic · ClaudeBotClaudeBot, anthropic-ai, Claude-Web
Trains Claude and fetches pages for Claude.ai citations.
Block
Perplexity · PerplexityBotPerplexityBot, Perplexity-User
Crawls for Perplexity's answer engine and follows links during user queries.
Block
Google-Extended (Gemini / AI Overviews)Google-Extended
Opts your site out of Gemini training + AI Overviews WITHOUT affecting normal Google search.
Block
GoogleOther (research / R&D crawl)GoogleOther
Internal Google crawls for non-search experiments and product R&D.
Block
Applebot-Extended (Apple Intelligence)Applebot-Extended
Opts out of Apple Intelligence training — separate from regular Applebot which powers Siri/Spotlight.
Block
Cohere · cohere-aicohere-ai
Trains Cohere's enterprise LLMs.
Block
ByteDance · BytespiderBytespider
TikTok / Doubao model training crawler. Frequently flagged for ignoring robots.txt — block at the firewall too if you mean it.
Block
Meta · meta-externalagentmeta-externalagent, FacebookBot
Llama training. Separate from facebookexternalhit (link previews) which most sites want to keep allowed.
Block
You.com · YouBotYouBot
You.com answer engine crawler.
Block
DiffbotDiffbot
Knowledge-graph extraction. Resold to multiple LLM training pipelines.
Block
Amazon · AmazonbotAmazonbot
Powers Alexa answers + Amazon's Q model.
Block
Webz.io · OmgilibotOmgilibot, Omgili
Bulk web archive resold to training-data brokers.
Block
Common Crawl · CCBotCCBot
Public web archive used as a base corpus by GPT-3/4, Llama, Falcon, and many open models.
Block

Legacy Search Crawlers

Standard SERP index bots

Allowed by default. Block carefully — these still power the non-AI search results most users rely on.

GooglebotGooglebot
Standard Google Search index. Blocking removes you from google.com — usually not what you want.
Allow
BingbotBingbot
Bing index — also powers ChatGPT's web tool. Blocking has follow-on AI effects.
Allow
DuckDuckBotDuckDuckBot
DuckDuckGo crawls partly via Bing — DuckDuckBot itself does a smaller verification crawl.
Allow
YandexBotYandexBot
Russian-market search index.
Allow

Sitemap URL (optional)

Custom Disallow paths (one per line, optional)

Applied to every allowed crawler (including the catch-allUser-agent: *).

Generated robots.txt

# robots.txt generated by DirJournal AI Crawler Firewall
# https://www.dirjournal.com/tools/robots-llm

# OpenAI — OpenAI · GPTBot + ChatGPT-User
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
Disallow: /

# Anthropic — Anthropic · ClaudeBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
Disallow: /

# Perplexity — Perplexity · PerplexityBot
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /

# Google — Google-Extended (Gemini / AI Overviews)
User-agent: Google-Extended
Disallow: /

# Google — GoogleOther (research / R&D crawl)
User-agent: GoogleOther
Disallow: /

# Apple — Applebot-Extended (Apple Intelligence)
User-agent: Applebot-Extended
Disallow: /

# Cohere — Cohere · cohere-ai
User-agent: cohere-ai
Disallow: /

# ByteDance — ByteDance · Bytespider
User-agent: Bytespider
Disallow: /

# Meta — Meta · meta-externalagent
User-agent: meta-externalagent
User-agent: FacebookBot
Disallow: /

# You.com — You.com · YouBot
User-agent: YouBot
Disallow: /

# Diffbot — Diffbot
User-agent: Diffbot
Disallow: /

# Amazon — Amazon · Amazonbot
User-agent: Amazonbot
Disallow: /

# Webz.io — Webz.io · Omgilibot
User-agent: Omgilibot
User-agent: Omgili
Disallow: /

# Common Crawl — Common Crawl · CCBot
User-agent: CCBot
Disallow: /

# Google — Googlebot
User-agent: Googlebot
Allow: /

# Microsoft — Bingbot
User-agent: Bingbot
Allow: /

# DuckDuckGo — DuckDuckBot
User-agent: DuckDuckBot
Allow: /

# Yandex — YandexBot
User-agent: YandexBot
Allow: /

# All other crawlers
User-agent: *
# noai / noimageai opt-out signals (informal but recognised by Cloudflare, Bytedance, others)
# X-Robots-Tag: noai, noimageai
Allow: /

Upload to the root of your domain at/robots.txt. Some bots ignore robots.txt — pair with WAF / Cloudflare rules if you need a hard block.

Why Block AI Scrapers?

Blocking unauthorized LLM crawlers protects your original content as intellectual property — once a page is ingested into a model's training set, that text is effectively relicensed to anyone who queries the model.

It also reduces server load. AI scraping bots refetch aggressively and ignore the politeness conventions that traditional search crawlers respect, so blocking them at robots.txt cuts wasted bandwidth and origin load.

How to Implement Your Rules

1Select the AI bots to block. Toggle the user agents you want disallowed — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others.
2Copy the generated directives. The tool emits a ready-to-paste block of User-agent + Disallow: / pairs.
3Paste them at the top of your domain's robots.txt file. Placement at the top ensures the specific agent rules are read before any later wildcard fallback.

Frequently Asked Questions

What is an AI crawler firewall?

An AI crawler firewall is a robots.txt-based access control that explicitly disallows known AI training bots and LLM scrapers from reading your site. It generates the precise User-agent and Disallow directives needed to block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and similar agents at the protocol level, before they ever request a page.

Does blocking GPTBot affect Google search rankings?

No, traditional search crawlers like Googlebot use entirely different user agents than AI training bots. Blocking GPTBot, ClaudeBot, or Google-Extended (the opt-out token for Bard/Gemini training) does not change what Googlebot indexes for classic search results, so your ranking signal is unaffected.

How do I block ChatGPT from scraping my website?

You must disallow 'GPTBot' and 'ChatGPT-User' in your robots.txt file. Use our generator to create the exact syntax to block OpenAI's crawlers.

Does blocking Google-Extended hurt my SEO?

No. Blocking 'Google-Extended' prevents your site from being used to train Google's AI models, but it does not affect how 'Googlebot' crawls and indexes your site for traditional Google Search.

What is CCBot?

CCBot is the crawler for Common Crawl, a massive open web archive frequently used as a base dataset to train large language models (LLMs) like GPT and Llama. Blocking it is a common step in preventing AI data scraping.