The DirJournal AI Crawler Firewall is a security utility that generates precise `robots.txt` directives to block known large language models (LLMs) and AI web scrapers from harvesting site content.
Modern AI systems train on the open web. Bots like GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Google-Extended (Gemini and AI Overviews) crawl your pages to build training datasets and to answer user questions directly with your content. Blocking them at robots.txt protects your intellectual property, saves the server bandwidth these bots waste on aggressive re-fetches, and prevents unauthorized ingestion of your writing into commercial LLM training runs.
AI & LLM Crawlers
Each row maps to one or more User-agent: lines in the output.
Trains ChatGPT, indexes pages for SearchGPT, and fetches links on user prompts.
Trains Claude and fetches pages for Claude.ai citations.
Crawls for Perplexity's answer engine and follows links during user queries.
Opts your site out of Gemini training + AI Overviews WITHOUT affecting normal Google search.
Internal Google crawls for non-search experiments and product R&D.
Opts out of Apple Intelligence training — separate from regular Applebot which powers Siri/Spotlight.
Trains Cohere's enterprise LLMs.
TikTok / Doubao model training crawler. Frequently flagged for ignoring robots.txt — block at the firewall too if you mean it.
Llama training. Separate from facebookexternalhit (link previews) which most sites want to keep allowed.
You.com answer engine crawler.
Knowledge-graph extraction. Resold to multiple LLM training pipelines.
Powers Alexa answers + Amazon's Q model.
Bulk web archive resold to training-data brokers.
Public web archive used as a base corpus by GPT-3/4, Llama, Falcon, and many open models.
Legacy Search Crawlers
Allowed by default. Block carefully — these still power the non-AI search results most users rely on.
Standard Google Search index. Blocking removes you from google.com — usually not what you want.
Bing index — also powers ChatGPT's web tool. Blocking has follow-on AI effects.
DuckDuckGo crawls partly via Bing — DuckDuckBot itself does a smaller verification crawl.
Russian-market search index.
Applied to every allowed crawler (including the catch-allUser-agent: *).
Blocking unauthorized LLM crawlers protects your original content as intellectual property — once a page is ingested into a model's training set, that text is effectively relicensed to anyone who queries the model.
It also reduces server load. AI scraping bots refetch aggressively and ignore the politeness conventions that traditional search crawlers respect, so blocking them at robots.txt cuts wasted bandwidth and origin load.
User-agent + Disallow: / pairs.robots.txt file. Placement at the top ensures the specific agent rules are read before any later wildcard fallback.An AI crawler firewall is a robots.txt-based access control that explicitly disallows known AI training bots and LLM scrapers from reading your site. It generates the precise User-agent and Disallow directives needed to block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and similar agents at the protocol level, before they ever request a page.
No, traditional search crawlers like Googlebot use entirely different user agents than AI training bots. Blocking GPTBot, ClaudeBot, or Google-Extended (the opt-out token for Bard/Gemini training) does not change what Googlebot indexes for classic search results, so your ranking signal is unaffected.
You must disallow 'GPTBot' and 'ChatGPT-User' in your robots.txt file. Use our generator to create the exact syntax to block OpenAI's crawlers.
No. Blocking 'Google-Extended' prevents your site from being used to train Google's AI models, but it does not affect how 'Googlebot' crawls and indexes your site for traditional Google Search.
CCBot is the crawler for Common Crawl, a massive open web archive frequently used as a base dataset to train large language models (LLMs) like GPT and Llama. Blocking it is a common step in preventing AI data scraping.