How to Block AI Crawlers on Magento 2

Googlebot and Bingbot build the search indexes that send you shoppers. They are not the problem this guide is about, and they must stay allowed. If you block them, they stop crawling your store and your pages fall out of normal search results. Google notes that a page it is not allowed to crawl can still show up, but only as a link without a description.

AI crawlers do three different jobs, and the difference matters when you decide what to block:

• Training: collecting content that may be used to train AI models, such as OpenAI's GPTBot, Anthropic's ClaudeBot and Meta's meta-externalagent. Google and Apple use robots.txt tokens rather than separate crawlers for this: Google-Extended and Applebot-Extended.
• AI search and answers: finding pages to show or cite in AI search, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and meta-webindexer. Block these and your store can appear less in those answers.
• User-initiated fetches: visiting a page because a user asked the assistant to, such as ChatGPT-User, Claude-User, Perplexity-User and meta-externalfetcher. Several of their owners say robots.txt may not apply to these.

Every name below comes from the owner's own documentation. The last column is what the owner says about robots.txt. We have left out crawlers whose owners do not publish clear documentation.

There is no single right answer, so it is worth being clear about the trade-off before you change anything.

• Block training only. Your content stays out of future model training, while AI search and answer tools can still find and cite your products. Google Search and Bing are unaffected.
• Block every AI crawler. The strongest opt-out, but your store can appear less in AI answers, which some shoppers now use to find products. OpenAI, for example, says sites that block OAI-SearchBot will not be shown in ChatGPT search answers.
• Slow them down instead. If the real issue is load rather than principle, rate limiting at the server keeps the bots but caps how hard they can hit you. Anthropic also supports the non-standard Crawl-delay line for its crawlers; Google's crawlers ignore it.

In the Magento admin, go to Content > Design > Configuration, choose the scope, open Search Engine Robots and use the Edit custom instruction of robots.txt file field. Keep Adobe's default instructions and add your AI crawler groups underneath. Then open /robots.txt on your store and check what it actually serves: if a robots.txt file already sits on the server, that may be the one visitors get.

Option 1, block training only:

# Block AI training only. Search and AI answer crawlers are still allowed.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Optional: Common Crawl's public archive
User-agent: CCBot
Disallow: /

Option 2, block every AI crawler: use the rules above plus these.

# Add these to the training-only rules to block AI search and answer bots too.
User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: meta-webindexer
Disallow: /

User-agent: meta-externalfetcher
Disallow: /

One rule of robots.txt catches people out. Google and Bing both say a crawler follows only the most specific group that names it and ignores the rest, so a group written for one bot does not inherit your User-agent: * rules. That is harmless for the Disallow: / groups above. It matters if you ever write a group for Googlebot or Bingbot, because you would then need to repeat your default rules inside it.

Google's own documentation is plain about this: robots.txt cannot enforce crawler behaviour, and it is up to each crawler to obey it. The major AI companies say their crawlers do, but the user-initiated fetchers above may not, and some bots fake their user agent. Common Crawl, for example, warns about crawlers falsely identifying themselves as CCBot.

The backstop is the web server. Your host or developer can refuse requests from named user agents, or rate limit them, before they reach Magento. On nginx that can be as simple as this, as general advice to adapt rather than paste:

# Example only: refuse named AI crawlers at the web server.
# Never add Googlebot or bingbot to this list.
map $http_user_agent $block_ai_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|meta-externalagent|CCBot) 1;
}

server {
    # ...your existing Magento server block...
    if ($block_ai_bot) { return 403; }
}

Blocking by user agent only works on bots that tell the truth about who they are. For the rest, blocking or rate limiting by IP address is the next step, and several owners publish their IP ranges so you can check a claim before acting on it: OpenAI, Anthropic and Perplexity publish lists of the addresses their crawlers use, Common Crawl runs CCBot from dedicated ranges with reverse DNS, and Google documents how to confirm that a visitor claiming to be Googlebot really is. Check before you block anything that says it is Googlebot.

This part is general advice rather than anything specific to one bot. A Magento storefront with layered navigation can produce a very large number of filter URL combinations, and a crawler that follows them all keeps asking for pages that have never been requested before. Varnish serves repeat requests for the same page from memory, but each new combination has to be built by PHP and the database the first time, and on-site search results work the same way. That is why a busy crawler can cost a Magento store more than its share of visits suggests.

Canonical tags and a considered robots.txt help, but be careful about disallowing filter URLs for every crawler: those rules apply to Google too, so decide what you want indexed first. If you are sizing a server, measure bot traffic from the server side, because analytics tools usually do not count it.

Which AI crawlers may read your store is your decision, and robots.txt is where you make it. Hosting helps with the bots that do not listen. On EveryHost plans, Sentinel security monitoring reads the web server logs and can automatically block abusive IP addresses at the server firewall, and our engineers review every block. The same engineers can help you add user agent or rate limiting rules on the server when a crawler ignores robots.txt, without touching Googlebot or Bingbot.

Aggressive crawlers are not the only automated traffic worth watching. Bots trying stolen card numbers at your checkout are a different problem with different fixes, covered in our guide to stopping card testing on Magento 2. If you would rather have a UK team who work only on Magento looking after all of this, our managed Magento VPS hosting includes Sentinel on every plan.