How to Block AI Crawlers on Magento 2

Googlebot and Bingbot build the search indexes that send you shoppers. They are not the problem this guide is about, and they must stay allowed. If you block them, they stop crawling your store and your pages fall out of normal search results. Google notes that a page it is not allowed to crawl can still show up, but only as a link without a description.

AI crawlers do three different jobs, and the difference matters when you decide what to block:

• Training: collecting content that may be used to train AI models, such as OpenAI's GPTBot, Anthropic's ClaudeBot and Meta's meta-externalagent. Google and Apple use robots.txt tokens rather than separate crawlers for this: Google-Extended and Applebot-Extended.
• AI search and answers: finding pages to show or cite in AI search, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and meta-webindexer. Block these and your store can appear less in those answers.
• User-initiated fetches: visiting a page because a user asked the assistant to, such as ChatGPT-User, Claude-User, Perplexity-User and meta-externalfetcher. Several of their owners say robots.txt may not apply to these.

Every name below comes from the owner's own documentation. The last column is what the owner says about robots.txt. We have left out crawlers whose owners do not publish clear documentation.

There is no single right answer, so it is worth being clear about the trade-off before you change anything.

• Block training only. Your content stays out of future model training, while AI search and answer tools can still find and cite your products. Google Search and Bing are unaffected.
• Block every AI crawler. The strongest opt-out, but your store can appear less in AI answers, which some shoppers now use to find products. OpenAI, for example, says sites that block OAI-SearchBot will not be shown in ChatGPT search answers.
• Slow them down instead. If the real issue is load rather than principle, rate limiting at the server keeps the bots but caps how hard they can hit you. Anthropic also supports the non-standard Crawl-delay line for its crawlers; Google's crawlers ignore it.

In the Magento admin, go to Content > Design > Configuration, choose the scope, open Search Engine Robots and use the Edit custom instruction of robots.txt file field. Keep Adobe's default instructions and add your AI crawler groups underneath. Then open /robots.txt on your store and check what it actually serves: if a robots.txt file already sits on the server, that may be the one visitors get.

Option 1, block training only:

# Block AI training only. Search and AI answer crawlers are still allowed.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Optional: Common Crawl's public archive
User-agent: CCBot
Disallow: /

Option 2, block every AI crawler: use the rules above plus these.

# Add these to the training-only rules to block AI search and answer bots too.
User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: meta-webindexer
Disallow: /

User-agent: meta-externalfetcher
Disallow: /

One rule of robots.txt catches people out. Google and Bing both say a crawler follows only the most specific group that names it and ignores the rest, so a group written for one bot does not inherit your User-agent: * rules. That is harmless for the Disallow: / groups above. It matters if you ever write a group for Googlebot or Bingbot, because you would then need to repeat your default rules inside it.

Google's own documentation is plain about this: robots.txt cannot enforce crawler behaviour, and it is up to each crawler to obey it. The major AI companies say their crawlers do, but the user-initiated fetchers above may not, and some bots fake their user agent. Common Crawl, for example, warns about crawlers falsely identifying themselves as CCBot.

The backstop is the web server. Your host or developer can refuse requests from named user agents, or rate limit them, before they reach Magento. On nginx it can look like this, as general advice to adapt rather than paste:

# Example only: refuse named AI crawlers at the web server.
# Never add Googlebot or bingbot to this list.
map $http_user_agent $ai_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|meta-externalagent|CCBot|Bytespider) 1;
}
# Still let those bots read /robots.txt, so they can see your rules.
map "$ai_bot:$uri" $block_ai_bot {
    default 0;
    "1:/robots.txt" 0;
    "~^1:" 1;
}

server {
    # ...your existing Magento server block...
    if ($block_ai_bot) { return 403; }
}

The second map matters: it lets the blocked bots still fetch /robots.txt, so they can read your rules, while every other request from them gets a 403.

If you would rather slow crawlers down than refuse them, rate limit them instead. This version gives each named crawler one shared counter, allows a burst of 20 requests, and answers anything over the limit with 429 Too Many Requests. Requests with an empty key are not counted, according to the nginx limit_req documentation, which is what keeps shoppers and search engines out of it:

# Example only. In the http { } block: one counter per named crawler.
# Everyone else gets an empty key, which limit_req does not count, so
# shoppers, Googlebot and bingbot are never limited by this rule.
map $http_user_agent $ai_crawler_key {
    default "";
    ~*(GPTBot|ClaudeBot|OAI-SearchBot|PerplexityBot|meta-externalagent|CCBot|Bytespider) $1;
}
limit_req_zone $ai_crawler_key zone=ai_crawlers:10m rate=60r/m;

# In your Magento server { } block:
limit_req zone=ai_crawlers burst=20 nodelay;
limit_req_status 429;

Blocking by user agent only works on bots that tell the truth about who they are. For the rest, blocking or rate limiting by IP address is the next step, and several owners publish their IP ranges so you can check a claim before acting on it: OpenAI, Anthropic and Perplexity publish lists of the addresses their crawlers use, Common Crawl runs CCBot from dedicated ranges with reverse DNS, and Google documents how to confirm that a visitor claiming to be Googlebot really is. Check before you block anything that says it is Googlebot.

Google's method is a reverse DNS lookup on the IP address from your logs, then a forward lookup on the name it returns. The name must end in googlebot.com, google.com or googleusercontent.com, and the forward lookup must give back the original IP address. With the host command:

# 1. Reverse lookup on the IP address from your logs
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# 2. Forward lookup on the name it returned: it must give back the same IP
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

This part is general advice rather than anything specific to one bot. A Magento storefront with layered navigation can produce a very large number of filter URL combinations, and a crawler that follows them all keeps asking for pages that have never been requested before. Varnish serves repeat requests for the same page from memory, but each new combination has to be built by PHP and the database the first time, and on-site search results work the same way. That is why a busy crawler can cost a Magento store more than its share of visits suggests.

Canonical tags and a considered robots.txt help, but be careful about disallowing filter URLs for every crawler: those rules apply to Google too, so decide what you want indexed first. If you are sizing a server, measure bot traffic from the server side, because analytics tools usually do not count it.

A quick way to see which crawlers are busiest is to count requests by user agent in the web server's access log. With nginx's default log format, the user agent is the sixth field when a line is split on double quotes:

# Top 20 user agents in an nginx access log (default combined format).
# Adjust the path to your store's access log.
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

AI Crawl Control is available on every Cloudflare plan and lets you allow or block individual AI crawlers from the dashboard. On the Free plan it recognises them by user agent. Leave Googlebot and Bingbot alone there too: blocking search engine crawlers can hurt your search visibility.

For the layered navigation load described above, a rate limiting rule can cap requests to filtered category pages. By default Magento's filters live in the query string, and matching on the query string needs the Pro plan or above: Free plan rules can match on the path and on whether a request comes from a verified bot. Whichever plan you are on, exclude verified bots from the rule so Googlebot and Bingbot are never caught by it, and set the limit well above what a shopper clicking through filters would reach.

Which AI crawlers may read your store is your decision, and robots.txt is where you make it. Hosting helps with the bots that do not listen. On EveryHost plans, Sentinel security monitoring reads the web server logs and can automatically block abusive IP addresses at the server firewall, and our engineers review every block. The same engineers can help you add user agent or rate limiting rules on the server when a crawler ignores robots.txt, without touching Googlebot or Bingbot.

Aggressive crawlers are not the only automated traffic worth watching. Bots trying stolen card numbers at your checkout are a different problem with different fixes, covered in our guide to stopping card testing on Magento 2. If you would rather have a UK team who work only on Magento looking after all of this, our managed Magento VPS hosting includes Sentinel on every plan.