Your robots.txt Is Probably Blocking AI Search, and Nobody Noticed

Sometime in 2024 a wave of advice went around: block the AI crawlers. Add GPTBot and a few others to robots.txt and stop the models from training on your content. Plugins added a checkbox. Hosts added a toggle. A lot of small business sites got it switched on without anyone thinking about it again.

Here is the problem. The bot that trains a model and the bot that fetches your page to answer a customer's question right now are different bots, and most of those blanket rules blocked both.

Retrieval bots versus training bots

There are two jobs an AI company's crawler can be doing on your site.

Training. Reading your pages to fold them into the next version of the model. Blocking this is a legitimate content decision. It does not affect whether the model can recommend you today.

Retrieval. Fetching your page in real time because a user just asked "who does tree removal in Cleveland, Tennessee" and the engine wants a current answer. Block this and the engine cannot read you. You do not appear in the answer, and the customer never learns you exist.

The names matter. As of this writing the retrieval side includes OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, Applebot-Extended for Apple Intelligence, plus Googlebot and Bingbot, which now also feed AI answers. The training side includes GPTBot, Google-Extended, ClaudeBot, anthropic-ai, Amazonbot, Bytespider, and CCBot.

Our monitor checks fifteen of them and classifies each one. A blocked training bot is a warning, and usually intentional. A blocked retrieval bot is critical, because it removes you from the answers your customers are reading.

The case that made us build the monitor

The first time we ran this check across our own fleet, a moving company's robots.txt was blocking Applebot-Extended. Nobody had done it on purpose. A plugin had shipped a "block AI" list, the list included Apple's bot, and from that day every iPhone user asking Siri or Apple Intelligence about movers in that area got answers that could not include this business.

The site ranked fine on Google. The owner had no complaints. There was simply a channel where the business did not exist, and no dashboard anywhere showed it.

Six training bots were blocked on the same site too. We left those alone and flagged them as intentional. The one retrieval bot became an urgent task, and it was fixed in a minute.

What robots.txt cannot tell you

Robots.txt is a request, not a wall. Two things break the picture.

Bots that ignore it. If your server logs show visits from a bot you have disallowed, the bot is not respecting the rule, or a firewall is letting it through anyway. We correlate the robots.txt verdict with real visit telemetry per bot, and the contradiction "blocked in robots, still visiting" is the most useful alert the monitor produces.

Firewalls that block what robots.txt allows. The opposite failure. Your robots.txt says allow, and a web application firewall or a hosting rule returns a 403 to anything with "bot" in its user agent. We probe this by requesting the homepage with the real user agent of each AI bot and comparing it to a normal browser request from the same address.

One honest caveat on that probe: behind Cloudflare it proves nothing. Cloudflare returns a 403 to anyone impersonating an AI bot from an unverified address, in every zone we tested, which is the right behavior for Cloudflare and means our probe cannot see the real policy. For those sites the answer lives in the zone's bot settings, not in a request from outside.

Check your own site in five minutes

1. Open yourdomain.com/robots.txt.
2. Search for each retrieval bot by name: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Applebot-Extended, Googlebot, Bingbot. Any Disallow that applies to them is a problem.
3. Look for a blanket User-agent: * rule with Disallow: / and no allow rules above it for the bots you want. That blocks everyone.
4. If you use a "block AI crawlers" plugin or host toggle, open its list and remove the retrieval bots. Keep the training bots blocked if that is your policy.
5. Ask ChatGPT and Perplexity a question your customers ask, in your city, and see whether you appear. If you do not, and the bots can reach you, the problem is content, not access. That is a different fix.

Why this belongs in the platform, not in a checklist

A checklist gets done once. Plugins update, hosts change defaults, and someone on the team turns the toggle back on during a security scare. The only thing that catches the regression is a check that runs on a schedule, classifies the result by what it actually costs the business, and opens a task when it matters.

That is what our Bot Access monitor does for every site we manage, and it is why this site runs the same check on itself.