Should You Let AI Crawlers Read Your Website?

There is a decision sitting on your website that has probably never been made. Right now, AI crawlers — GPTBot, ClaudeBot, PerplexityBot and a growing list of others — are either reading your pages or being turned away at the door.

Whichever is happening, there is a good chance nobody chose it. It is being decided by a default in your CDN, a setting a developer toggled two years ago, or a line in a robots.txt file copied from a template.

We found this on our own site while auditing everything else, which is a common way to find it.

Why this is not the same question as blocking scrapers

The instinct is to treat AI crawlers as a nuisance to be blocked, in the same category as content scrapers and bandwidth thieves. For some businesses that instinct is right. For most service businesses it is expensive.

The reason is that these crawlers now feed two different things. One is model training. The other is retrieval — the assistant looking something up, right now, to answer a question a person is asking it.

Someone asking an assistant "who does bathroom renovations in Brunswick" is a customer at the moment of intent. That is a referral you cannot win if the crawler cannot read your site. It is the same logic that made being absent from Google unacceptable, arriving about twenty years later and much faster.

The asymmetry that decides it for most businesses

Ask what your website is for. There are broadly two answers, and they point in opposite directions.

If your content is the product — you are a publisher, a course seller, a research firm, a stock photography business — then your pages are inventory. Letting a model absorb them so it can paraphrase them for free is giving away the thing you sell. Blocking is a reasonable commercial decision.

If your website is a sales asset — you are a trade, a clinic, a law firm, a restaurant, an agency — then your pages exist to be found by people who might buy. Being unreadable to an assistant is a cost with no matching benefit. Nobody is going to stop hiring plumbers because a model learned what your pricing page says.

Most businesses are in the second category and have been treating themselves as though they were in the first, usually by accident.

How to find out what your site is doing

Two places, and they can disagree — which is the situation worth knowing about.

Start with the file you control directly:

Look for `User-agent: GPTBot`, `ClaudeBot`, `PerplexityBot`, `CCBot`, `Google-Extended` and similar, each followed by an `Allow` or `Disallow`. If none of them appear, your robots.txt expresses no policy at all — which usually means the real policy lives somewhere else.

That somewhere else is your CDN. Cloudflare, for example, has a setting that blocks AI scrapers and crawlers at the edge, before the request ever reaches your server or consults your robots.txt. It can be switched on by a managed rule, by a one-click toggle, or by a default on a plan you signed up for. Fastly, Akamai and others have their equivalents.

This is why the two can disagree, and why checking only robots.txt gives false comfort: a site can politely invite a crawler in the file while the edge turns it away.

A robots.txt that says nothing is not a neutral position. It means your policy is whatever your CDN's default happens to be — set by a vendor, for their reasons, and liable to change without you noticing.

Making the decision, and writing it down

If you decide to allow: confirm the CDN is not blocking, then state the permission explicitly in robots.txt rather than leaving it implied. An explicit `Allow` is documentation for the next person who looks.

If you decide to block: that is a legitimate answer, and it deserves the same treatment. Put the `Disallow` lines in robots.txt so the position is visible, and note why. Otherwise this gets re-litigated every quarter by whoever last read an article about it.

Either way the thing that matters is that it becomes a decision with a reason attached, rather than a default nobody remembers setting.

What allowing does not do

Allowing crawlers does not make you appear in AI answers. It makes it possible. This is the same relationship indexing has to ranking, and it is worth being clear about because the disappointment is predictable otherwise.

What appears to help is the same work that has always helped: pages that answer a specific question directly, in plain language, near the top; content that is genuinely about one thing; and a site that is structurally legible — real HTML, sensible headings, structured data that matches what the page says.

There is no separate discipline here to buy. An assistant reading your page has much the same problem a person skimming it does, and rewards much the same things.