A small site publishing tutorials notices one day that its server traffic is up 30%, mostly from crawlers run by AI companies, programs that read website content automatically. The owner’s first impulse is to block them all, but then worries that if they do, their articles won’t appear in AI search answers and readers won’t find the site.

The dilemma comes from treating every AI crawler as the same thing. They actually come to do three different things, and what they trade with a site is completely different. So instead of deciding “block AI or not,” first look at what each crawler came to do, then set different default rules for each purpose.

Cloudflare split AI crawlers into three purposes

On July 1, 2026, the internet services company Cloudflare replaced its single “block AI bots” switch with three categories that sites can set on their own:

  • Search: collecting and indexing a site’s content to answer questions later. It’s closest to a traditional search engine and, in principle, sends readers back to the site.
  • Agent: acting in real time for a person, such as a chatbot reading a page mid-conversation or operating a browser for a user. There’s usually a person waiting for an answer on the other end.
  • Training: taking content to train or fine-tune a model. Cloudflare describes this data as permanently absorbed into the model.

Cloudflare also announced that from September 15, 2026, newly added domains would block agent and training use by default on pages that show ads, while search stays allowed by default. The reasoning: pages with ads want human attention, and search is the most likely to bring people back. TechCrunch also reported that Cloudflare’s per-crawl payment mechanism would shift to charging by the value content generates.

This classification won’t suit every site, but it offers a more useful question than “block or don’t”: what is this crawler here to do, and what does the site get in return?

Three purposes, three trades

Search trades for exposure. The site offers indexable content and expects readers to click through in return. The thing to track is whether the traffic AI search sends back matches how much is crawled. If a lot is crawled and almost no readers come back, the trade isn’t working, and you can switch to monitoring or rate limiting.

Agents bring cost and permission questions. AI reading pages live for users may hit prices, specs, or site search heavily in a short time. What needs managing is how often it reads, and whether it can reach logged-in content or submit forms.

Training is a licensing question. Once content is absorbed into a model, it may not bring back any readers. This is closer to “do I license my work to others” than a traffic question. The right default is to require a license or payment, not to quietly allow it.

Advertisement

Protect your most valuable pages first

There’s no need to write a full rulebook at the start. List the pages that are most valuable or most easily pulled in bulk: member content, paid courses, pricing, original research, and entry points like site search and filters that programs can query repeatedly. Even under purposes you allow, these pages deserve extra limits.

Then put the rules where they can actually be enforced. Cloudflare itself explains that robots.txt, the file at a site’s root that tells crawlers where they may go, and its new content-use field only express a site’s preferences; on their own, they don’t block any access. Real limits come from a firewall or bot management tool that throttles or blocks based on traffic patterns. If an agent needs to query lots of data, an API with authentication and rate limits is easier to manage than letting it crawl pages. Training use and paid content ultimately have to be handled through contracts and licenses, with technical rules as support.

Classifying only works if you can tell the purpose

Classifying by purpose assumes you can tell what a crawler is for. But a crawler’s self-description can be faked, and some large companies’ crawlers do search and training at once. With crawlers of unknown identity or mixed purpose, all you can do is judge by behavior: which pages it fetched, how often, and whether it went where it shouldn’t.

Also, rules aren’t set-and-forget. Traffic from AI search and crawler behavior both change. A practical approach is to look back once a month at the request volume for each purpose, the readers sent back, server costs, and how high-value pages are being accessed, then decide whether to adjust.

As a next step, write a one-page policy: the default decision for each of the three purposes, the pages needing extra protection, which tool enforces each rule, and who reviews it monthly. If your site isn’t yet ready for AI to read it properly, see Does Your Site Need an AI Entry Point? Ask Whether the Job Is Read, Look Up, or Act. If what you care about is what each page gets when AI search quotes it, see Before Opting Out of Google’s AI Search, Ask Whether Readers Still Have a Reason to Visit After the Summary.

AI handoff card

Ask AI to draft an AI crawler access policy This prompt asks about your site’s content and current situation, sets default decisions for search, agent, and training use, and marks the pages that need extra protection and how to enforce it.

I want to draft an AI crawler access policy for my website.
The method comes from this article: https://boosterminiclass.com/en/posts/ai-crawler-access-policy-checklist/
Its view: don't decide with a single "block AI or not" switch. AI crawlers serve three purposes: search (a trade for exposure), live agent reading (bringing cost and permission questions), and training (a licensing question), and each deserves a different default. robots.txt only expresses preferences; real limits come from firewalls, API rate limits, or contracts.

Ask me 3 questions first:
1. What kind of content does the site have, and which parts are public versus member-only or paid?
2. Which AI crawlers can I see fetching today, and roughly how many requests does each make (say "unknown" if I don't know)?
3. Who decides on content licensing, and who manages the site's technical settings?

Based on my answers, give me:
- A default decision for each of the three purposes (only: allow / monitor or rate-limit / require a license or payment / block), with reasons;
- A list of high-value pages that need extra protection;
- For each rule, whether robots.txt, a firewall or bot management tool, API rate limits, or a contract should enforce it;
- The numbers to check each month.
Mark anything without enough information as "to confirm" rather than guessing. Finally, remind me that the policy is the site owner's decision.

The policy is only a starting point. What content you’re willing to let AI use is ultimately the site owner’s call.

Advertisement

Share

Share this mini class

If this lesson helps untangle a work bottleneck, share it with someone deciding how to use AI.

References