Free account · 3 live checks/day

AI Crawler Access Checker

Enter a site: the tool reads its robots.txt and shows, for each known AI crawler, whether it may crawl, which line decides, and what the crawler is for: training, search or a user action.

Privacy and cost

This tool runs a live check on our servers. Free accounts get 3 live checks per day in total across all such tools. We keep your report in your account so you can delete it any time.

How it works

A single request fetches the site's /robots.txt file.
Each crawler is evaluated per RFC 9309 (most specific group, longest rule, Allow wins ties).
The table shows the operator, purpose, allowed or blocked state and the responsible rule, with a summary per purpose.

Example

With "User-agent: GPTBot / Disallow: /", GPTBot (training) shows as blocked while OAI-SearchBot (search) stays allowed as long as no rule targets it.

What the result means

Blocking a training crawler stops collection for model training; blocking a search crawler can reduce presence in AI answers. Robots.txt remains a request: some agents acting on a user's behalf (for example ChatGPT-User, Perplexity-User) may not follow it, according to their operators.

Common problems this tool helps with

A Disallow: / for all bots that also blocks AI search crawlers.
Blocking GPTBot thinking it also blocks ChatGPT search (OAI-SearchBot is separate).
Obsolete tokens such as anthropic-ai still in place.
An unreachable robots.txt treated as a full block.

Frequently asked questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects data for training OpenAI's models; OAI-SearchBot powers search. They are set separately in robots.txt.

Is Google-Extended a separate crawler?

It is a control token: it has no HTTP user-agent of its own and governs the use of your content by certain Google AI products. It does not affect Search indexing.

Is the list complete?

No. It covers crawlers documented by their operators as of the date shown in the result, and changes regularly.