Robots Blocking Audit
Enter a URL: the tool reads the site's robots.txt, applies RFC 9309 for each crawler and also checks the page's noindex directives.
Privacy and cost
This tool runs a live check on our servers. Free accounts get 3 live checks per day in total across all such tools. We keep your report in your account so you can delete it any time.
How it works
- We fetch /robots.txt and the page (two requests).
- The file is interpreted per RFC 9309: most specific group, longest rule, Allow wins ties, wildcards * and $.
- The result shows, per crawler, whether crawling is allowed and which line decides, then whether the page carries noindex (meta or X-Robots-Tag).
Example
A robots.txt containing "Disallow: /private/" blocks /private/page for every crawler: the tool shows the responsible line. If the page also has noindex, it warns that Google cannot read it.
What the result means
Blocked by robots.txt means the crawler does not visit the URL, not that it leaves the index: a blocked but linked URL can be indexed without content. To remove a page, keep it crawlable and use noindex. A robots.txt answering 5xx is treated as a full block.
Common problems this tool helps with
- A strategic section blocked by an over-broad Disallow.
- A noindex on a page also blocked by robots.txt: Google never sees it.
- An unreachable robots.txt (5xx) that blocks the whole site.
- AI crawlers blocked or allowed by mistake.
Frequently asked questions
Which standard is applied?
RFC 9309 (Robots Exclusion Protocol): longest match, Allow wins ties, path matching is case-sensitive.
Does robots.txt prevent indexing?
No, it prevents crawling. To prevent indexing use noindex on a crawlable page.
Can a crawler ignore robots.txt?
Yes, it is a request for good behaviour, not access control. Never protect sensitive content with robots.txt.
Related tools
- Single Page SEO AuditEnter a page address: the tool downloads it, runs about fifty technical checks and returns a score per category with the issues to fix first.Free account · 3 live checks/day
- AI Crawler Access CheckerEnter a site: the tool reads its robots.txt and shows, for each known AI crawler, whether it may crawl, which line decides, and what the crawler is for: training, search or a user action.Free account · 3 live checks/day
- Redirect Chain CheckerPaste a URL to see every redirect it goes through before the final page, with the status code and timing of each hop.Free account · 3 live checks/day
- GEO Readiness AuditEnter a URL: the tool checks the technical and editorial signals that make a page accessible and understandable to AI search assistants, and scores each of them.Free account · 3 live checks/day
- Hreflang Live AuditEnter a URL: the tool reads its hreflang tags (HTML and HTTP Link header), then visits up to eight alternates to confirm they respond and link back to your page.Free account · 3 live checks/day