Robots.txt Checker
Paste your file or enter a domain. The tool applies the RFC 9309 rules, flags errors and tells you whether a given crawler may fetch a URL.
Privacy and cost
This tool requests the public URL you enter from our servers. It is free, rate-limited to prevent abuse, and we do not store the URL or the page content.
How it works
- Enter a domain to fetch its robots.txt, or paste the file content.
- Read the syntax errors and warnings with line numbers, plus the parsed groups.
- Test URLs for a crawler (Googlebot, Bingbot, GPTBot…): the tool shows the deciding rule, the longest match winning.
Example
On a sample file with two groups, the tool flags "Noindex" (line 6, unsupported by Google) and a Crawl-delay (line 5, ignored by Google). For Googlebot, /private/report.html is blocked by "Disallow: /private/" (line 2), /private/press-kit.pdf is allowed by "Allow: /private/press-kit.pdf" (line 3) and /shoes?sort=price is blocked by "Disallow: /*?sort=" (line 4).
What the result means
A URL is allowed unless a Disallow rule matches it. When several rules match, the most specific (longest) one wins and Allow beats Disallow on a tie. Crawlers use the group of the most specific user-agent, otherwise the "*" group. Crawlers ignore everything beyond the first 500 KiB.
Common problems this tool helps with
- A Disallow rule that is too broad and silently blocks important pages.
- Ignored directives (Noindex, typos, Crawl-delay for Google) that people believe are active.
- A Googlebot group that does not repeat the "*" group rules: crawlers apply only one group.
- An unreachable (5xx) or oversized robots.txt whose behaviour is misunderstood.
Frequently asked questions
Why does a blocked URL still show up in Google?
robots.txt blocks crawling, not indexing. Google can index the bare URL if other pages link to it. To exclude it, leave it crawlable and use noindex.
Does the tool fetch my file from my browser?
No: fetching goes through our server, which contacts the site with a protected request, because a browser cannot read another domain's robots.txt. The analysis of the content then runs in your browser.
What is the maximum size of a robots.txt?
Google reads the first 500 KiB (the RFC 9309 minimum). Rules beyond that are ignored, and the tool warns you.
Related tools
- Robots.txt GeneratorPick a preset, add your sitemap and rules for specific crawlers. The robots.txt is generated and re-read by an RFC 9309 parser before you download it.Free · runs in your browser
- Meta Robots & X-Robots-Tag CheckerEnter a URL, or paste the HTML and headers: the tool reads every robots directive, spots contradictions and explains what each one does.Free · no account needed
- XML Sitemap ValidatorEnter the sitemap address or paste its XML: the tool checks the structure, the limits and every value, and works out how many files you need.Free · no account needed
- XML Sitemap GeneratorPaste your URLs, one per line: the tool outputs a valid, correctly escaped sitemap.xml and splits it with an index if you go over the limits.Free · runs in your browser
- Robots Blocking AuditEnter a URL: the tool reads the site's robots.txt, applies RFC 9309 for each crawler and also checks the page's noindex directives.Free account · 3 live checks/day
- AI Crawler Access CheckerEnter a site: the tool reads its robots.txt and shows, for each known AI crawler, whether it may crawl, which line decides, and what the crawler is for: training, search or a user action.Free account · 3 live checks/day