Skip to content
Rankora
EN
Sign in
All toolsTechnical SEOOn-Page SEOKeyword ResearchSERP & RankingsBacklinksSchema & JSON-LDSitemaps & RobotsPerformanceInternational SEOGEO / AI SearchGoogle Search ConsoleASO / App StorePricing
Free · runs in your browser

Robots.txt Generator

Pick a preset, add your sitemap and rules for specific crawlers. The robots.txt is generated and re-read by an RFC 9309 parser before you download it.

Privacy and cost

This tool runs in your browser. What you type or paste is not sent to our servers and it costs nothing to use.

How it works

  1. Choose a starting preset: allow all, block all, WordPress, e-commerce or block AI training crawlers.
  2. Enter your sitemap URL and, if needed, add a rule group for one crawler (Bingbot, Googlebot-Image, GPTBot…).
  3. Copy or download the file and upload it to the root of your domain: https://your-domain/robots.txt.

Example

The WordPress preset with the sitemap https://example.com/sitemap.xml outputs 8 directive lines: Allow /wp-admin/admin-ajax.php, five Disallow rules (/wp-admin/, /wp-login.php, /?s=, /search/, /*?replytocom=) and the Sitemap line. The built-in parser reports no error.

What the result means

robots.txt tells crawlers which URLs they may crawl. It is not a security tool and not a way to deindex pages: a blocked URL can still be indexed if other sites link to it. To keep a page out of results, leave it crawlable and use noindex. Only crawlers that follow the protocol obey it.

Common problems this tool helps with

  • A forgotten "Disallow: /" after a launch blocks the whole site for search engines.
  • A misspelled directive or a path without a leading slash is silently ignored.
  • Blocking CSS and JavaScript folders stops Google from rendering your pages properly.
  • Trying to block AI crawlers without knowing their exact user-agent names.

Frequently asked questions

Where does robots.txt go?

At the root of the host, for example https://example.com/robots.txt. Each subdomain and each protocol has its own file, and it does not apply to subfolders.

Does blocking GPTBot or ClaudeBot stop AI training?

It is a request to crawlers that respect robots.txt. It does not affect crawlers that ignore it or data already collected. The tool's AI preset blocks training crawlers only and leaves Google and Bing untouched.

Can robots.txt remove a page from Google?

No. It controls crawling, not indexing. To remove a page, keep it crawlable and add a meta robots noindex tag or an X-Robots-Tag header.

Related tools