Sitemaps & Robots
Control what search engines crawl: XML sitemap, robots.txt and meta robots directives.
Tools in Sitemaps & Robots
- Robots.txt GeneratorPick a preset, add your sitemap and rules for specific crawlers. The robots.txt is generated and re-read by an RFC 9309 parser before you download it.Free · runs in your browser
- Robots.txt CheckerPaste your file or enter a domain. The tool applies the RFC 9309 rules, flags errors and tells you whether a given crawler may fetch a URL.Free · no account needed
- Meta Robots GeneratorTick the directives you need: the tool outputs the meta robots tag, the X-Robots-Tag header and the matching Apache and Nginx snippets.Free · runs in your browser
- Meta Robots & X-Robots-Tag CheckerEnter a URL, or paste the HTML and headers: the tool reads every robots directive, spots contradictions and explains what each one does.Free · no account needed
- XML Sitemap GeneratorPaste your URLs, one per line: the tool outputs a valid, correctly escaped sitemap.xml and splits it with an index if you go over the limits.Free · runs in your browser
- XML Sitemap ValidatorEnter the sitemap address or paste its XML: the tool checks the structure, the limits and every value, and works out how many files you need.Free · no account needed
- Sitemap URL ExtractorEnter a sitemap or just a domain: the tool lists every URL and its lastmod, follows a sitemap index one level deep and exports the result.Free · no account needed
- Robots Blocking AuditEnter a URL: the tool reads the site's robots.txt, applies RFC 9309 for each crawler and also checks the page's noindex directives.Free account · 3 live checks/day