robots.txt & sitemap.xml Builder

Generate robots.txt from path rules and turn a page list into sitemap.xml with lastmod, priority and frequency

File

User-agentSeparate crawlers with commas or new lines; * means everyoneDisallowOne path per line; a leading slash is added for youAllow

Crawl delay (seconds)Host

Sitemap URLs

robots.txt

User-agent: *
Disallow: /admin
Disallow: /api/private
Disallow: /search
Allow: /api/public

Sitemap: https://example.com/sitemap.xml

robots.txt governs crawling, not indexing: to keep a page out of results you still need meta name="robots" content="noindex". Sitemap locs must be absolute, and one sitemap caps at 50,000 URLs and 50MB uncompressed. Both files belong at the site root.