robots.txt Builder

Build a robots.txt for crawlers — User-agent groups, Allow/Disallow paths, sitemap.

Groups

User-agent

Crawl-delay

Allow paths (one per line)

Disallow paths (one per line)

Sitemap URL (optional)

robots.txt

User-agent: *
Allow: /
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml