User-agent: * Disallow: /admin Disallow: /private Sitemap: https://www.example.com/sitemap.xml
Free Online Robots.txt Generator
Build a custom robots.txt with per-crawler Allow/Disallow rules and sitemap references, with live warnings for common mistakes. It helps site owners write crawler directives for a domain while keeping the rules readable and easy to review before deployment.
What robots.txt controls
robots.txt is a plain-text file at the root of a domain that gives web crawlers instructions about which parts of a site they're allowed to request. It's the first thing well-behaved crawlers check before crawling anything else, using simple User-agent, Allow, and Disallow directives, plus optional Sitemap lines pointing crawlers at your XML sitemap so they can discover pages more efficiently.
How to use this generator
Add one or more user-agent groups — use * to target every crawler, or a specific bot name like Googlebot for crawler-specific rules — then list the paths each group should or shouldn't crawl. Add your sitemap URL so crawlers can find it immediately, and watch the warnings panel for common mistakes like a relative sitemap URL or a rule that accidentally blocks your entire site.
Common use cases
- Blocking crawlers from admin panels, staging paths, or internal search-result pages
- Pointing all crawlers at your XML sitemap to speed up discovery of new pages
- Giving one crawler (like an aggressive scraper bot) different, more restrictive rules than the rest
- Double-checking an existing robots.txt for a mistake that's accidentally blocking the whole site
Everything is generated locally in your browser — no part of your configuration is sent to a server.
How to use Robots.txt Generator for related tasks
Choose user-agent rules, add allow or disallow paths, generate the file, and remember that robots.txt is guidance rather than access control.
Related tools: Meta Tag / Open Graph Preview, Slug Generator.
Frequently asked questions
Where does robots.txt need to be placed?
At the root of your domain, e.g. https://example.com/robots.txt — search engine crawlers only look for it at that exact location, not in a subdirectory.
Does robots.txt actually prevent a page from being indexed?
Not reliably on its own — it asks well-behaved crawlers not to fetch matching URLs, but a URL can still appear in search results (without its content) if other pages link to it. To reliably keep a page out of search results, use a noindex meta tag or header on that page instead, which requires the page to be crawlable in the first place.
Can I have different rules for different crawlers?
Yes — add multiple groups, each with its own User-agent line (like Googlebot or Bingbot, or * for all crawlers). Crawlers use the most specific matching group that names them; everything else falls back to the * group.
Is this validated against a live site?
No — this only checks the syntax and structure of what you enter (like flagging a relative sitemap URL or a group that blocks everything). It doesn't know your site's actual URL structure, so double-check the paths match what you intend to allow or block.
Can robots.txt block private data?
No. Crawlers may ignore it and visitors can still request a URL; protect private resources with authentication and server authorization.