Free SEO tool · runs in your browser
Robots.txt Generator
Build a clean robots.txt file for your website: choose a starting point, add the paths to block, list your sitemap and decide whether AI training crawlers may read your content.
Your rules
robots.txt
Choose your options to generate the file.
How this tool works
- Every file starts with rules for all crawlers (
User-agent: *). - In custom mode, each path you list becomes a
Disallowline, and exceptions becomeAllowlines. Full URLs are trimmed to their path, and paths without a leading slash get one. - If you choose to block AI training crawlers, a separate group is added for each documented training crawler.
- Your sitemap URLs are added at the end as
Sitemaplines.
Example
User-agent: * Disallow: /cart/ Disallow: /search/ # OpenAI AI training crawler User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Common mistakes
- Leaving
Disallow: /from a staging site on a live site, which stops search engines from crawling everything. - Blocking CSS or JavaScript files that pages need to render.
- Using robots.txt to hide private pages. The file is public, so it can point people straight to them.
Sources: Introduction to robots.txt (Google Search Central), Overview of OpenAI crawlers, Anthropic crawlers, Google common crawlers (Google-Extended) and Common Crawl CCBot.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, for example https://example.com/robots.txt. Each subdomain needs its own file. Many platforms (such as Shopify and some website builders) generate robots.txt for you and only allow limited edits, so check your platform’s documentation before replacing it.
Does robots.txt keep a page out of Google?
No. robots.txt controls crawling, not indexing. Google says a blocked URL can still be indexed if other pages link to it. To keep a page out of search results, allow crawling and use a noindex meta tag or HTTP header, or protect the page with a password.
What does “block AI training crawlers” add?
Rules that disallow GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google) and CCBot (Common Crawl). These are the tokens each company documents for collecting content that may be used to train AI models. Search crawlers such as Googlebot, OAI-SearchBot and Claude-SearchBot are not blocked, so your pages can still appear in search. Compliance with robots.txt is voluntary for crawlers.
Is my data stored?
No. The file is generated in your browser; nothing you type is sent to a server.