Robots.txt File Generator
Free robots.txt generator: add disallow and allow paths, crawl-delay, your sitemap, WordPress or Shopify presets, and one-click blocking of AI crawlers like GPTBot and ClaudeBot.
Tool Configuration Options
What is a Robots.txt File Generator?
A robots.txt generator is a webmaster tool that creates a standardized robots.txt configuration file to guide search engine crawlers (such as Googlebot, Bingbot, and DuckDuckBot) on how to index and crawl your website. It specifies which folders, URLs, and file paths search bots are permitted or forbidden to access, preventing server overload and keeping duplicate or staging pages out of search engine indexes. Placed at the root of your domain, it serves as the foundational protocol for search crawler management.
Execution Privacy:Toolbrix processes this utility using🔒 Processed locally — text generation runs client-side.
How to use this Robots.txt File Generator
Pick a platform preset (WordPress, Shopify, Next.js) or leave it on None to write your own rules.
List folders to block under Disallow Paths, for example /admin/, /cart, /search. Add exceptions under Allow Paths.
Optionally block AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended…) or aggressive SEO crawlers.
Enter your full sitemap URL so search engines can find every page.
Copy or download the file and upload it to the root of your domain, so it opens at https://yourdomain.com/robots.txt.
Frequently Asked Questions
Q:What is a robots.txt file?
A plain-text file at the root of your site that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol (RFC 9309), which Google, Bing and most legitimate bots respect.
Q:Where do I put my robots.txt file?
Only at the top-level root of each host, for example https://www.example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own file.
Q:Does robots.txt stop Google from indexing a page?
No. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
Q:How do I block ChatGPT and other AI crawlers?
Turn on "Block AI Crawlers". The generator adds Disallow: / rules for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, PerplexityBot, Bytespider, Applebot-Extended and meta-externalagent. Blocking Google-Extended does not affect normal Google Search.
Q:Does Google respect Crawl-delay?
No. Googlebot ignores Crawl-delay; Bing and Yandex honour it. If Google crawls too fast, lower the crawl rate from your server by returning 429 or 503 responses temporarily.
Q:What robots.txt should a WordPress site use?
A common safe setup blocks /wp-admin/ while allowing /wp-admin/admin-ajax.php, and blocks internal search URLs like /?s=. Never block /wp-content/ or /wp-includes/ completely, because Google needs your CSS and JavaScript to render pages.
Q:Should I include my XML sitemap in robots.txt?
Yes. A Sitemap: line with the absolute URL helps every crawler, not only those you have submitted the sitemap to in Search Console or Bing Webmaster Tools.
Related Tools
Explore similar micro-utilities and developer tools on Toolbrix.
Sitemap.xml Protocol Generator
Generate valid Google Search Console `sitemap.xml` files from a list of site URLs.
SEO Meta Tag & Open Graph Tag Generator
Generate HTML `<meta>` tags for titles, descriptions, viewport, and Open Graph social cards.
Apache .htaccess Configuration Generator
Generate Apache `.htaccess` web server rules for HTTPS redirect, Gzip compression, and 301 redirects.