Robots.txt Generator
Build and preview a robots.txt file with user-agent rules, allow/disallow paths, crawl-delay, and sitemap URL.
Quick Presets
Global Settings
User-agent: *
Rules
robots.txt Preview
User-agent: *
Upload instructions
Place robots.txt in the root of your domain (e.g. https://example.com/robots.txt). Changes take effect on the next crawl.
Simple workflow
How to build a robots.txt file
Step 1
Pick a crawler
Start with * for all bots, or target Googlebot, Bingbot, or AI crawlers like GPTBot.
Step 2
Set allow and disallow paths
Block admin, cart, and search URLs. Leave public pages crawlable.
Step 3
Add your sitemap
Include the full sitemap URL, then download or copy the file to your site root.
Frequently asked questions
Where do I put robots.txt?
At the domain root: https://yoursite.com/robots.txt. Subfolder files are ignored.
Does robots.txt hide a page from Google?
No. It asks crawlers not to fetch the URL. The URL can still appear in results. Use noindex if you want it out of the index.
Should I block AI bots?
Only if you do not want your content used for training or AI answers. Use the Block AI preset, then review which agents you actually want to allow.
Can a bad robots.txt hurt SEO?
Yes. Disallowing / by accident can stop Google from crawling the whole site.