Technical SEO
How to create a robots.txt file without blocking your site
Write a safe robots.txt: what to allow, what to block, how sitemap lines work, and the one mistake that can deindex a whole site.
What robots.txt actually does
robots.txt is a request to crawlers. It lives at https://yoursite.com/robots.txt. It can save crawl budget on junk URLs. It cannot reliably hide a URL from search results.
If you need a page out of Google, use a noindex tag or password protection. Blocking a URL in robots.txt can even prevent Google from seeing the noindex.
Safe defaults for most sites
Allow the public site. Disallow admin, cart, account, and internal search paths. Add one Sitemap line with the full HTTPS URL.
Be careful with Disallow: /. That rule tells every matching bot to skip the entire site. Preview the file before you upload it.
- One file at the root — not in a subdirectory
- Use user-agent groups for Googlebot vs AI crawlers if you need different rules
- Test in Google Search Console after you deploy
Build it in ToolVerse
The Robots.txt Generator lets you pick agents, add allow/disallow paths, set crawl-delay where it applies, and attach a sitemap. Copy the output to your server root and confirm it loads without a redirect.
Frequently asked questions
Should I block CSS and JS?
No. Google needs those files to render the page. Blocking them can hurt how your site is understood.
Do all bots obey robots.txt?
Good bots do. Malicious scrapers may ignore it. robots.txt is not a security control.
Can I have more than one sitemap line?
Yes. List each sitemap or sitemap index on its own Sitemap: line.