About robots.txt
A robots.txt generator builds the robots.txt file that tells search-engine crawlers which parts of your site they may or may not crawl. Choose which user-agents to target, which paths to allow or disallow, and add your sitemap URL — then copy the file to your site root. Correct robots rules help search engines crawl your important pages efficiently. It runs in your browser.
How Robots.txt works
How to create a robots.txt
- Choose the user-agent (all crawlers, or a specific one).
- Add the paths to allow or disallow, and your sitemap URL.
- Copy the generated file and upload it to your site root as
/robots.txt.
What robots.txt does
It guides crawlers with Allow and Disallow rules per User-agent, and points them to your Sitemap. It is a request, not a security control — do not use it to hide private data.
Common rules
Allow everything, block admin or staging paths, or block all crawlers on a private site. Always include your sitemap URL so search engines find your pages.
Common uses
- Control which pages crawlers can access
- Block admin, cart or staging paths
- Point search engines to your sitemap
- Allow all crawlers on a public site
- Disallow all crawlers on a private site
- Set rules for a specific bot
- Avoid wasting crawl budget
- Generate a valid robots.txt fast
Frequently asked questions
How do I make a robots.txt file?
Choose the user-agent, set allow/disallow paths and your sitemap, then copy the file to your site root.
Where does robots.txt go?
At the root of your domain, served at /robots.txt.
Does robots.txt hide pages?
No — it asks crawlers not to crawl paths, but it is not security. Do not use it to protect private data.
Should I include my sitemap?
Yes — adding the Sitemap line helps search engines discover your pages.
What does Disallow: / do?
It asks crawlers not to crawl any page — use it only on private or staging sites.
Is anything uploaded?
No — the file is generated entirely in your browser.
Will this affect my rankings?
Good rules help crawlers focus on important pages; blocking the wrong paths can hurt, so review before publishing.
Is it free?
Yes — completely free with no sign-up.