Robots.txt Generator
Build a clean robots.txt file.
01 Overview
About Robots.txt Generator
Robots.txt Generator builds a clean robots.txt file for your website — the small text file, placed at your domain's root, that tells search engine crawlers which parts of your site they may visit and which to leave alone, so they spend their time on content that matters instead of admin or duplicate sections. Double-check every disallow rule before saving: one wrong rule can hide pages you want found. And robots.txt is a request polite crawlers follow, not a security control — never rely on it to hide sensitive pages. It runs entirely in your browser.
02 Workflow
How to use Robots.txt Generator
- 1Decide what crawlers may visitWork out which sections of your site should be crawlable and which should be left alone.
- 2Enter the rulesAdd the allow and disallow rules for those sections. The file is built on your device.
- 3Review every disallow ruleCheck the generated file carefully — one wrong rule can block search engines from important pages.
- 4Save it at your rootSave the file as robots.txt at the root of your domain (yourdomain.com/robots.txt), then test it to confirm crawlers can reach your important pages.
03 Good to know
Frequently Asked Questions
What is a robots.txt file?
It is a text file placed at the root of a website that gives instructions to search engine crawlers about which pages and sections they may crawl.
Where do I put the robots.txt file?
Place it at the root of your domain, so it is available at yourdomain.com/robots.txt. Crawlers look for it there automatically.
Can robots.txt keep a page private?
No. It is a request that polite crawlers follow, not a security control. Do not rely on it to hide sensitive pages; use proper access controls for those.
What is the difference between robots.txt and a meta noindex tag?
Robots.txt asks crawlers not to visit a page at all, while a noindex tag lets them visit it but asks them not to show it in results. Use noindex for pages that must stay out of search results, and robots.txt to save crawl effort on unimportant sections.
What is a User-agent in robots.txt?
It names which crawler a rule applies to. An asterisk means the rule applies to all crawlers, while a specific name, such as a search engine's crawler, limits the rule to that crawler only.
Should I block my admin pages in robots.txt?
Blocking them only keeps polite crawlers away; it does not hide them. Admin and private pages must be protected with login controls and proper access restrictions, not with a robots.txt rule.
How do I test my robots.txt file?
Save it at the root of your domain, then use your search console's robots.txt tester to confirm the rules work as intended and that crawlers can still reach your important pages.