seoFree

Robots.txt Generator

Build a clean robots.txt file.

Free — no signupYour input stays on this device for this operation.Learn how to use Robots.txt Generator →
Robots.txt Generator illustration

https://example.com/page

Crawl rules are public instructions, not access controls. Protect private pages with authentication.

01 Overview

About Robots.txt Generator

Robots.txt Generator builds a clean robots.txt file for your website — the small text file, placed at your domain's root, that tells search engine crawlers which parts of your site they may visit and which to leave alone, so they spend their time on content that matters instead of admin or duplicate sections. Double-check every disallow rule before saving: one wrong rule can hide pages you want found. And robots.txt is a request polite crawlers follow, not a security control — never rely on it to hide sensitive pages. It runs entirely in your browser.

Adjust the options and review your result before downloading.
See our privacy policy for input handling and usage information.
Use a modern browser. Availability and performance depend on your device and the selected tool.

02 Workflow

How to use Robots.txt Generator

  1. 1
    Decide what crawlers may visitWork out which sections of your site should be crawlable and which should be left alone.
  2. 2
    Enter the rulesAdd the allow and disallow rules for those sections. The file is built on your device.
  3. 3
    Review every disallow ruleCheck the generated file carefully — one wrong rule can block search engines from important pages.
  4. 4
    Save it at your rootSave the file as robots.txt at the root of your domain (yourdomain.com/robots.txt), then test it to confirm crawlers can reach your important pages.

03 Good to know

Frequently Asked Questions

What is a robots.txt file?

It is a text file placed at the root of a website that gives instructions to search engine crawlers about which pages and sections they may crawl.

Where do I put the robots.txt file?

Place it at the root of your domain, so it is available at yourdomain.com/robots.txt. Crawlers look for it there automatically.

Can robots.txt keep a page private?

No. It is a request that polite crawlers follow, not a security control. Do not rely on it to hide sensitive pages; use proper access controls for those.

What is the difference between robots.txt and a meta noindex tag?

Robots.txt asks crawlers not to visit a page at all, while a noindex tag lets them visit it but asks them not to show it in results. Use noindex for pages that must stay out of search results, and robots.txt to save crawl effort on unimportant sections.

What is a User-agent in robots.txt?

It names which crawler a rule applies to. An asterisk means the rule applies to all crawlers, while a specific name, such as a search engine's crawler, limits the rule to that crawler only.

Should I block my admin pages in robots.txt?

Blocking them only keeps polite crawlers away; it does not hide them. Admin and private pages must be protected with login controls and proper access restrictions, not with a robots.txt rule.

How do I test my robots.txt file?

Save it at the root of your domain, then use your search console's robots.txt tester to confirm the rules work as intended and that crawlers can still reach your important pages.