What does Disallow: / do?

Disallow: / tells a crawler not to fetch any URL on the site for that user-agent. It is the basis of the Block all preset, so use it carefully on a live site.

Where do I put the robots.txt file?

It must live at the root of your domain, served at https://example.com/robots.txt. Crawlers only look there, so a robots.txt in a subfolder is ignored.

Does Disallow remove a page from Google?

No. Disallow asks compliant crawlers not to fetch the URL, but a blocked page can still appear in results from other links. Use a noindex meta tag or header to keep a page out of the index.

Can I set different rules for different bots?

Yes. Add a separate user-agent group for each bot, for example one for Googlebot and one for *. Each group gets its own Allow and Disallow lines in the output.

What is Crawl-delay and should I use it?

Crawl-delay asks a crawler to wait that many seconds between requests. It is honored by some engines like Bing but ignored by Googlebot, so it is optional and left blank by default.

Is my robots.txt sent to a server?

No. The robots.txt generator builds the file in your browser as you type, so the rules and sitemap URLs you enter stay on your device.

robots.txt Generator

Visually build a robots.txt file with user-agent rules, allow and disallow paths, crawl-delay, and sitemaps. Everything runs in your browser.

Your robots.txt rules and sitemap URLs are assembled locally in your browser and are never uploaded.

Need a sitemap too? Try the Sitemap Generator.

About robots.txt Generator

This robots.txt generator lets you build a robots.txt file visually instead of remembering the exact syntax. Add one or more rule groups, each with a User-agent (default *) plus Allow and Disallow path rows, then set an optional Crawl-delay and list your Sitemap URLs. Preset buttons fill in common setups: Allow all, Block all, Block common admin paths like /admin and /cgi-bin, and a WordPress configuration. The output panel updates live as you type, formatting each group correctly with a blank line between groups and Sitemap lines at the end. When a path rule does not start with a slash, you get a gentle warning rather than a block. Copy the result or download it as robots.txt to drop at the root of your site. Everything is generated in your browser, so nothing you type is uploaded.

Features

Build multiple user-agent groups, each with its own Allow and Disallow paths
Add or remove individual path rows so you can fine-tune what crawlers may visit
One-click presets: Allow all, Block all, Block common admin paths, and WordPress
Optional Crawl-delay and one or more absolute Sitemap URLs
Live output formats groups with a blank line between them and sitemaps at the end
Warns when a path rule does not start with a slash instead of blocking you
Copy the generated robots.txt or download it as a robots.txt file
Runs entirely in your browser with no upload of your rules

How to use the robots.txt Generator

Pick a preset like Allow all or Block all, or start a fresh user-agent group.
Set the User-agent and add Allow or Disallow path rows for that group.
Optionally add a Crawl-delay and your Sitemap URLs.
Review the live robots.txt in the output panel on the right.
Click Copy or Download to save robots.txt and upload it to your site root.

Example

Input

User-agent: *  →  Disallow: /admin, /cgi-bin/
Sitemap: https://example.com/sitemap.xml

Output

User-agent: *
Disallow: /admin
Disallow: /cgi-bin/

Sitemap: https://example.com/sitemap.xml

A single group blocking admin paths, with a sitemap appended at the end.

Common errors & troubleshooting

A path rule has no effect because it does not start with a slash. — Begin every Allow and Disallow value with /, for example /admin or /private/. The generator flags rules that do not.
Disallow: / accidentally blocks the whole site from search engines. — Disallow: / tells crawlers to skip every URL. Use the Allow all preset or an empty Disallow value to permit crawling again.
The Sitemap line is ignored by crawlers. — Sitemap values must be absolute URLs like https://example.com/sitemap.xml, not relative paths such as /sitemap.xml.
Crawlers still index a page you disallowed. — robots.txt only requests that compliant crawlers not fetch a URL; it does not remove pages from an index. Use a noindex meta tag or header for that.

Frequently asked questions

What does Disallow: / do?: Disallow: / tells a crawler not to fetch any URL on the site for that user-agent. It is the basis of the Block all preset, so use it carefully on a live site.
Where do I put the robots.txt file?: It must live at the root of your domain, served at https://example.com/robots.txt. Crawlers only look there, so a robots.txt in a subfolder is ignored.
Does Disallow remove a page from Google?: No. Disallow asks compliant crawlers not to fetch the URL, but a blocked page can still appear in results from other links. Use a noindex meta tag or header to keep a page out of the index.
Can I set different rules for different bots?: Yes. Add a separate user-agent group for each bot, for example one for Googlebot and one for *. Each group gets its own Allow and Disallow lines in the output.
What is Crawl-delay and should I use it?: Crawl-delay asks a crawler to wait that many seconds between requests. It is honored by some engines like Bing but ignored by Googlebot, so it is optional and left blank by default.
Is my robots.txt sent to a server?: No. The robots.txt generator builds the file in your browser as you type, so the rules and sitemap URLs you enter stay on your device.

Related tools

XML Sitemap Generator — Turn a list of URLs into a standards-compliant sitemap.xml with optional lastmod, changefreq and priority.
.gitignore Generator — Build a .gitignore from common language and tool templates.
HTTP Status & MIME Types — Searchable reference of HTTP status codes and MIME content types.
Open Graph Generator — Generate Open Graph & Twitter meta tags with a live social-card preview.
URL Parser — Break a URL into its parts and list query parameters.