This robots.txt generator lets you build a robots.txt file visually instead of remembering the exact syntax. Add one or more rule groups, each with a User-agent (default *) plus Allow and Disallow path rows, then set an optional Crawl-delay and list your Sitemap URLs. Preset buttons fill in common setups: Allow all, Block all, Block common admin paths like /admin and /cgi-bin, and a WordPress configuration. The output panel updates live as you type, formatting each group correctly with a blank line between groups and Sitemap lines at the end. When a path rule does not start with a slash, you get a gentle warning rather than a block. Copy the result or download it as robots.txt to drop at the root of your site. Everything is generated in your browser, so nothing you type is uploaded.
Features
Build multiple user-agent groups, each with its own Allow and Disallow paths
Add or remove individual path rows so you can fine-tune what crawlers may visit
One-click presets: Allow all, Block all, Block common admin paths, and WordPress
Optional Crawl-delay and one or more absolute Sitemap URLs
Live output formats groups with a blank line between them and sitemaps at the end
Warns when a path rule does not start with a slash instead of blocking you
Copy the generated robots.txt or download it as a robots.txt file
Runs entirely in your browser with no upload of your rules
How to use the robots.txt Generator
Pick a preset like Allow all or Block all, or start a fresh user-agent group.
Set the User-agent and add Allow or Disallow path rows for that group.
Optionally add a Crawl-delay and your Sitemap URLs.
Review the live robots.txt in the output panel on the right.
Click Copy or Download to save robots.txt and upload it to your site root.
A single group blocking admin paths, with a sitemap appended at the end.
Common errors & troubleshooting
A path rule has no effect because it does not start with a slash. — Begin every Allow and Disallow value with /, for example /admin or /private/. The generator flags rules that do not.
Disallow: / accidentally blocks the whole site from search engines. — Disallow: / tells crawlers to skip every URL. Use the Allow all preset or an empty Disallow value to permit crawling again.
The Sitemap line is ignored by crawlers. — Sitemap values must be absolute URLs like https://example.com/sitemap.xml, not relative paths such as /sitemap.xml.
Crawlers still index a page you disallowed. — robots.txt only requests that compliant crawlers not fetch a URL; it does not remove pages from an index. Use a noindex meta tag or header for that.
Frequently asked questions
What does Disallow: / do?
Disallow: / tells a crawler not to fetch any URL on the site for that user-agent. It is the basis of the Block all preset, so use it carefully on a live site.
Where do I put the robots.txt file?
It must live at the root of your domain, served at https://example.com/robots.txt. Crawlers only look there, so a robots.txt in a subfolder is ignored.
Does Disallow remove a page from Google?
No. Disallow asks compliant crawlers not to fetch the URL, but a blocked page can still appear in results from other links. Use a noindex meta tag or header to keep a page out of the index.
Can I set different rules for different bots?
Yes. Add a separate user-agent group for each bot, for example one for Googlebot and one for *. Each group gets its own Allow and Disallow lines in the output.
What is Crawl-delay and should I use it?
Crawl-delay asks a crawler to wait that many seconds between requests. It is honored by some engines like Bing but ignored by Googlebot, so it is optional and left blank by default.
Is my robots.txt sent to a server?
No. The robots.txt generator builds the file in your browser as you type, so the rules and sitemap URLs you enter stay on your device.
Related tools
XML Sitemap Generator — Turn a list of URLs into a standards-compliant sitemap.xml with optional lastmod, changefreq and priority.
.gitignore Generator — Build a .gitignore from common language and tool templates.
Cache-Control Header Builder — Assemble a correct HTTP Cache-Control header from plain toggles and human time units, with copy-ready nginx, Apache, and meta snippets.
Meta Tag Generator — Generate SEO, Open Graph and Twitter Card meta tags for your page from a simple form.
CSP Header Generator — Build a Content-Security-Policy header from directive toggles, with report-only and copy output.
htpasswd Generator — Generate Apache/nginx .htpasswd entries with bcrypt or SHA-1 hashed passwords, in your browser.
Tailwind Color Generator — Generate a full 50-950 Tailwind color scale from one brand color, with config-ready output.
CSS Grid Generator — Build CSS grid layouts visually and copy grid-template-columns, rows and gap.
CSS Clamp Calculator — Calculate a fluid CSS clamp() value from min and max sizes and viewport widths.
Cubic Bezier Generator — Design a cubic-bezier() easing curve with draggable handles and an animated preview.