Does robots.txt hide a page from Google?

Not reliably. It asks well-behaved crawlers not to fetch a URL, but a blocked URL can still be indexed if other sites link to it. To keep a page out of results, use a noindex directive and leave the page crawlable so the directive can be seen.

What does an empty Disallow line mean?

A Disallow with no path means nothing is disallowed, so the crawler may fetch everything. The tool keeps that line for Allow all and adds it to any group that has no rules, because a group without rules is invalid.

Does Googlebot obey Crawl-delay?

No, Google ignores the directive, although some other crawlers honor it. The field is optional and left out of the file when empty.

How does the tool tidy my paths?

It adds a missing leading slash, turns a lone star into /*, and cuts a pasted full URL down to its path and query, since rules match URL paths only.

Where does the finished file go?

At the top level of the host, for example https://example.com/robots.txt. Each subdomain and protocol needs its own file, and it should be served as plain text.

Developer toolsInstantPrivateNo account

Robots.txt Generator

Builds a robots.txt file from one or more user-agent groups, each with Allow and Disallow rules and an optional crawl delay, plus sitemap URLs.

Back to category

About Robots.txt generator

Builds a robots.txt file from one or more user-agent groups, each with Allow and Disallow rules and an optional crawl delay, plus sitemap URLs. Five presets give a complete starting point, and warnings flag mistakes such as a missing crawler name or a block-everything file that still advertises a sitemap. Output follows the Robots Exclusion Protocol, RFC 9309.

How to use it

Enter your Site URL, then click a preset: Allow all, Block all, Allow search block AI crawlers, WordPress or E-commerce.
Edit each group. Type crawler names under User-agents, one per line, or click the suggested names such as Googlebot and Bingbot.
Add rules with Add rule, choosing Disallow or Allow and a path. A missing leading slash is corrected for you.
Fill in Crawl-delay, the Sitemap URLs, a Header comment and an optional Host directive under Sitemaps & metadata.
Copy or download the preview and place it at /robots.txt in the root of your domain.

Keep search engines, turn away AI crawlers

Input

Site URL: https://example.com/
Preset: Allow search, block AI crawlers

Output

# Search engines allowed; AI training crawlers blocked

User-agent: *
Disallow:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: CCBot
User-agent: PerplexityBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

When to use it

Keeping cart, checkout and customer-area URLs out of search results on an online store.
Asking AI training crawlers to stay away while search engines carry on indexing the site.
Blocking a staging site entirely with the Block all preset so it does not appear in results.
Advertising several sitemap files for a large site from a single file.

Good to know. Rules are grouped per crawler, and a crawler follows only the most specific group that names it, so a Googlebot group does not inherit anything from the star group. The Host directive is a legacy extra that most crawlers ignore.

Questions about Robots.txt generator

Does robots.txt hide a page from Google?: Not reliably. It asks well-behaved crawlers not to fetch a URL, but a blocked URL can still be indexed if other sites link to it. To keep a page out of results, use a noindex directive and leave the page crawlable so the directive can be seen.
What does an empty Disallow line mean?: A Disallow with no path means nothing is disallowed, so the crawler may fetch everything. The tool keeps that line for Allow all and adds it to any group that has no rules, because a group without rules is invalid.
Does Googlebot obey Crawl-delay?: No, Google ignores the directive, although some other crawlers honor it. The field is optional and left out of the file when empty.
How does the tool tidy my paths?: It adds a missing leading slash, turns a lone star into /*, and cuts a pasted full URL down to its path and query, since rules match URL paths only.
Where does the finished file go?: At the top level of the host, for example https://example.com/robots.txt. Each subdomain and protocol needs its own file, and it should be served as plain text.

Related tools

Developer toolsInstantPrivateNo account

Robots.txt Generator

Builds a robots.txt file from one or more user-agent groups, each with Allow and Disallow rules and an optional crawl delay, plus sitemap URLs.

Back to category

WorkbenchClient-side · Private

About Robots.txt generator

How to use it

Enter your Site URL, then click a preset: Allow all, Block all, Allow search block AI crawlers, WordPress or E-commerce.
Edit each group. Type crawler names under User-agents, one per line, or click the suggested names such as Googlebot and Bingbot.
Add rules with Add rule, choosing Disallow or Allow and a path. A missing leading slash is corrected for you.
Fill in Crawl-delay, the Sitemap URLs, a Header comment and an optional Host directive under Sitemaps & metadata.
Copy or download the preview and place it at /robots.txt in the root of your domain.

Keep search engines, turn away AI crawlers

Input

Site URL: https://example.com/
Preset: Allow search, block AI crawlers

Output

# Search engines allowed; AI training crawlers blocked

User-agent: *
Disallow:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: CCBot
User-agent: PerplexityBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

When to use it

Keeping cart, checkout and customer-area URLs out of search results on an online store.
Asking AI training crawlers to stay away while search engines carry on indexing the site.
Blocking a staging site entirely with the Block all preset so it does not appear in results.
Advertising several sitemap files for a large site from a single file.

Questions about Robots.txt generator

Does robots.txt hide a page from Google?: Not reliably. It asks well-behaved crawlers not to fetch a URL, but a blocked URL can still be indexed if other sites link to it. To keep a page out of results, use a noindex directive and leave the page crawlable so the directive can be seen.
What does an empty Disallow line mean?: A Disallow with no path means nothing is disallowed, so the crawler may fetch everything. The tool keeps that line for Allow all and adds it to any group that has no rules, because a group without rules is invalid.
Does Googlebot obey Crawl-delay?: No, Google ignores the directive, although some other crawlers honor it. The field is optional and left out of the file when empty.
How does the tool tidy my paths?: It adds a missing leading slash, turns a lone star into /*, and cuts a pasted full URL down to its path and query, since rules match URL paths only.
Where does the finished file go?: At the top level of the host, for example https://example.com/robots.txt. Each subdomain and protocol needs its own file, and it should be served as plain text.

Search tools

Robots.txt Generator

About Robots.txt generator

How to use it

Keep search engines, turn away AI crawlers

When to use it

Questions about Robots.txt generator

Does robots.txt hide a page from Google?

What does an empty Disallow line mean?

Does Googlebot obey Crawl-delay?

How does the tool tidy my paths?

Where does the finished file go?

Related tools

Robots.txt Generator

About Robots.txt generator

How to use it

Keep search engines, turn away AI crawlers

When to use it

Questions about Robots.txt generator

Does robots.txt hide a page from Google?

What does an empty Disallow line mean?

Does Googlebot obey Crawl-delay?

How does the tool tidy my paths?

Where does the finished file go?

Related tools