Robots.txt Checker

About Robots.txt checker

Give this tool a domain and it downloads its robots.txt, parses every group and rule, and points out problems with line numbers. It shows crawler groups, Allow and Disallow paths, crawl delays and declared sitemaps, so you can catch mistakes such as an accidental sitewide block before search engines do.

How to use it

1. Enter a domain or URL in the Domain or URL field. Only the site origin is used.
2. Select Check robots.txt. The address is sent to our server, which fetches the file.
3. Read the headline for the number of problems or warnings, plus the HTTP status.
4. Go through the Findings list, noting each line number and severity.
5. Review each User-agent table and the Declared sitemaps, then use Copy to take the raw file.

A rule stranded after a blank line

User-agent: *
Disallow: /admin/
Allow: /admin/help

Disallow: /tmp/
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Warning on line 5: rules continue after a blank line, which ends a record, so many crawlers start a new group there and ignore the rule. One group is parsed for the wildcard agent with three rules and a crawl delay of 10, and one sitemap is declared.

When to use it

• Making sure a staging rule Disallow: / did not ship to production.
• Finding a misspelled directive that crawlers silently ignore.
• Checking that your sitemap is declared where search engines will look for it.
• Confirming the file stays under the 500 KB limit that Google enforces.

Questions about Robots.txt checker

Which file does it fetch?

Always /robots.txt at the origin of whatever you enter. A path in the input is dropped, and a non-http scheme such as ftp is rejected rather than rewritten.

What happens when the site has no robots.txt?

A 404 or 410 is reported as a warning, since crawlers treat a missing file as permission to crawl everything. Other error statuses are shown as a failed check.

Which problems does it detect?

Missing colons, unknown directives, rules before any User-agent line, empty User-agent values, non-numeric crawl delays, sitemaps that are not absolute URLs, empty files and Disallow: / blocks.

How much of the file is shown?

The parser reads up to about 600 KB, flags anything above 500 KB, and the raw view shows the first 20,000 characters. That is enough to spot most issues.

Does it test whether a specific URL is blocked?

No. It parses and validates the file, and lists the rules per group, but it does not evaluate a particular path against them.

Related tools

• Sitemap checkerValidate a sitemap.xml for malformed entries, bad lastmod dates, cross-host URLs and protocol limits.
• Meta Tags CheckerCheck meta tags on a webpage
• Password Strength CheckerCheck the strength of a password
• Safe URL CheckerCheck if a URL is safe to visit
• File MIME Type CheckerCheck the MIME type of a file
• Gravatar CheckerCheck if an email has a Gravatar