Robots.txt Checker and Validator
Fetch a site's robots.txt or paste one, enter a user-agent token and a URL or path, and see whether the rules allow it, which group and rule decided that, and any lines with problems. Fetching requests https://your-site/robots.txt from this server and follows up to 5 redirects. Pasted text is checked in your browser, and nothing is saved.
How to check a robots.txt file
- Enter your domain, such as example.com, and press Fetch robots.txt. Or paste the file into the box.
- Read the file check for syntax errors and directives that crawlers ignore.
- Enter a user-agent token, such as Googlebot, and the URL or path you want to test.
- The result shows whether the URL is allowed, and which group, rule and line decided it.
allowed for Googlebot
Result
Allowed
Googlebot testing /private/press-kit/logo.png
- Winning rule
Allow: /private/press-kit/on line 4- Why
- No group names "Googlebot", so the * group(s) starting on line 2 apply. "Allow: /private/press-kit/" on line 4 decides the result because it is the longest matching rule.
- Selected groups
User-agent: *on line 23 rules
- All matching rules
Disallow: /private/on line 3, specificity 9Allow: /private/press-kit/on line 4, specificity 19
This result applies the rules on this page. Other crawlers may interpret the same file differently, and an allowed URL is not guaranteed to be crawled or indexed.
File check
Invalid syntax (1)
These lines do not follow the robots.txt format. Lines marked Error are ignored. Lines marked Warning are still read as the message describes.
- Line 9 error"Disallow /drafts/" has no colon, so it is not a "field: value" line and is ignored.
Unsupported directives (1)
These fields are not syntax errors. This checker ignores them, so they do not change the result, and other crawlers may treat them differently.
- Line 6 ignoredCrawl-delay is well-formed but is not an allow/disallow rule, so it does not affect this result. Other crawlers may treat it differently.
What robots.txt does not do
- It is not access control. Compliant crawlers choose to follow it, and anyone can read it and request the paths it lists. The RFC security section says it does not replace content security measures.
- It does not itself remove an indexed URL from search. A blocked URL can stay indexed if other pages link to it, and blocking it stops crawlers from seeing a
noindextag on the page. - Other crawlers may interpret rules differently. This tool reports what the rules below produce, not what any specific crawler will do, and it says nothing about rankings.
Worked examples
These examples are self-contained and are not taken from this site's robots.txt. Each result is computed by the same code as the checker above.
The longest matching rule wins
User-agent: * Disallow: /docs/ Allow: /docs/public/
Googlebot testing /docs/public/guide: Allowed by Allow: /docs/public/ on line 3.
Both rules match, and the longer Allow path is more specific.
Allow wins a tie
User-agent: * Allow: /fish Disallow: /fis*
Googlebot testing /fish/salmon: Allowed by Allow: /fish on line 2.
Both rules are five characters long, so they are equally specific. The * counts toward the length here, following Google's open-source parser. Google's documentation treats a trailing * as adding nothing, and other crawlers may decide this differently.
A wildcard with an end anchor
User-agent: * Disallow: /*.pdf$
Googlebot testing /files/report.pdf: Disallowed by Disallow: /*.pdf$ on line 2.
The * spans the directories, and $ requires the URL to end in .pdf.
The end anchor includes the query string
User-agent: * Disallow: /*.pdf$
Googlebot testing /files/report.pdf?download=1: Allowed because no rule matches.
The same rule no longer matches, because the URL now ends in the query.
A named group replaces the * group
User-agent: * Disallow: / User-agent: ExampleBot Disallow: /private/
ExampleBot testing /blog/post: Allowed because no rule matches.
ExampleBot follows only its own group, so the Disallow: / in the * group does not apply to it.
An empty Disallow matches nothing
User-agent: * Disallow:
Googlebot testing /anything: Allowed because no rule matches.
The line is valid, and with no matching rule the URL is allowed by default.
The rules this checker applies
These rules follow RFC 9309 and how Google interprets the robots.txt specification. "Interpretation" marks behaviour the sources leave open or disagree on, where this tool had to choose.
- Group selection
- A group applies when its product token equals the crawler token, ignoring case and any text after the token. Otherwise the * group applies. With neither, no rules apply. A related token such as googlebot does not cover googlebot-news.RFC 9309 §2.2.1 · Google: Order of precedence for user agents · Google: Matching of user-agent fields
- Group merging
- All groups matching the same token are combined. Specific groups and * groups are never combined with each other.RFC 9309 §2.2.1 · Google: Grouping of rules
- Group boundaries
- Only a user-agent line after a rule, or the end of the file, ends a group. Sitemap and other records never end one.RFC 9309 §2.1 · RFC 9309 §2.2.4 · Google: Grouping of lines and rules
- Invalid user-agent linesInterpretation
- An invalid user-agent line after a rule still ends that group, and the rules below it are reported as outside any group instead of being attached to the previous crawler.RFC 9309 §2.1 · RFC 9309 §2.2.2
- Longest match
- The matching rule with the most octets wins, measured on the rule as written, with non-ASCII characters percent-encoded and every * and $ counted. A rule matches from the first octet of the path, which includes the query string.RFC 9309 §2.2.2 · Google: Order of precedence for rules · Google: open-source parser
- Ties
- When an Allow and a Disallow rule are equally specific, Allow wins.RFC 9309 §2.2.2 · Google: Order of precedence for rules
- Default result
- A URL matched by no rule is allowed, and /robots.txt itself is always allowed.RFC 9309 §2.2.2
- Empty rules
- An Allow or Disallow line with no path is valid and matches nothing.RFC 9309 §2.2 · Google: disallow · Google: allow
- WildcardsInterpretation
- * matches zero or more characters and $ marks the end of the URL. A trailing * does not change which URLs match, so /fish* matches the same URLs as /fish. On specificity the sources conflict: Google's documentation lists the two as equivalent, while Google's open-source parser counts the *. This tool follows the parser, so Disallow: /* beats Allow: /. Other crawlers may decide this differently.RFC 9309 §2.2.3 · Google: URL matching based on path values · Google: open-source parser
- Literal * and $
- %2A and %24 in a rule match a literal * and $ in the URL.RFC 9309 §2.2.3
- $ before the end of a ruleInterpretation
- A $ anywhere but the end of a rule is matched literally and flagged, because the spec only defines it as the end of the pattern.RFC 9309 §2.2.3
- Rule pathsInterpretation
- A rule path starts with /. A leading * is also applied, with a warning, because the RFC grammar and Google require / while the RFC example uses *.gif$. Any other value, or one containing raw whitespace, is reported and not applied.RFC 9309 §2.2 · RFC 9309 §5.1 · Google: Syntax
- Case
- Field names and product tokens ignore case. Paths are case-sensitive.RFC 9309 §2.2.1 · RFC 9309 §2.2.2 · Google: Syntax
- Percent-encoding
- Both sides are compared in percent-encoded UTF-8. Encoded unreserved characters are decoded, so %62 equals b. Other encoded octets stay encoded, so %2F does not equal / in a path.RFC 9309 §2.2.2 · Google: URL matching based on path values
- Reserved characters in queriesInterpretation
- A : or / after the ? also matches its encoded form, as in the RFC example that pairs https://foo.bar with https%3A%2F%2Ffoo.bar. A lone % is treated as %25. Dot segments are not resolved.RFC 9309 §2.2.2
- Other fields
- Other well-formed fields such as Crawl-delay are valid syntax that this evaluator ignores, and they are reported separately from invalid lines. Misspelled field names are not corrected. Other crawlers may treat these fields differently.RFC 9309 §2.2.4 · Google: Syntax
- Invalid lines
- Invalid lines are reported and ignored, and every parseable line is still used. Rules before the first user-agent line are ignored.RFC 9309 §2.2.2 · RFC 9309 §2.3.1.5 · Google: File format
- Comments and line endings
- Everything after # is a comment. Lines end in CR, LF or CRLF. Space around a line, field or value is ignored.RFC 9309 §2.2 · RFC 9309 §2.2.3 · Google: File format · Google: Syntax
- Byte order mark
- A leading Unicode byte order mark is ignored.Google: File format
- Size limit
- Only the first 500 KiB of UTF-8 is parsed, and content after that is ignored. The RFC requires any limit to be at least 500 KiB.RFC 9309 §2.5 · Google: File format
Frequently asked questions
Does this robots.txt checker fetch my site?
Only if you ask it to. When you enter a site and press Fetch robots.txt, this server requests /robots.txt from that host over HTTPS, follows up to 5 redirects as Google does, and sends the text back to your browser only if the final URL is a robots.txt file. It never requests the URL you test or a sitemap in the file, and nothing is stored. Text you paste is checked in your browser and is not sent anywhere.
How do I test robots.txt like Google?
Google retired its old robots.txt Tester. Search Console now has a robots.txt report that shows the files Google found for your site, when it last crawled them, and any parsing problems. To check a single URL, this tester follows the matching rules Google documents, listed below with their sources: the most specific group, the longest matching rule, and Allow on a tie.
Is robots.txt a form of access control?
No. Crawlers choose to follow it, and anyone can read the file and request a listed path. RFC 9309 says it is not a substitute for valid content security measures, so protect private pages with authentication.
Will a Disallow rule remove a URL from search results?
Not by itself. A blocked URL can still be indexed if other pages link to it, and a crawler that is blocked cannot see a noindex tag on the page. To keep a page out of search, allow crawling and use noindex, or put the page behind authentication.
Why might a crawler disagree with this result?
This tool applies the rules listed on this page. Crawlers may implement them differently, ignore robots.txt altogether, or use their own fields such as Crawl-delay. Where the sources leave a choice open, the rule below is marked as an interpretation.