Robots.txt Tester & Validator
Paste a robots.txt file to check it line by line, then test URLs against it as Googlebot, Bingbot or any other crawler. For each URL you'll see whether it's allowed or blocked, which group applied and exactly which rule decided it.
How to use the Robots.txt Tester & Validator
Paste the file
Open https://yoursite.com/robots.txt, copy everything and paste it in. A draft works too.
Review validation notes
Syntax errors, unknown directives and rules outside any group are listed with line numbers.
Add URLs to test
Paths (/blog/) or full URLs. Query strings are included in matching, as crawlers do.
Pick a user-agent
Results change by crawler: a specific group replaces the * group for that bot.
What it checks
- Syntax
- Every line should be a directive, a comment or blank; missing colons and unknown directives are flagged.
- Group structure
- Allow/Disallow lines before any User-agent line, and empty groups.
- Unsupported rules
- Directives such as noindex or host that major crawlers don't support in robots.txt.
- Sitemaps
- Sitemap lines must contain absolute URLs.
- Size
- Google processes the first 500 KiB of a robots.txt file; larger files are flagged.
- URL decisions
- Longest-match rule selection, Allow winning ties, and * and $ wildcards — the matching model documented by Google and RFC 9309.
Interpreting the results
For each URL the result shows Allowed or Blocked, the user-agent group that applied, and the deciding rule with its line number. When no rule matches, the URL is allowed by default.
Matching follows the published standard (RFC 9309) and Google's documentation: the crawler picks the group with the most specific matching user-agent; among that group's rules, the longest matching path wins; if an Allow and Disallow are equally long, Allow wins. * matches any sequence of characters and $ anchors the end of the URL.
The validation panel lists problems separately from test results. A “not supported” note means the line is ignored by major crawlers — not that the whole file fails.
Detected marks facts read directly from what you provided. Guidance marks general recommendations based on those facts — rules of thumb, not guarantees or search engine requirements.
Practical guidance
The most useful time to run this tester is before you publish changes. A single misplaced line — Disallow: / instead of Disallow: /search — can block a whole site, and it's easy to miss in review. Paste the new file, test your most important URLs (homepage, key categories, top landing pages) and confirm they're all allowed.
A realistic surprise: you add a User-agent: Googlebot group to block one folder, and suddenly Googlebot ignores all your * rules — because the more specific group replaces the general one. Testing as Googlebot and as * side by side shows this immediately.
Pages you want to rank, and pages receiving links from link building, must be crawlable. Pair this with the Robots.txt Generator and check that sitemap URLs listed in the file exist using the XML Sitemap Validator. Our common link building mistakes guide covers blocked targets too.
Limitations
- It tests the file you paste. It doesn't fetch your live robots.txt, so make sure you're testing the current version.
- Crawlers differ in details. Results follow RFC 9309 and Google's documented behaviour; other crawlers may interpret edge cases differently.
- A URL being allowed doesn't mean it will be crawled or indexed — only that robots.txt doesn't prevent it.
Frequently asked questions
Why is a URL allowed when I expected it to be blocked?
Usually a longer Allow rule matches, or the crawler is using a more specific group that doesn't contain your Disallow. The result shows the deciding rule.
Are paths case-sensitive?
Yes. /Blog/ and /blog/ are different paths. User-agent names are not case-sensitive.
Does Google support noindex in robots.txt?
No. Google stopped supporting it in 2019. Use a noindex meta tag or X-Robots-Tag header instead.
Is my file uploaded?
No. It's parsed and tested in your browser.
Related tools
- Robots.txt Generator
Build a robots.txt with default policy, custom rules, AI crawler blocks and sitemaps. - XML Sitemap Validator
Check sitemap XML for syntax, namespace, URL, lastmod, duplicate and size problems. - HTTP Status Code Checker
Live check of up to 10 URLs: status, full redirect chain, final URL and timing. - SEO Page Analyzer
Fetch a live URL and review status, tags, headings, links, images and schema in one report.
More in Technical SEO Tools · All 25 free SEO tools