robots.txt Tester
Paste a robots.txt file, pick a crawler and list the URLs you want to check. See instantly which are allowed, which are blocked, and exactly which rule decided it.
Results for Googlebot
/
Allowed - no rule matches, and anything not disallowed is allowed
/cart
Blocked - matched "Disallow: /cart"
/search?q=shoes
Blocked - matched "Disallow: /search"
/search/help
Allowed - matched "Allow: /search/help"
/products/blue-shoes
Allowed - no rule matches, and anything not disallowed is allowed
Sitemap directives
- https://example.com/sitemap.xml
How it works
The tester follows the same rules search engines use: it finds the most specific user-agent group that matches your chosen crawler (falling back to the wildcard group), then compares every Allow and Disallow rule against each path. Where rules conflict, the longest matching rule wins, and Allow beats Disallow on a tie - exactly how Google resolves it. Wildcards (*) and end-of-URL anchors ($) are supported.