robots.txt Tester
Paste a robots.txt file, pick a crawler and list the URLs you want to check. See instantly which are allowed, which are blocked, and exactly which rule decided it.
Results for Googlebot
/
Allowed - no rule matches, and anything not disallowed is allowed
/cart
Blocked - matched "Disallow: /cart"
/search?q=shoes
Blocked - matched "Disallow: /search"
/search/help
Allowed - matched "Allow: /search/help"
/products/blue-shoes
Allowed - no rule matches, and anything not disallowed is allowed
Sitemap directives
- https://example.com/sitemap.xml
How it works
The tester follows the same rules search engines use: it finds the most specific user-agent group that matches your chosen crawler (falling back to the wildcard group), then compares every Allow and Disallow rule against each path. Where rules conflict, the longest matching rule wins, and Allow beats Disallow on a tie - exactly how Google resolves it. Wildcards (*) and end-of-URL anchors ($) are supported.
Basic guide
Paste your robots.txt contents, select the crawler you want to test and enter the URLs you want to check.
The tool tells you whether each URL is allowed or blocked and identifies the rule responsible for the decision.
Example
Given:
User-agent: * Disallow: /cart Disallow: /search Allow: /search/help
Test:
/cart /search /search/help /products/example
The tester shows which URLs are blocked and which are allowed, helping you identify accidental crawl restrictions.