robots.txt Generator
Pick your crawl rules and get a valid robots.txt you can drop straight into your site root.
robots.txt
User-agent: * Disallow: /cart Disallow: /checkout Disallow: /search Disallow: /*?sort=
Save this as robots.txt in your site root, e.g. https://example.com/robots.txt
How it works
Add your sitemap URL, choose which crawlers to address, then add Allow and Disallow rules. The tool builds a valid robots.txt file you can paste into your site root.
Basic guide
Choose whether crawling should be allowed, add paths that should not be crawled and optionally specify your sitemap URL.
You can also block selected crawlers. Review the generated file before placing it at the root of your website as /robots.txt.
Example
You want search engines to crawl the site but not crawl internal search or checkout pages.
Disallowed paths:
/cart /checkout /search
Sitemap:
https://example.com/sitemap.xml
The tool generates a robots.txt file containing the corresponding crawl rules and sitemap declaration.