Robots.txt, sitemaps and SEO: the simple checks that prevent avoidable problems

Robots.txt is usually appropriate for internal search results, private areas, staging environments and technical paths that should not consume crawl attention. It should not be used as a casual way to hide poor or duplicate pages already indexed; use the right page-level controls or canonical strategy instead.

Before publishing a change, test the exact path. A broad disallow rule can inadvertently prevent crawlers from reaching product pages, category pages, JavaScript or images required to render the site. Use the site’s robots.txt tools to draft and check the syntax, then test the deployed file in the relevant webmaster platform.

Include the canonical, indexable versions of important pages: categories, products, editorial guides and major tools. Exclude redirected URLs, parameter variants, internal results, error pages and pages deliberately marked noindex. Update the sitemap when the catalogue or guide library changes, and submit it through the appropriate webmaster platform.

A sitemap is a discovery signal, not an order to index every URL. Search engines still decide whether a page is eligible and worthwhile. The stronger signal is a page that is useful, accessible and naturally linked from relevant category or guide pages.

Review crawl and indexing reports for unexpected exclusions, redirect chains, canonical conflicts and server errors. Pick a few priority URLs and confirm that a crawler can fetch the same useful content that a visitor sees. If a page is intentionally unavailable, make the response consistent rather than leaving broken internal links.

For Google’s current guidance on sitemap construction and submission, refer to its official sitemap documentation.