Robots & Sitemap
Check robots.txt and discover & validate a site's sitemaps.
No results yet
Enter a host above and press Run to start the check.
About the Robots & Sitemap
The Robots & Sitemap tool retrieves a site's robots.txt file, parses its crawl directives, and discovers and validates the XML sitemaps it references. It shows which paths are allowed or disallowed for crawlers and confirms that your declared sitemaps actually exist and are well-formed. This helps you catch accidental blocks and broken sitemap links that quietly keep pages out of search results.
How to use
- Enter a domain or site URL to analyze.
- Fetch and parse the site's robots.txt directives.
- Review the allow/disallow rules and any declared sitemap locations.
- Open and validate each discovered sitemap for proper XML structure and reachable URLs.
Frequently asked questions
- Where should my sitemap be declared?
- The most reliable place is a Sitemap: line inside robots.txt, which lets crawlers discover it automatically. You can also submit it directly in tools like Google Search Console.
- A page is missing from Google. Could robots.txt be the cause?
- Yes. A broad Disallow rule can block crawlers from reaching a page. This tool highlights the matching rules so you can confirm whether a path is intentionally or accidentally blocked.
- Does a Disallow rule remove a page from search results?
- Not directly. Disallow only stops crawling, not indexing. A blocked URL can still appear in results without a snippet. To remove a page from the index, allow crawling and use a noindex directive instead.