Robots & Sitemap
Check robots.txt and discover & validate a site's sitemaps.
No results yet
Enter a host above and press Run to start the check.
About the Robots & Sitemap
The Robots & Sitemap tool retrieves a site's robots.txt file, parses its crawl directives, and discovers and validates the XML sitemaps it references. It shows which paths are allowed or disallowed for crawlers and confirms that your declared sitemaps actually exist and are well-formed. This helps you catch accidental blocks and broken sitemap links that quietly keep pages out of search results.
How to use
- Enter a domain or site URL to analyze.
- Fetch and parse the site's robots.txt directives.
- Review the allow/disallow rules and any declared sitemap locations.
- Open and validate each discovered sitemap for proper XML structure and reachable URLs.
How robots.txt and sitemaps actually interact
The most common misconception is that `Disallow` keeps a page out of the index. It does not — it stops a crawler fetching the page. If other sites link to that URL, a search engine can still index it from those links alone, which produces the familiar result with a title and no description. To keep a page out of the index you need a `noindex` meta tag or header, which means the crawler must be *allowed* to fetch the page in order to see it. Disallowing a page you have marked noindex guarantees the directive is never read.
Order and specificity decide which rule wins, not position in the file. Within the group that matches a crawler's user agent, the most specific matching path takes precedence, and `Allow` beats `Disallow` when both match at equal length. A broad `Disallow: /api/` with a narrow `Allow: /api/public/` behaves as intended for exactly this reason.
A sitemap is a discovery hint, not an instruction. Listing a URL does not force indexing, and omitting one does not prevent it. Two rules are worth respecting: every URL should be canonical and return 200, because listing redirects or 404s wastes crawl budget and erodes trust in the file; and a sitemap must not list URLs blocked by robots.txt, since the two files then contradict each other. The `Sitemap:` line in robots.txt is user-agent independent and applies to every crawler regardless of which group it appears in.
Frequently asked questions
- Where should my sitemap be declared?
- The most reliable place is a Sitemap: line inside robots.txt, which lets crawlers discover it automatically. You can also submit it directly in tools like Google Search Console.
- A page is missing from Google. Could robots.txt be the cause?
- Yes. A broad Disallow rule can block crawlers from reaching a page. This tool highlights the matching rules so you can confirm whether a path is intentionally or accidentally blocked.
- Does a Disallow rule remove a page from search results?
- Not directly. Disallow only stops crawling, not indexing. A blocked URL can still appear in results without a snippet. To remove a page from the index, allow crawling and use a noindex directive instead.