Being listed in a sitemap is not the same as being indexed by a search engine. A sitemap is essentially a request: it tells a crawler which URLs you believe are important and where to find them. A crawler may still ignore that request, fail to retrieve the URL, or decide not to include the page in its search results.
For example, a page can appear in a sitemap and still be unavailable because the server returns an error, the URL redirects unexpectedly, or the page contains a noindex directive. A crawler can visit a page and discover that directive, while a page that has not been crawled cannot yet have a meaningful index status assigned to it. Understanding these stages helps separate a publishing problem from a crawl-discovery problem and an indexing-directive problem.
Why this matters
Search engines use several distinct processes. Submission means making a URL known to a search engine, usually by including it in a sitemap. Crawling means retrieving the URL to see its content and directives. Indexing means retaining enough information about the page to make it eligible to appear in search results. Each stage depends on the previous one, but none guarantees the next.
The noindex directive is particularly easy to misunderstand. When a crawler retrieves a page containing a noindex directive, that directive says the page should not be kept in the search index. It does not prevent the crawler from fetching the page. A crawler can therefore crawl a URL and confirm that it is excluded from the index. The directive can also affect how search engines handle links and preview information, depending on how it is implemented.
This distinction matters when diagnosing a missing page. If a URL is absent from the sitemap, the first issue is discovery. If it is present but returns a noindex directive, the issue is an explicit exclusion instruction. If it is crawled and has no noindex directive, the remaining questions concern canonicalization, page quality, duplication, and the search engine's broader indexing decisions. A browser tool cannot replace a search engine's own coverage or index report, but it can perform useful checks on the URL and the site around it.
The process
- Define the URL you are checking. Use the complete, canonical page address, including the scheme and any meaningful path. For example, check https://example.com/articles/indexing-basics rather than assuming that the home page and the article have the same indexing behavior. A page can be indexed independently of the rest of the site.
- Inspect the page for a noindex directive. Enter the URL into noindex-checker. The tool's source is associated with an analyze-page request, so it is intended to analyze the target page rather than search a sitemap index. If the URL field is empty, the tool reports Please enter a URL. If retrieval fails, it reports Failed to fetch. Check the URL. Treat a fetch failure as an inconclusive result, not proof that the page is missing from the index.
- Interpret the noindex result according to the page's state. If the page contains noindex, the crawler can discover the page while being instructed not to index it. If the page is accessible and the checker does not find that directive, the URL is no longer being excluded by that particular check, but that still does not prove that it has been indexed. It may not have been discovered or crawled yet.
- Examine the site structure with sitemap-generator. Start with the site's root URL and review Crawl Depth: and Max Pages:. These controls determine how much of the site the generator explores. A shallow depth may show only the home page and top-level pages, while a larger maximum may reveal additional URLs. The Page List displays the URLs the generator found.
- Check whether the target URL appears in the generated page list. Appearance in that list confirms that the sitemap generator discovered the URL during its crawl. It does not confirm that a search engine has submitted, crawled, or indexed it. It is evidence that the URL is structurally reachable from the site, not evidence of index membership.
- Inspect the sitemap data and copy it when needed. The tool provides Copy XML for copying the generated sitemap content. Use that output when checking what URLs are being advertised to crawlers. The generator can also download the result. Reviewing the actual XML makes it easier to spot a missing URL, an unexpected redirect, or a page that is included in the list even though its content carries a noindex directive.
- Review the page's search-result presentation separately. In google-search-preview, enter the Page Title, Page URL, and Meta Description as they currently appear on the page. This tool works entirely in the browser and helps evaluate the text and URL that would form a search-result snippet. Its character limits are 160 characters for the meta description, 160 characters for the page title, and 60 characters for the page URL.
- Draw a stage-by-stage conclusion. Record four separate facts: the noindex check result, whether the URL was found by sitemap-generator, whether the URL is included in the generated sitemap, and what search-result text appears in google-search-preview. Only a search engine's own indexing information can directly confirm index membership. The other checks explain whether the page is discoverable, crawlable, and eligible.
Worked example
Suppose you publish an article at https://example.com/guides/check-indexing. The article is linked from a category page, but it is not appearing in search results. First, use noindex-checker with that complete URL. If the page is accessible and the checker reports a noindex directive, you have identified an explicit indexing exclusion. The page can still be linked and can still be crawled; the directive tells the crawler not to retain it as an indexed result. Remove or correct the directive before using a sitemap as an indexing remedy.
Next, run sitemap-generator against https://example.com. Set Crawl Depth: to a level that reaches the article's section and set Max Pages: high enough to include the article. Check Page List for /guides/check-indexing. If it is absent, the generator's crawl did not find that path from the chosen starting point. That does not necessarily mean a search engine cannot find it, but it does show that the sitemap generator is not advertising it. If it is present, copy the XML with Copy XML and verify that the exact URL is included rather than a redirecting or variant address.
Finally, use google-search-preview to review the page's current Page Title, Page URL, and Meta Description. For example, a title of How to Check That a Page Is Actually Indexed and a description such as Check whether a URL is discoverable, crawlable, and free of a noindex directive. can be evaluated within the listed character limits. This step checks how the page may be presented in results; it does not change the page's indexing state. If the noindex check is clear and the URL is in the sitemap, the remaining issue is the search engine's discovery and indexing process, which these tools cannot independently certify.
Common mistakes
- Treating sitemap inclusion as an index check. A sitemap entry says that a URL should be crawled. It does not report whether the URL was retrieved, whether it was excluded, or whether it is currently available in results.
- Assuming noindex prevents crawling. A crawler may need to fetch a page before it can see a noindex directive. The page can therefore be crawled while remaining deliberately unindexed.
- Calling a fetch error a missing page. The Failed to fetch. Check the URL. message from noindex-checker means the check could not retrieve the target. It does not establish that the search engine has not indexed it.
- Checking only the home page. Indexing decisions can differ by URL. A home page may be indexed while an article is redirected, unavailable, or marked noindex.
- Using a preview tool as an index-status tool. google-search-preview evaluates title, description, and URL presentation in the browser. It has no connection to a search engine's index and cannot confirm membership.
- Testing too few URLs at a shallow crawl depth. A sitemap-generator run that does not reach the relevant section may produce a misleadingly short Page List. Adjust Crawl Depth: and Max Pages: before concluding that the URL is absent.
- Ignoring inconsistent URL forms. A path with and without a trailing slash, or a URL with different query parameters, can represent separate crawler requests. Compare the exact address in the page list and sitemap XML with the address you checked.
Limitations of this approach
- These tools can inspect or generate information about a URL, but they cannot query a search engine's definitive index. Only the search engine's own indexing or coverage information can provide that confirmation.
- A successful page fetch does not prove that a crawler visited the page. noindex-checker retrieves the page for your check; that retrieval is separate from the search engine's crawl history.
- The absence of a detected noindex directive does not guarantee indexing. Search engines may make independent decisions about duplicate content, canonical URLs, page quality, and whether a result is worth retaining.
- A browser-based sitemap generator is not a full search-engine crawler. Its Page List and generated XML reflect its own crawl of the supplied site, including the chosen Crawl Depth: and Max Pages:.
- noindex-checker cannot diagnose a page that it cannot fetch. Network restrictions, redirects, server errors, and access controls can all make the result inconclusive.
- google-search-preview is a presentation tool, not a submission or diagnostics tool. It cannot tell you whether the URL is indexed, whether the site is blocked, or whether a search engine has chosen a different title and description.
Frequently asked questions
Does being in a sitemap mean a page is indexed?
No. A sitemap is a request to discover and crawl a URL. The URL must still be retrieved and considered for indexing.
Can a page be crawled but not indexed?
Yes. A crawler can retrieve a page and then encounter a noindex directive. The retrieval proves access, not index membership.
Does noindex stop a crawler from reading the page?
No. The crawler generally has to read the page to discover a directive placed in the page content. Noindex controls indexing rather than making the page impossible to fetch.
If noindex-checker does not find noindex, is the page definitely indexed?
No. The check only addresses the directive it can inspect. The page may still be undiscovered, uncrawled by the search engine, canonicalized elsewhere, or excluded for other reasons.
Does sitemap-generator prove that a search engine has seen my sitemap?
No. It generates sitemap information from the site you provide. It does not confirm that a sitemap file has been submitted or processed by a search engine.
Can google-search-preview confirm an index listing?
No. It lets you review the page title, meta description, and URL that would appear in a result preview, subject to their displayed character limits. It has no index lookup capability.
What should I check first when a new page is missing from results?
Start by confirming that the exact URL is fetchable and checking for noindex. Then determine whether the URL appears in a generated sitemap page list. Finally, review the result text separately. This sequence distinguishes explicit exclusion from poor discoverability and from presentation issues.
Does removing noindex guarantee immediate indexing?
No. It removes an explicit reason for exclusion, but the search engine still needs to discover and crawl the URL and decide to retain it. The tools can verify the directive and sitemap structure, but they cannot guarantee a search engine's later decision or timing.