web dev//SEO//indexability

Indexability is the permission for a page's content to be filed in a search index, the last of three permissions that are easy to confuse: knowing a URL exists is not the same as being able to crawl it, and being able to crawl it is not the same as being able to index it. A URL passes three gates before it can enter an index: it must be known (discovered), crawlable (the crawler can reach the resource and retrieve it) and indexable (the content may be filed in a search index).

Indexability is the permission for a page's content to be filed in a search index, the last of three permissions that are easy to confuse: knowing a URL exists is not the same as being able to crawl it, and being able to crawl it is not the same as being able to index it. A URL passes three gates before it can enter an index: it must be known (discovered), crawlable (the crawler can reach the resource and retrieve it) and indexable (the content may be filed in a search index).

robots.txt governs crawlability. A noindex tag or header governs indexability, and it only works if the page can be crawled: a page blocked in robots.txt cannot show its noindex, so it may stay in the index from links alone.

A page can be crawled and never indexed (thin, duplicate, low priority); a page can be indexed without being fetched recently. The stages are discovery, fetching and indexing.