web dev//SEO//indexability

Three different permissions that are easy to confuse: knowing a URL exists is not the same as being able to crawl it, and being able to crawl it is not the same as being able to index it.

Three different permissions that are easy to confuse: knowing a URL exists is not the same as being able to crawl it, and being able to crawl it is not the same as being able to index it.

Known: the URL has been discovered.

Crawlable: the crawler can access the resource and retrieve it. "I can get in."

Indexable: the content may be incorporated into a search index. "I can file it in my catalog."

robots.txt governs crawlability. A noindex tag or header governs indexability, and it only works if the page can be crawled: a page blocked in robots.txt cannot show its noindex, so it may stay in the index from links alone.

A page can be crawled and never indexed (thin, duplicate, low priority); a page can be indexed without being fetched recently. The stages are discovery, fetching and indexing.