web dev//crawling//indexing

Transforming and storing content so it can be retrieved efficiently later through searches. It is not "understanding the page"; it is putting it in a library with a useful catalog.

Transforming and storing content so it can be retrieved efficiently later through searches. It is not "understanding the page"; it is putting it in a library with a useful catalog.

The index is a data structure (an inverted index from terms to documents, plus signals) built from what parsing produced. It answers "which documents mention this" in milliseconds, which no pile of fetched pages can.

Crawler and indexer are different jobs: the crawler gets documents, the indexer prepares them to be found. The chain is crawler, then pages, then indexer, then index, then the search engine.

Whether a page may enter the index at all is indexability; which of several copies represents it is canonical. A page can be crawled and never indexed.