web dev//crawling//indexing

Indexing is the stage of a crawl pipeline that transforms and stores content so that it can be retrieved efficiently later through searches, and it is what makes a fetched page findable by a search engine. It puts the page in a library with a useful catalog, without understanding it.

Indexing is the stage of a crawl pipeline that transforms and stores content so that it can be retrieved efficiently later through searches, and it is what makes a fetched page findable by a search engine. It puts the page in a library with a useful catalog, without understanding it.

1Crawler2Pages3Indexer4Index5Search engine

The index is a data structure (an inverted index from terms to documents, plus signals) built from what parsing produced. It finds which documents mention a term in milliseconds, which no pile of fetched pages can.

Whether a page may enter the index at all is indexability; which of several copies represents it is canonical.

A page can be crawled and never indexed. The crawler collects documents and the indexer prepares them for retrieval, so a site can be fully crawled and absent from search if the second job never happens; crawl budget and index budget are separate scarcities.