web dev//crawling//search engine

A system that combines crawling, processing, indexing, ranking and retrieval. It is not simply a crawler: the crawler is one of its workers.

A system that combines crawling, processing, indexing, ranking and retrieval. It is not simply a crawler: the crawler is one of its workers.

The workers, in order: the crawler gathers pages through the crawling pipeline, the indexer files them, and at query time ranking decides which of the millions of matches deserve attention first.

What the site can influence is the input to those workers: what is crawlable, what is indexable, which URL is canonical, what the structured data says (SEO). What it cannot see is the ranking function.

The AI-era variant answers instead of listing: an answer engine reads the same index and cites a few sources, which changes what "being found" means (AEO).