web dev//crawling//search engine

A search engine is a system that combines crawling, processing, indexing, ranking and retrieval. It is more than a crawler: the crawler is one of its workers, and the workers act in order.

A search engine is a system that combines crawling, processing, indexing, ranking and retrieval. It is more than a crawler: the crawler is one of its workers, and the workers act in order.

1The crawler gathers pagesthrough the crawling pipeline2The indexer files them3At query time ranking orders the matches

What the site can influence is the input to those workers: what is crawlable, what is indexable, which URL is canonical, what the structured data says (SEO). What it cannot see is the ranking function. The AI-era variant answers instead of listing: an answer engine reads the same index and cites a few sources, which changes what it means for a page to be found (AEO).