web dev//crawling//search engine//ranking

Ranking is the ordering of results or URLs by some estimate of relevance or priority, which decides which of a hundred possible things deserve attention first.

Ranking is the ordering of results or URLs by some estimate of relevance or priority, which decides which of a hundred possible things deserve attention first.

Ranking happens twice in a search engine: once in the scheduler, to decide which URLs to fetch first, and once at query time, to order the documents the index returns.

The classic link-based signal, PageRank, treats the web as a graph and scores a page by the probability that a random surfer following links ends up on it: the stationary distribution of that random walk, with a teleport term that makes the chain irreducible and aperiodic so the distribution exists and is unique. Text relevance, freshness, user signals and hundreds of other features are layered on top; the exact function is the engine's secret.

Anything designed to manipulate that estimate rather than to earn it is spam.