web dev//crawling//scheduler

The component that decides which URL of the URL frontier to visit, when, and with what priority. The doorman of the queue.

The component that decides which URL of the URL frontier to visit, when, and with what priority. The doorman of the queue.

Its inputs are priority signals (importance, freshness, how often a page changes), politeness constraints per host (rate limits, the Crawl-delay directive of robots.txt) and a finite budget: a search engine allocates limited crawl resources per site, so large sites live or die by which pages the scheduler picks first.

Scheduling is where "known" becomes "visited now". The next stage is fetching.