web dev//crawling//scheduler

The scheduler is the component of a crawler that decides which URL of the URL frontier to visit, when, and with what priority, as the doorman of the queue.

The scheduler is the component of a crawler that decides which URL of the URL frontier to visit, when, and with what priority, as the doorman of the queue.

Its inputs are priority signals (importance, freshness, how often a page changes), politeness constraints per host (rate limits, the Crawl-delay directive of robots.txt) and a finite budget: a search engine allocates limited crawl resources per site, so large sites live or die by which pages the scheduler picks first. Politeness works only because the scheduler chooses to obey: robots.txt is written by the site and enforced here, where Disallow removes URLs from the queue and Crawl-delay slows the rate. Scheduling is where a known URL becomes a URL visited now, and the next stage is fetching.