web dev//crawling//URL frontier

The URL frontier is the set of known URLs still waiting for a decision about whether they will be visited: the list of places a crawler might go later.

The URL frontier is the set of known URLs still waiting for a decision about whether they will be visited: the list of places a crawler might go later.

It grows with every discovery and shrinks with every fetch. On the open web it never empties: the frontier is always larger than what will ever be crawled, so the decision of what to take from it matters more than the list itself.

The scheduler is the doorman of that queue: priority, politeness per host, freshness and crawl budget are its criteria.

A frontier is a queue with a policy, not a to-do list. Deduplication (the same page under several URLs, see canonical) and robots rules (robots.txt) prune it before anything is fetched. A crawler prunes by rules (depth, host, freshness); an agent prunes the same list by judgment, asking which fetch would change its answer most (value of information).