web dev//crawling//URL frontier

The set of known URLs still waiting for a decision about whether they will be visited. The list of places you might go later.

The set of known URLs still waiting for a decision about whether they will be visited. The list of places you might go later.

It grows with every discovery and shrinks with every fetch. On the open web it never empties: the frontier is always larger than what will ever be crawled, so the decision of what to take from it matters more than the list itself.

The scheduler is the doorman of that queue: priority, politeness per host, freshness and crawl budget are its criteria.

A frontier is a queue with a policy, not a to-do list. Deduplication (the same page under several URLs, see canonical) and robots rules (robots.txt) prune it before anything is fetched.