web dev//crawling//discovery

The moment a system learns that a URL exists. Nothing has been visited yet; an address has simply been written down.

The moment a system learns that a URL exists. Nothing has been visited yet; an address has simply been written down.

Sources of discovery: a link on a page already fetched, an entry in a sitemap or an RSS feed, a redirect, a submission, a URL seen in another dataset.

Discovery is the entry point of the crawling pipeline: a discovered URL goes to the URL frontier and waits for the scheduler to decide whether and when to fetch it.

Knowing a URL, being allowed to crawl it and being able to index it are three different things (indexability).