web dev//crawling//discovery

Discovery is the moment a system learns that a URL exists. Nothing has been visited yet; an address has simply been written down.

Discovery is the moment a system learns that a URL exists. Nothing has been visited yet; an address has simply been written down.

Sources of discovery: a link on a page already fetched, an entry in a sitemap or an RSS feed, a redirect, a submission, a URL seen in another dataset. Links give discovery as a side effect of crawling; a sitemap gives it without crawling, which is why it matters exactly for the pages no server-rendered link points to.

Discovery is the entry point of the crawling pipeline: a discovered URL goes to the URL frontier and waits for the scheduler to decide whether and when to fetch it.

Knowing a URL, being allowed to crawl it and being able to index it are three different things (indexability).