web dev//crawling//fetching

Fetching is the step of crawling that actually makes the HTTP request and downloads the resource.

Fetching is the step of crawling that actually makes the HTTP request and downloads the resource.

If crawling decides to go to a house, fetching is knocking on the door and taking what is handed over.

One fetch is one request over HTTPS: headers, status code, body. The crawler identifies itself with a User-Agent; the server may answer with content, a redirect, an error, or a login wall (authentication).

What comes back is raw: an HTML string, JSON, XML, a PDF. It has not been understood yet; that is parsing. If the page builds itself with JavaScript, the fetched HTML may be an empty shell, which is where the rendering crawler comes in.

Fetching is the only stage of crawling that touches the network, so it is where politeness, rate limits and blocking happen.