web dev//crawling//fetching

Actually making the HTTP request and downloading the resource. Crawling says "let's go to that house"; fetching is knocking on the door and taking what they hand you.

Actually making the HTTP request and downloading the resource. Crawling says "let's go to that house"; fetching is knocking on the door and taking what they hand you.

One fetch is one request over HTTPS: headers, status code, body. The crawler identifies itself with a User-Agent; the server may answer with content, a redirect, an error, or a login wall (authentication).

What comes back is raw: an HTML string, JSON, XML, a PDF. It has not been understood yet; that is parsing. If the page builds itself with JavaScript, the fetched HTML may be an empty shell, which is where the rendering crawler comes in.

Fetching is the only stage of crawling that touches the network, so it is where politeness, rate limits and blocking happen.