web dev//crawling

Traversing URLs systematically to discover and retrieve resources, and everything that happens to a page after that. The idea worth memorizing: crawler, parser, scraper and indexer are not four bots; they are four different jobs that one system may combine.

Traversing URLs systematically to discover and retrieve resources, and everything that happens to a page after that. The idea worth memorizing: crawler, parser, scraper and indexer are not four bots; they are four different jobs that one system may combine.

Discovery: "I know it exists."

Frontier: "I might visit it."

Scheduling: "I visit it now."

Fetching: "I download the resource."

Rendering: "I run the JavaScript if needed."

Parsing: "I understand its structure."

Scraping: "I extract specific data" (optional).

Indexing: "I prepare it to be found later" (optional).

Ranking and retrieval: "I decide which result matters."

Agent: "I use that information to reach a goal."

One note per stage: discovery, URL frontier, scheduler, fetching, rendering crawler, parsing, scraping, indexing, and on top of them the search engine with its ranking.

Crawling is exploring a map by following its roads; the program that does it is the crawler. Fetching is knocking on one door. Parsing is reading what they hand you. Scraping is keeping only the dish name and the price. Indexing is filing it in a library with a useful catalog.

The site's side of the same story is SEO: robots.txt, sitemaps, canonical URLs and feeds are the signals a crawler consumes. The reader's side, when the reader has a goal instead of a route, is the web agent.

A browser that acts like a mechanical user is browser automation; it serves crawlers, scrapers and agents alike.