web dev//crawling

Crawling is the systematic traversal of URLs to discover and retrieve resources, the job that feeds pages to search engines, scrapers and web agents, and the word also covers everything that happens to a page after that. Crawler, parser, scraper and indexer are four different jobs that one system may combine, and the program that does the traversing is the crawler.

Crawling is the systematic traversal of URLs to discover and retrieve resources, the job that feeds pages to search engines, scrapers and web agents, and the word also covers everything that happens to a page after that. Crawler, parser, scraper and indexer are four different jobs that one system may combine, and the program that does the traversing is the crawler.

A crawler takes each URL through the same stages, one at a time:

1Discoverythe URL is known2Frontierwaiting its turn3Schedulingits turn comes4Fetchingdownloaded5Renderingscripts run, if needed6Parsingstructure read7Scrapingdata pulled out, optional8Indexingmade findable, optional9Ranking and retrievalchosen for a query10Agentused toward a goal

Crawling is exploring a map by following its roads.

Fetching is knocking on one door. Parsing is reading what they hand you. Scraping is keeping only the dish name and the price. Indexing is filing it in a library with a useful catalog.

Each stage has its own note: discovery, URL frontier, scheduler, fetching, rendering crawler, parsing, scraping, indexing, and on top of them the search engine with its ranking.

The site's side of the same story is SEO: robots.txt, sitemaps, canonical URLs and feeds are the signals a crawler consumes. The reader's side, when the reader has a goal instead of a route, is the web agent.

A browser that acts like a mechanical user is browser automation; it serves crawlers, scrapers and agents alike.