web dev//crawling//scraping

Scraping is the extraction of the specific data a program cares about from a page's content, the selective step that comes after parsing: where parsing understands the whole menu, scraping keeps only the dish name and the price. You cannot scrape what you have not parsed, and parsing alone gives you no data.

Scraping is the extraction of the specific data a program cares about from a page's content, the selective step that comes after parsing: where parsing understands the whole menu, scraping keeps only the dish name and the price. You cannot scrape what you have not parsed, and parsing alone gives you no data.

Getting from the page to the data takes steps, each with its own job.

1Take the HTML2Parse itan understandable structure3Scrape itthe concrete data

A scraper decides what to take out of pages; a crawler decides where to go. One program can do both, but they are different decisions, and a scraper often runs on a single known page with no crawling at all.

Scraping is brittle by nature: it depends on the page's layout, which the site can change any day. Scraping reads the labels in the shop and breaks when the shelves are rearranged; an API asks the warehouse for product 123 in JSON and breaks only when its contract changes. Same data, opposite fragility.

If the data only exists after JavaScript runs, scraping needs a rendered page (rendering crawler, browser automation). Whether scraping is welcome is a question of terms of service and robots rules, not of technical possibility (robots.txt).