web dev//crawling//scraping

Extracting from the content the specific data you care about. Parsing understands the menu; scraping says "keep only the dish name and the price".

Extracting from the content the specific data you care about. Parsing understands the menu; scraping says "keep only the dish name and the price".

HTML, then parse, then an understandable structure, then scrape, then the concrete data.

A scraper decides what to take out of pages; a crawler decides where to go. One program can do both, but they are different decisions, and a scraper often runs on a single known page with no crawling at all.

Scraping is brittle by nature: it depends on the page's layout, which the site can change any day. When a site offers an API, asking the warehouse directly beats reading the shelf labels.

If the data only exists after JavaScript runs, scraping needs a rendered page (rendering crawler, browser automation). Whether scraping is welcome is a question of terms of service and robots rules, not of technical possibility (robots.txt).