web dev//crawling//browser automation

Software that controls a real browser or its equivalent: executes JavaScript, clicks, types, waits, scrolls. Instead of simply requesting the HTML, it acts like a mechanical user.

Software that controls a real browser or its equivalent: executes JavaScript, clicks, types, waits, scrolls. Instead of simply requesting the HTML, it acts like a mechanical user.

It serves three masters. A rendering crawler uses it to see the page a human sees. A scraper uses it when the data appears only after interaction (scraping). An web agent uses it as a tool to act, not just to read.

The cost is a browser per session: memory, seconds per page, and a fingerprint sites can detect and block. Plain fetching is a thousand times cheaper and is the right default whenever the HTML already carries the content (SSR).

Because it runs the site's JavaScript, it is also a security boundary: the automation sees cookies, forms and logged-in state, which is why an agent driving a browser needs the same care as a person with the same access (authentication).