web dev//crawling//browser automation

Browser automation is software that controls a real browser or its equivalent: it executes JavaScript, clicks, types, waits and scrolls. Instead of simply requesting the HTML, it acts like a mechanical user.

Browser automation is software that controls a real browser or its equivalent: it executes JavaScript, clicks, types, waits and scrolls. Instead of simply requesting the HTML, it acts like a mechanical user.

It serves three masters. A rendering crawler uses it to see the page a human sees. A scraper uses it when the data appears only after interaction (scraping). A web agent uses it as a tool to act, not just to read: the most general tool an agent can hold, where every page becomes both an observation and a surface to act on (tool use describes the decision; the automation is the hand).

The cost is a browser per session: memory, seconds per page, and a fingerprint sites can detect and block. Plain fetching is a thousand times cheaper and is the right default whenever the HTML already carries the content (SSR).

Because it runs the site's JavaScript, it is also a security boundary: the automation sees cookies, forms and logged-in state, which is why an agent driving a browser needs the same care as a person with the same access (authentication).