web dev//crawling//rendering crawler

A crawler that, besides fetching, uses a browser or renderer to execute the page's JavaScript before analyzing it. A normal crawler looks at the package that arrived; a rendering crawler opens the package, assembles everything, and only then inspects.

A crawler that, besides fetching, uses a browser or renderer to execute the page's JavaScript before analyzing it. A normal crawler looks at the package that arrived; a rendering crawler opens the package, assembles everything, and only then inspects.

It exists because of client-side rendering: the fetched HTML can be a near-empty shell, and the content only appears in the DOM after scripts run and data is fetched. Googlebot renders; many simpler bots and most LLM fetchers do not.

Rendering is expensive (a browser per page, seconds instead of milliseconds), so it is rationed: a rendering queue behind the fetch queue, often days later. That delay is a reason sites move to server-side rendering rather than hoping the crawler renders.

The engine underneath is browser automation.