web dev//crawling//rendering crawler

A rendering crawler is a crawler that, besides fetching, uses a browser or renderer to execute the page's JavaScript before analyzing it. A normal crawler looks at the package that arrived; a rendering crawler opens the package, assembles everything, and only then inspects.

A rendering crawler is a crawler that, besides fetching, uses a browser or renderer to execute the page's JavaScript before analyzing it. A normal crawler looks at the package that arrived; a rendering crawler opens the package, assembles everything, and only then inspects.

It exists because of client-side rendering: the fetched HTML can be a near-empty shell, and the content only appears in the DOM after scripts run and data is fetched. Googlebot renders; many simpler bots and most LLM fetchers do not.

Rendering is expensive (a browser per page, seconds instead of milliseconds), so it is rationed: a rendering queue behind the fetch queue, often days later. That delay is a reason sites move to server-side rendering rather than hoping the crawler renders: SSR moves the work of building the page to the site, a rendering crawler moves it to the crawler, and only the first is under the site's control (rendering).

The engine underneath is browser automation.