web dev//SEO//crawler

A crawler is an automated program, also called a spider or a bot, that systematically browses the web to discover and index content. Crawlers identify themselves through the `User-Agent` HTTP header, which robots.txt uses to grant or deny access crawler by crawler.

A crawler is an automated program, also called a spider or a bot, that systematically browses the web to discover and index content. Crawlers identify themselves through the User-Agent HTTP header, which robots.txt uses to grant or deny access crawler by crawler.

Three generations of crawlers exist, and each reads a different part of a page:

Search engine crawlers (Googlebot, Bingbot) index pages for search results. They are the oldest and the most mature.

Social crawlers (the WhatsApp, Twitter and Discord bots) fetch Open Graph tags to generate link previews. They only read the <head> and do not index the body.

AI crawlers (GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended) fetch content for LLM training data or real-time retrieval and read everything. They are the AEO audience.

The rest of a crawler's life is shaped by a few constraints:

Crawlers that do not execute JavaScript see an empty <div id="root"></div> on a single-page app. The solutions are SSR, static pre-rendering, or edge functions that inject content for bot user-agents.

Search engines allocate a limited crawl budget to each site, so large sites have to prioritize which pages get crawled through the sitemap and internal linking.

Crawler, scraper and indexer name different jobs, and one program can combine them. The crawler decides where to go and traverses URLs; the scraper decides what to take out of the pages; the indexer prepares the documents to be found later. One program can do all three. The stages in order are the crawling pipeline: discovery, frontier, scheduling, fetching, rendering, parsing.

A rendering crawler runs the page's JavaScript before looking, which is the crawler-side answer to the SPA problem above; the site-side answer is SSR.