web dev//SEO//robots.txt

robots.txt is a plain text file at the site root (`/robots.txt`) that tells crawlers which paths they may or may not access.

robots.txt is a plain text file at the site root (/robots.txt) that tells crawlers which paths they may or may not access.

Its directives are User-agent (which crawler), Allow and Disallow (which paths), Crawl-delay (a politeness throttle) and Sitemap (a pointer to the sitemap). It is a gentleman's agreement rather than access control: well-behaved bots (Googlebot, GPTBot, ClaudeBot) respect it, malicious bots ignore it, and it only shapes what a cooperative scheduler puts in its queue.

AI crawlers can be blocked selectively while search engines stay allowed. User-agent: GPTBot / Disallow: / keeps OpenAI's crawler from training on a site's content.

Blocking crawlers from CSS and JS files makes the page unrenderable for Googlebot, which needs to render the page to index it.

The file itself is public, so anyone can read a site's robots.txt to see what it is hiding.

The protocol behind it is RFC 9309 (2022), which fixes the matching rules.

A warning26 · mar 08Read noteDo not use robots.txt for security. It is a request that malicious bots simply ignore, and the file is public, so listing a path there points straight at what it was meant to hide.