web dev//SEO//robots.txt

- A plain text file at the site root (`/robots.txt`) that tells crawlers which paths they may or may not access.

A plain text file at the site root (/robots.txt) that tells crawlers which paths they may or may not access.

Directives: User-agent (which crawler), Allow / Disallow (which paths), Crawl-delay (politeness throttle), Sitemap (pointer to sitemap).

It is a gentleman's agreement, not access control. Malicious bots ignore it. Well-behaved bots (Googlebot, GPTBot, ClaudeBot) respect it.

AI-era relevance: you can selectively block AI crawlers while allowing search engines. User-agent: GPTBot / Disallow: / blocks OpenAI's crawler from training on your content.

Common mistake: blocking crawlers from CSS/JS files makes your page unrenderable for Googlebot (it needs to render the page to index it).

The file itself is public. Anyone can read your robots.txt to see what you are hiding. Do not use it for security.

The protocol behind it is RFC 9309 (2022), which fixes the matching rules and states plainly that this is not an authorization mechanism. The intuition: "do not enter" written on an open door. What must stay private needs authentication; robots.txt only shapes what a cooperative scheduler puts in its queue.