web dev//SEO//robots.txt//RFC 9309

RFC 9309 is the standard (2022) that formalizes the Robots Exclusion Protocol, which says how crawlers should interpret a robots.txt. The protocol is a norm of courtesy, with no lock behind it.

RFC 9309 is the standard (2022) that formalizes the Robots Exclusion Protocol, which says how crawlers should interpret a robots.txt. The protocol is a norm of courtesy, with no lock behind it.

The sign that says no advertising, thank you.

RFC 9309 standardizes that sign for robots.

It fixes what had been folklore since 1994: where the file lives, how User-agent groups match, how Allow and Disallow paths are resolved (longest match wins), the size limits, and how to behave when the file is unreachable. A crawler that ignores it is rude, and nothing stops it. The standard itself says its rules are no form of access control: it is a sign on an open door and authentication is the lock, and confusing the two is how private pages end up in public indexes.