- AiNews.com
- Posts
- Anthropic’s ClaudeBot Violates Website Anti-AI Scraping Policies
Anthropic’s ClaudeBot Violates Website Anti-AI Scraping Policies
![A screenshot of a social media post by iFixit CEO Kyle Wiens discussing the aggressive web scraping by Anthropic’s ClaudeBot. The post highlights the impact on iFixit's servers and the violation of their Terms of Use. The interface shows Kyle Wiens' profile picture, the text of his post, and responses from other users. The background is a clean and modern social media interface, emphasizing the issue of AI scraping policies](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/76538a64-ef86-4ede-9cd0-e523b98d3dcf/Anthropic_s_ClaudeBot_Violates_Website_Anti-AI_Scraping_Policies.png?t=1722014802)
Anthropic’s ClaudeBot Violates Website Anti-AI Scraping Policies
Anthropic’s ClaudeBot web crawler has come under fire for ignoring websites' anti-AI scraping policies, causing significant issues for site owners like iFixit.
iFixit CEO's Complaint
iFixit CEO Kyle Wiens revealed that ClaudeBot hit their website's servers nearly a million times in just 24 hours, violating the company's Terms of Use. Wiens took to X to express his frustration, stating, “If any of those requests accessed our terms of service, they would have told you that use of our content is expressly forbidden. But don’t ask me, ask Claude!” He posted images showing Anthropic’s chatbot acknowledging that iFixit’s content was off-limits and added, “You’re not only taking our content without paying, you’re tying up our devops resources. If you want to have a conversation about licensing our content for commercial use, we’re right here.”
Impact on iFixit
Wiens described the situation as an anomaly, stating, “The rate of crawling was so high that it set off all our alarms and spun up our devops team.” Despite iFixit's familiarity with handling web crawlers due to its high traffic, the aggressive scraping by ClaudeBot was unprecedented.
Terms of Use Violations
iFixit’s Terms of Use explicitly prohibit the reproduction, copying, or distribution of any content from their website without prior written permission, specifically including “training a machine learning or AI model.” When questioned, Anthropic referred to an FAQ page stating their crawler can be blocked via a robots.txt file extension.
Measures Taken
Wiens confirmed that iFixit added the crawl-delay extension to its robots.txt, which stopped the scraping. “Based on our logs, they did stop after we added it to the robots.txt,” Wiens says. Anthropic spokesperson Jennifer Martinez stated, “We respect robots.txt and our crawler respected that signal when iFixit implemented it.”
Wider Issues with AI Scraping
iFixit is not alone in this experience. Read the Docs co-founder Eric Holscher and Freelancer.com CEO Matt Barrie reported similar issues with Anthropic’s crawler. ClaudeBot’s aggressive scraping behavior has been a concern for months, with several reports on Reddit and an incident involving the Linux Mint web forum in April attributing site outages to ClaudeBot’s activities.
Challenges with robots.txt
Disallowing crawlers via robots.txt is the common opt-out method for AI companies like OpenAI. However, this method lacks flexibility, preventing website owners from specifying what scraping is permissible. Another AI company, Perplexity, is known to ignore robots.txt exclusions entirely. Despite its limitations, the robots.txt file remains one of the few tools available for companies to protect their data from AI training materials, as seen in Reddit's recent crackdown on web crawlers.