Anthropic’s robot ignores website scraping rules

The ClaudeBot robot, which Anthropic uses to collect training data for AI models like Claude, attacked the iFixit website nearly a million times in 24 hours, apparently violating the repair company’s terms of service.

“If any of these requests were to access our terms of service, you would be told that use of our content is expressly prohibited. But don’t ask me, ask Claude!” he said iFixit CEO Kyle Wiens on Xby posting images of the Anthropic chatbot admitting that iFixit’s content is unavailable. “Not only are you taking our content without paying, you’re also tying up our DevOps resources. If you want to talk about licensing our content for commercial use, we’re here.”

“The indexing speed was so fast that it set off all our alarms and accelerated the work of our DevOps team,” Wiens says. Edge“iFixit gets a lot of traffic. Being one of the top sites on the internet, we’re very familiar with crawlers and bots. We handle that load with no problem, but this was an anomaly.”

iFixit Terms of Use states that “reproduction, copying, or distribution” of any content from the website is “strictly prohibited without the prior express written consent” of the company, with a specific focus on “training a machine learning or artificial intelligence model.” When Anthropic was asked about it by 404 Mediahowever, the artificial intelligence company provided a link to the FAQ page from which it follows that his robot can only be blocked via the robots.txt file extension.

Wiens says iFixit has since added index delay extension to the robots.txt file. “Based on our logs, they stopped after we added it to the robots.txt file,” Wiens says.

“We respect the robots.txt file, and our robot respected that signal when iFixit implemented it,” Anthropic spokeswoman Jennifer Martinez said. Edge.

iFixit doesn’t seem to be alone, as Read the Docs co-founder Eric Holscher and Freelancer.com CEO Matt Barrie said in Wiens’ thread that their site was also aggressively scraped by the Anthropic crawler. This doesn’t seem to be modern behavior for ClaudeBot either, as couple months Reddit threads reporting a dramatic augment in Anthropic web scraping. In April of this year, Linux Mint Internet Forum determined that the site outage was caused by an overload caused by ClaudeBot scraping data.

Updates, July 25:Added statements by Wiens and Anthropic.