According to Shadowy Visitors founder Gavin King, most major AI agents still adhere to the robots.txt file. “It’s pretty consistent,” he says. But not all site owners have the time or know-how to keep their robots.txt files up to date. And even when they do, some bots bypass the file’s directives: “They try to hide traffic.”
Prince says Cloudflare’s bot blocking won’t be a mandate this type of bad actor can ignore. “Robots.txt is like putting up a ‘no trespassing’ sign,” he says. “It’s like a physical wall patrolled by armed guards.” Just as it flags other types of suspicious online behavior, such as price-scraping bots used to illegally monitor prices, the company has built processes to detect even the most carefully disguised AI crawlers.
Cloudflare also teases an upcoming marketplace where customers can negotiate scraping terms with AI companies, whether that’s paying to employ content or exchanging AI credits for scraping. “We don’t really care what the transaction is, but we think there needs to be some way to provide value back to the original content creators,” Prince says. “The compensation doesn’t have to be dollars. It could be a credit or a credit. It could be a variety of things.”
No specific launch date has been given for the marketplace, but even if it does happen this year, it will join an increasingly crowded group of projects aimed at facilitating licensing deals and other permissions arrangements between AI companies, publishers, platforms, and other websites.
What do AI companies think? “We’ve talked to most of them, and their response has ranged from ‘That makes sense and we’re open’ to ‘Go to hell,’” says Prince. (Though he declined to name names.)
The project moved along rather quickly. Prince cites a conversation with Atlantic CEO (and former WIRED editor-in-chief) Nick Thompson as the inspiration for the project; Thompson talked about how many different publishers had encountered hidden Web scrapers. “I love that he’s doing this,” Thompson says. If even major news organizations are struggling with the influx of scrapers, Prince reasoned, independent bloggers and site owners will have an even harder time.
Cloudflare has been a leading network security company for years, providing much of the infrastructure that sustains the web. It has historically remained as neutral as possible about the content of the websites it services; on the sporadic occasions when it has made exceptions to that rule, Prince has emphasized that he does not want Cloudflare to be the arbiter of what is allowed on the web.
Here, he sees Cloudflare as uniquely positioned to take a stand. “The path we’re on is not sustainable,” Prince says. “We hope we can be a part of making sure people get paid for their work.”