Reddit Now Blocks Major Search Engines and AI Bots — Except Those That Pay

Reddit is stepping up its crackdown on crawlers. In the past few weeks, Reddit has begun blocking search engines from displaying recent posts and comments unless the search engine pays, according to the report 404 Media.

Currently, Google is the only popular search engine that shows the latest search results for Reddit posts using the “site:reddit.com” trick404 Media reports. That rules out Bing, DuckDuckGo, and other alternatives — and that’s likely because Google has a $60 million deal that allows the company to train its AI models on Reddit content.

“This is completely unrelated to our recent work with Google,” Reddit spokesman Tim Rathschmidt said in a statement to Edge“We’ve had discussions with a number of search engines. We haven’t been able to reach an agreement with all of them because some are unable or unwilling to make enforceable promises about how they will use Reddit content, including for AI purposes.”

It’s a bold move for a massive site like Reddit to block some of the most popular search engines, but it’s not surprising. Over the past year, Reddit has become more protective of its data as it looks to open up another revenue stream and keep recent investors content. After making its API more costly for some third-party developers, Reddit reportedly threatened to cut Google off if it didn’t stop using the platform’s data to train AI for free.

Last month, in an effort to enforce its anti-scraping policy, Reddit updated the site’s robots.txt file, which tells crawlers whether they can access the site. “That’s a signal to those who don’t have a contract with us that they shouldn’t be accessing Reddit data,” Reddit general counsel Ben Lee told my colleague Alex Heath in Command line.

As AI chatbots fill the internet with questionable content, finding things written by another human has never been more crucial. I, like many others, have started adding “Reddit” to many of my searches to get human answers, and it’s pretty frustrating to know that now I’ll only be able to do that on Google (or search engines that rely on it) — especially when I do a lot of my searches on Bing.

Update, July 24: Added statement from Reddit.