Reddit Blocks Major Search Engines and AI Bots, Except Paying Ones

Google is currently the only mainstream search engine displaying recent results from Reddit, according to a report by 404 Media. This development is part of Reddit's broader effort to control web crawlers and protect its data.

Recent Changes and Current Policy

In recent weeks, Reddit has started blocking search engines from accessing recent posts and comments unless they pay for the privilege. While Google has a $60 million deal allowing it to train AI models on Reddit content, other search engines like Bing and DuckDuckGo are excluded.

Reddit spokesperson Tim Rathschmidt clarified to The Verge, “This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines but have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”

Implementation and Compliance

To enforce its anti-scraping policy, Reddit updated its robots.txt file last month, instructing web crawlers on whether they can access the site. Ben Lee, Reddit’s chief legal officer, emphasized that this update signals to those without an agreement that they should not access Reddit data.

Microsoft spokesperson Caitlin Roulston stated to The Verge, “Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models.” Consequently, Bing stopped crawling Reddit following the platform’s robots.txt update on July 1st.

Implications and Context

Blocking major search engines is a bold move for Reddit but not entirely surprising. Over the past year, Reddit has become more protective of its data, aiming to open new revenue sources and satisfy investors. The platform recently increased API costs for some third-party developers and reportedly threatened to cut off Google’s access if it continued using Reddit data for free AI training.

With AI-generated content becoming more prevalent, finding genuine human-written information is increasingly important. Many users, including myself, add "Reddit" to searches to get human answers, making it frustrating that this will now only work on Google or search engines relying on it, especially for those who prefer Bing.