Reddit to restrict the Internet Archive from indexing it


This is AI generated summarization, which may have errors. For context, always refer to the full article.

Reddit spokesperson Tim Rathschmidt says the Internet Archive ‘provides a service to the open web, but we’ve been made aware of instances where AI companies violate platform policies, including ours, and scrape data from the Wayback Machine’

MANILA, Philippines – Reddit will block the nonprofit organization Internet Archive from indexing most of the site after it called out AI companies using the Internet Archive’s Wayback Machine to scrape content from Reddit.

According to an report from The Verge on Monday, August 11, the Wayback Machine will no longer be able to crawl post detail pages, comments, or profiles, and will only be able to index the Reddit.com homepage. As a result, the Internet Archive will only be able to archive insights into which news headlines and posts were most popular on a given day, but not much else.

Reddit spokesperson Tim Rathschmidt, in a statement, said the Internet Archive “provides a service to the open web, but we’ve been made aware of instances where AI companies violate platform policies, including ours, and scrape data from the Wayback Machine.”

Calling the block a bid to protect Reddit users, Rathschmidt said of the Wayback Machine, “Until they’re able to defend their site and comply with platform policies… we’re limiting some of their access to Reddit data to protect redditors.”

The news follows moves to effectively sell access to Reddit’s content, alongside blocking who can archive the data on the site. In April 2023, Reddit announced plans to make companies pay up to use its data with AI. Google struck a deal with Reddit into adding its data into search and AI results in February 2024, and OpenAI made its own deal in May 2024 to bring Reddit content to ChatGPT. In June 2025, Reddit sued Anthropic for allegedly scraping its data without permission.

Mark Graham, director of the Wayback Machine, meanwhile told The Verge, “We have a longstanding relationship with Reddit and continue to have ongoing discussions about this matter.” – Rappler.com

Leave a Reply

Your email address will not be published. Required fields are marked *