So I was trying to help someone in /rs/ find something that I knew was posted before the crash so I directed them to the Wayback Machine, and when they said they couldn't find it I went looking myself and discovered that the robots.txt for the site was recently changed to disallow any specific threads or anything beyond the first page of each board Why was this implemented anyway? Was it to assuage fears of artists finding leaked paysite content through Google or something? In any case, there are many things that were here pre-crash that were still available through caches, and some things were ONLY available here and are essentially lost now that the /board/ directory is blocked You can see why this would be a problem for people looking for something relatively rare I know this isn't a bug, even though I don't know why it was ever implemented, but I see no reason to believe archive.org just deletes all its backup data after crawlers are disallowed so it's most likely all still there At the very least I'd like to hear back from a mod
|