Fedibridge

795 readers

3 users here now

A community to organize and discuss the growth of the fediverse as a whole

founded 6 months ago

MODERATORS

submitted 2 days ago* (last edited 2 days ago) by Sunshine@piefed.ca to c/fedibridge@lemmy.dbzer0.com

7 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] riskable@programming.dev 18 points 2 days ago

So let me get this straight: Instead of wasting Reddit's bandwidth, AI companies have been scraping the wayback machine.

Because of this, Reddit is going to block the wayback machine from crawling it's site which will ensure the AI companies crawl Reddit, directly.

...Because if you think they're suddenly going to stop crawling reddit—robots.txt be damned—you're dreaming.