this post was submitted on 21 Mar 2025
143 points (98.6% liked)

Linux

6585 readers
299 users here now

A community for everything relating to the GNU/Linux operating system

Also check out:

Original icon base courtesy of lewing@isc.tamu.edu and The GIMP

founded 2 years ago
MODERATORS
Ategon@programming.dev
anzo@programming.dev
dwraf_of_ignorance@programming.dev
143
FOSS infrastructure is under attack by AI companies (thelibre.news)
submitted 15 hours ago* (last edited 15 hours ago) by throws_lemy@lemmy.nz to c/linux@programming.dev
25 comments fedilink hide all child comments
 

LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.

all 26 comments
sorted by: hot top controversial new old
[–] grrgyle@slrpnk.net 50 points 14 hours ago* (last edited 14 hours ago) (1 children)

Wow that was a frustrating read. I dd not know it was quite that bad. Just to highlight one quote

they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. [...] If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

[–] jatone@lemmy.dbzer0.com 21 points 14 hours ago (2 children)

the solution here is to require logins. thems the breaks unfortunately. it'll eventually pass as the novelty wears off.

[–] nao@sh.itjust.works 8 points 14 hours ago (2 children)

Next you'll have to invest in preventing automated signups

[–] hisao@ani.social 3 points 14 hours ago (1 children)

Signups in most platforms are quite hard. Straight up give your phone and do SMS verification, or at least give email and to register that email you will have to provide phone anyway. Captchas nowadays became so hard that even humans struggle with them and it often takes multiple attempts to get it right.

[–] nao@sh.itjust.works 2 points 4 hours ago (1 children)

provide phone number to look at this foss project's website, not too sure about that

[–] Taleya@aussie.zone 2 points 52 minutes ago

Honestly if any site demands my phone number it can get fucked.

[–] jatone@lemmy.dbzer0.com 2 points 8 hours ago

not really, just tie it with 2fa SMS style and the hurdle is large enough most companies won't bother.

[–] possiblylinux127@lemmy.zip 6 points 11 hours ago (2 children)

Alternative: require a proof of work calculation.

[–] marauding_gibberish142@lemmy.dbzer0.com 2 points 9 hours ago (1 children)

This is exactly what we need to do. You'd think that a FOSS WAF exists out there somewhere that can do this

[–] LiveLM@lemmy.zip 2 points 8 hours ago (2 children)

There is. That screenshot you see in the article is a picture of a brand new one, Anubis

[–] marauding_gibberish142@lemmy.dbzer0.com 3 points 7 hours ago

Yeah I realised that after posting. I think we need a better one to deal with the cases of letting legitimate users in easier though

[–] possiblylinux127@lemmy.zip 1 points 7 hours ago

It kind of sucks but it is the best we have for the moment

[–] ulterno@programming.dev 0 points 1 hour ago

Make them mine a BTC block in the Browser!

^Sorry, I'm low in blood and full of mosquito vomit. That's probably making me think weird stuff.^

[–] hisao@ani.social 18 points 14 hours ago

This is the most crazy read on subject in a while. Most articles just talk about hypothetical issues of tomorrow, while this one actually full of today's problems and even costs of those issues in numbers and hours of pointless extra work. Had no idea it's already this bad.

[–] 4am@lemm.ee 5 points 11 hours ago (1 children)

How much you wanna bet that at least part of this traffic is Microsoft just using other companies infrastructure to mask the fact that it’s them

[–] possiblylinux127@lemmy.zip 2 points 7 hours ago

I doubt it since Microsoft is big enough to be a little more responsible.

What you should be worried about is the fresh college graduates with 200k of venture capital money.

[–] coldsideofyourpillow@lemmy.cafe 4 points 13 hours ago (1 children)

Sometimes, I hate humanity.

[–] luciole@beehaw.org 9 points 11 hours ago

just hate the techbros

[–] marauding_gibberish142@lemmy.dbzer0.com 2 points 9 hours ago (2 children)

I'm perfectly fine with Anubis but I think we need a better algorithm for PoW

[–] possiblylinux127@lemmy.zip 1 points 7 hours ago* (last edited 7 hours ago) (1 children)

Tor has one now

Maybe it can be reused for the clearnet.

[–] marauding_gibberish142@lemmy.dbzer0.com 1 points 7 hours ago (1 children)

You mean the TOR project?

[–] possiblylinux127@lemmy.zip 1 points 7 hours ago* (last edited 7 hours ago) (1 children)

And Tor itself

It is part of the denial of service protection

[–] sudo@programming.dev 2 points 4 hours ago (1 children)

Whats confusing the hell out of me is: why are they bothering to scrape the git blame page? Just download the entire git repo and feed that into your LLM!

9/10 the best solution is to block nonresidential IPs. Residential proxies exist but they're far more expensive than cloud proxies and providers will ask questions. Residential proxies are sketch AF and basically guarded like munitions. Some rookie LLM maker isn't going to figure that out.

Anubis also sounds trivial to beat. If its just crunching numbers and not attempting to fingerprint the browser then its just a case of feeding the page into playwright and moving on.

[–] refalo@programming.dev 2 points 3 hours ago* (last edited 3 hours ago)

I don't like the approach of banning nonresidential IPs. I think it's discriminatory and unfairly blocks out corporate/VPN users and others we might not even be thinking about. I realize there is a bot problem but I wish there was a better solution. Maybe purely proof-of-work solutions will get more popular or something.