this post was submitted on 21 Aug 2025
205 points (89.6% liked)

Selfhosted

50759 readers
739 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
205
Why are anime catgirls blocking my access to the Linux kernel? (lock.cmpxchg8b.com)
submitted 3 days ago by tofu@lemmy.nocturnal.garden to c/selfhosted@lemmy.world
94 comments fedilink hide all child comments
 

Some thoughts on how useful Anubis really is. Combined with comments I read elsewhere about scrapers starting to solve the challenges, I'm afraid Anubis will be outdated soon and we need something else.

top 50 comments
sorted by: hot top controversial new old
[–] rtxn@lemmy.world 191 points 3 days ago* (last edited 3 days ago) (4 children)

The current version of Anubis was made as a quick "good enough" solution to an emergency. The article is very enthusiastic about explaining why it shouldn't work, but completely glosses over the fact that it has worked, at least to an extent where deploying it and maybe inconveniencing some users is preferable to having the entire web server choked out by a flood of indiscriminate scraper requests.

The purpose is to reduce the flood to a manageable level, not to block every single scraper request.

[–] poVoq@slrpnk.net 87 points 3 days ago* (last edited 3 days ago) (11 children)

And it was/is for sure the lesser evil compared to what most others did: put the site behind Cloudflare.

I feel people that complain about Anubis have never had their server overheat and shut down on an almost daily basis because of AI scrapers 🤦

[–] tofu@lemmy.nocturnal.garden 16 points 2 days ago* (last edited 2 days ago) (1 children)

Yeah, I'm just wondering what's going to follow. I just hope everything isn't going to need to go behind an authwall.

[–] rtxn@lemmy.world 33 points 2 days ago (2 children)

The developer is working on upgrades and better tools. https://xeiaso.net/blog/2025/avoiding-becoming-peg-dependency/

[–] grysbok@lemmy.sdf.org 21 points 2 days ago (1 children)

I'll say the developer is also very responsive. They're (ambiguous 'they', not sure of pronouns) active in a libraries-fighting-bots slack channel I'm on. Libraries have been hit hard by the bots: we have hoards of tasty archives and we don't have money to throw resources at the problem.

[–] lilith267@lemmy.blahaj.zone 9 points 2 days ago (1 children)

The Anubis repo has an enbyware emblem fun fact :D

load more comments (1 replies)
[–] tofu@lemmy.nocturnal.garden 6 points 2 days ago

Cool, thanks for posting! Also the reasoning for the image is cool.

[–] mobotsar@sh.itjust.works 5 points 2 days ago (2 children)

Is there a reason other than avoiding infrastructure centralization not to put a web server behind cloudflare?

[–] poVoq@slrpnk.net 22 points 2 days ago

Yes, because Cloudflare routinely blocks entire IP ranges and puts people into endless captcha loops. And it snoops on all traffic and collects a lot of metadata about all your site visitors. And if you let them terminate TLS they will even analyse the passwords that people use to log into the services you run. It's basically a huge survelliance dragnet and probably a front for the NSA.

[–] bjoern_tantau@swg-empire.de 10 points 2 days ago (1 children)

Cloudflare would need https keys so they could read all the content you worked so hard to encrypt. If I wanted to do bad shit I would apply at Cloudflare.

[–] mobotsar@sh.itjust.works 7 points 2 days ago* (last edited 2 days ago) (2 children)

Maybe I'm misunderstanding what "behind cloudflare" means in this context, but I have a couple of my sites proxied through cloudflare, and they definitely don't have my keys.

I wouldn't think using a cloudflare captcha would require such a thing either.

[–] starkzarn@infosec.pub 14 points 2 days ago (1 children)

That's because they just terminate TLS at their end. Your DNS record is "poisoned" by the orange cloud and their infrastructure answers for you. They happen to have a trusted root CA so they just present one of their own certificates with a SAN that matches your domain and your browser trusts it. Bingo, TLS termination at CF servers. They have it in cleartext then and just re-encrypt it with your origin server if you enforce TLS, but at that point it's meaningless.

[–] mobotsar@sh.itjust.works 7 points 2 days ago

Oh, I didn't think about the fact that they're a CA. That's a good point; thanks for the info.

[–] bjoern_tantau@swg-empire.de 5 points 2 days ago* (last edited 2 days ago)

Hmm, I should look up how that works.

Edit: https://developers.cloudflare.com/ssl/origin-configuration/ssl-modes/#custom-ssltls

They don't need your keys because they have their own CA. No way I'd use them.

Edit 2: And with their own DNS they could easily route any address through their own servers if they wanted to, without anyone noticing. They are entirely too powerful. Is there some way to prevent this?

load more comments (9 replies)
[–] AnUnusualRelic@lemmy.world 22 points 2 days ago (1 children)

The problem is that the purpose of Anubis was to make crawling more computationally expensive and that crawlers are apparently increasingly prepared to accept that additional cost. One option would be to pile some required cycles on top of what's currently asked, but it's a balancing act before it starts to really be an annoyance for the meat popsicle users.

[–] rtxn@lemmy.world 23 points 2 days ago

That's why the developer is working on a better detection mechanism. https://xeiaso.net/blog/2025/avoiding-becoming-peg-dependency/

[–] 0_o7@lemmy.dbzer0.com 18 points 2 days ago

The article is very enthusiastic about explaining why it shouldn't work, but completely glosses over the fact that it has worked

This post was originally written for ycombinator "Hacker" News which is vehemently against people hacking things together for greater good, and more importantly for free.

It's more of a corporate PR release site and if you aren't known by the "community", calling out solutions they can't profit off of brings all the tech-bros to the yard for engagement.

load more comments (1 replies)
[–] unexposedhazard@discuss.tchncs.de 60 points 2 days ago (1 children)

This… makes no sense to me. Almost by definition, an AI vendor will have a datacenter full of compute capacity.

Well it doesnt fucking matter what "makes sense to you" because it is working...
Its being deployed by people who had their sites DDoS'd to shit by crawlers and they are very happy with the results so what even is the point of trying to argue here?

[–] daniskarma@lemmy.dbzer0.com 11 points 2 days ago* (last edited 2 days ago)

It's working because it's not very used. It's sort of a "pirate seagull" theory. As long a few people use it it works. Because scrappers don't really count on Anubis so they don't implement systems to surpass it.

If it were to become more common it would be really easy to implement systems that would defeat the purpose.

As of right now sites are ok because scrappers just send https requests and expect a full response. If someone wants to bypass Anubis protection they would need to take into account that they will receive a cryptographic challenge and have to solve it.

The thing is that cryptographic challenges can be very optimized. They are designed to run in a very inefficient environment as it is a browser. But if someone would take the challenge and solve it in a better environment using CUDA or something like that it would take a fraction of the energy defeating the purpose of "being so costly that it's not worth scrapping".

At this point it's only a matter of time that we start seeing scrappers like that. Specially if more and more sites start using Anubis.

[–] rtxn@lemmy.world 31 points 2 days ago (1 children)

New developments: just a few hours before I post this comment, The Register posted an article about AI crawler traffic. https://www.theregister.com/2025/08/21/ai_crawler_traffic/

Anubis' developer was interviewed and they posted the responses on their website: https://xeiaso.net/notes/2025/el-reg-responses/

In particular:

Fastly's claims that 80% of bot traffic is now AI crawlers

In some cases for open source projects, we've seen upwards of 95% of traffic being AI crawlers. For one, deploying Anubis almost instantly caused server load to crater by so much that it made them think they accidentally took their site offline. One of my customers had their power bills drop by a significant fraction after deploying Anubis. It's nuts.

So, yeah. If we believe Xe, OOP's article is complete hogwash.

[–] tofu@lemmy.nocturnal.garden 8 points 2 days ago

Cool article, thanks for linking! Not sure about that being a new development though, it's just results, but we already knew it's working. The question is, what's going to work once the scrapers adapt?

[–] Dremor@lemmy.world 18 points 2 days ago (1 children)

Anubis is no challenge like a captcha. Anubis is a ressource waster, forcing crawler to resolve a crypto challenge (basically like mining bitcoin) before being allowed in. That how it defends so well against bots, as they do not want to waste their resources on needless computing, they just cancel the page loading before it even happen, and go crawl elsewhere.

[–] tofu@lemmy.nocturnal.garden 6 points 2 days ago (2 children)

No, it works because the scraper bots don't have it implemented yet. Of course the companies would rather not spend additional compute resources, but their pockets are deep and some already adapted and solve the challenges.

[–] Dremor@lemmy.world 13 points 2 days ago (5 children)

To solve it or not do not change that they have to use more resources for crawling, which is the objective here. And by contrast, the website sees a lot less load compared to before the use of Anubis. In any case, I see it as a win.

But despite that, it has its detractors, like any solution that becomes popular.

But let's be honest, what are the arguments against it?
It takes a bit longer to access for the first time? Sure, but that's not like you have to click anything or write anything.
It executes foreign code on your machine? Literally 90% of the web does these days. Just disable JavaScript to see how many website is still functional. I'd be surprised if even a handful does.

The only people having any advantages at not having Anubis are web crawler, be it ai bots, indexing bots, or script kiddies trying to find a vulnerable target.

load more comments (5 replies)
[–] EncryptKeeper@lemmy.world 10 points 1 day ago* (last edited 1 day ago)

The point was never that Anubis challenges are something scrapers can’t get past. The point is it’s expensive to do so.

Some bots don’t use JavaScript and can’t solve the challenges and so they’d be blocked, but there was never any point in time where no scrapes could solve them.

[–] possiblylinux127@lemmy.zip 12 points 2 days ago* (last edited 2 days ago) (1 children)

Anubis sucks

However, the number of viable options is limited.

[–] seralth@lemmy.world 15 points 2 days ago

Yeah but at least Anubis is cute.

I'll take sucks but cute over dead internet and endless swarmings of zergling crawlers.

[–] CrackedLinuxISO@lemmy.dbzer0.com 10 points 2 days ago* (last edited 2 days ago) (1 children)

There are some sites where Anubis won't let me through. Like, I just get immediately bounced.

So RIP dwarf fortress forums. I liked you.

[–] sem@lemmy.blahaj.zone 10 points 2 days ago (1 children)

I don't get it, I thought it allows all browser with JavaScript enabled.

load more comments (1 replies)
[–] Lumisal@lemmy.world 10 points 2 days ago

Have you tried accessing it by using Nyarch?

[–] TwiddleTwaddle@lemmy.blahaj.zone 6 points 2 days ago

I'm constantly unable to access Anubis sites on my primary mobile browser and have to switch over to Fennec.

[–] VitabytesDev@feddit.nl 5 points 2 days ago

I love that domain name.

load more comments
view more: next ›