I got into the self-hosting scene this year when I wanted to start up my own website run on old recycled thinkpad. A lot of time was spent learning about ufw, reverse proxies, header security hardening, fail2ban.
Despite all that I still had a problem with bots knocking on my ports spamming my logs. I tried some hackery getting fail2ban to read caddy logs but that didnt work for me. I nearly considered giving up and going with cloudflare like half the internet does. But my stubbornness for open source self hosting and the recent cloudflare outages this year have encouraged trying alternatives.
Coinciding with that has been an increase in exposure to seeing this thing in the places I frequent like codeberg. This is Anubis, a proxy type firewall that forces the browser client to do a proof-of-work security check and some other nice clever things to stop bots from knocking. I got interested and started thinking about beefing up security.
I'm here to tell you to try it if you have a public facing site and want to break away from cloudflare It was VERY easy to install and configure with caddyfile on a debian distro with systemctl. In an hour its filtered multiple bots and so far it seems the knocks have slowed down.
My botspam woes have seemingly been seriously mitigated if not completely eradicated. I'm very happy with tonights little security upgrade project that took no more than an hour of my time to install and read through documentation. Current chain is caddy reverse proxy -> points to Anubis -> points to services
Good place to start for install is here
Kinda sucks how it makes websites inaccessible to folks who have to disable JavaScript for security.
I kinda sucks how AI scrapers make websites inaccessible to everyone 🙄
and they dont respect robots.txt
Not if the admin has a cache. It's not a difficult problem for most websites
You clearly don't know what you are talking about.
Lol I'm the sysadmin for many sites that doesn't have these issues, so obviously I do..
It you're the one that thinks you need this trash pow fronting for a static site, then clearly you're the one who is ignorant
Obviously I don't think you need Anubis for a static site. And if that is what your admin experience is limited too, than you have a strong case of dunning krueger.
there's a fork that has non-js checks. I don't remember the name but maybe that's what should be made more known
Please share if you know.
The only way I know how to do this is running a Tor Onion Service, since the tor protocol has built-in pow support (without js)
It's this one: https://git.gammaspectra.live/git/go-away
the project name is a bit unfortunate to show for users, maybe change that if you will use it.
some known privacy services use it too, including the invidious at nadeko.net, so you can check there how it works. It's one of the most popular inv servers so I guess it cannot be bad, and they use multiple kinds of checks for each visitor
ps: I was wrong it's not a fork, but a different thing doing the same and more