There really aren't that many modem/router combos out there you can actually flash with open source firmware.
Router? Sure, no problem.
Modem? Outside of some very niche Fritzbox devices and others listed on OpenWRT Table of Hardware
There really just aren't that many.
So, fine. I will use a proprietary modem.
I will put it into bridge mode, let it decode the DSL analog signals into ethernet and feed that into a router of my choice.
Question is: Which modem?
I don't want two devices heating up my shelf. I just want a tiny modem that converts analog to digital into an RJ45 and then let's the router do its thing.
Does such a magical device exist, because my search on this front has proved largely fruitless
Not sure if they're still in production. There was also an Italian company selling one (maybe based on the same hardware).
Like u/floquant I would look at Mikrotik. But you'll need to pay attention to port speed (they still sell 100Mbps gear) and their tested routing throughput numbers to be sure they can hit the numbers you need.