There really aren't that many modem/router combos out there you can actually flash with open source firmware.
Router? Sure, no problem.
Modem? Outside of some very niche Fritzbox devices and others listed on OpenWRT Table of Hardware
There really just aren't that many.
So, fine. I will use a proprietary modem.
I will put it into bridge mode, let it decode the DSL analog signals into ethernet and feed that into a router of my choice.
Question is: Which modem?
I don't want two devices heating up my shelf. I just want a tiny modem that converts analog to digital into an RJ45 and then let's the router do its thing.
Does such a magical device exist, because my search on this front has proved largely fruitless
You don't say which modem you mean, but from context it seems to be xDSL. I'm running an own cable modem (Vodafone in Germany) which works a lot better than Vodafone's own POS (which insists to reboot itself every day or two despite being in bridge mode). There is a thin client running opnsense behind it.
As to picking the smallest and lowest power, I think it is better to pick one that works best. When I used DSL I bought a cheap hackable DSL modem from an italian ISP and put OpenWRT on it. It also ran in bridge mode. It had an SFP slot which can come handy for galvanic separation, but I didn't bother with it.