I don't think you're missing anything. I think that your co-workers bragging is one of the toxic effects of how we tend to think about productivity nowadays, especially in America. I think that there's a tendency to glorify suffering (i.e. sacrificing time with your family to do so much work that by the time you get home to your family, you're too exhausted to be fully present with them).
I know fathers who effectively didn't have a choice about spending time with their newborns, because of a mixture of social pressures (especially gendered pressure from extended family) and financial pressure (such as not having access to paternity leave), who then go on to brag about how much they worked and sacrificed, framing it as if it's a choice they're glad they made. I think that for some people, this nonsense rhetoric is what they tell themselves to cope with the fact they were effectively coerced into something they regret.
Long story short, you're not missing anything. You are, in some ways though, going against the grain: even in places that have paid paternity leave, that alone isn't enough to change the tide of social attitudes. That change happens because of people like you who go "fuck this nonsense, I'm not making a martyr of myself to support my family when I can do a much better job supporting them if I'm there with them".
Unfortunately, based on reports from friends who are fathers, this is just scratching the surface of people being weird about men who are enthusiastic and engaged fathers. It sounds like you've got your priorities in order though. Your coworkers are very silly, and even if you don't feel it appropriate or necessary to tell them how absurd they are, you should at least internalise the fact that you are the sensible one here. An analogy that comes to mind is how, if your employer matches your 401k contributions, it's a no-brainer to take advantage of what is basically free money. If someone has "spare" salary and asked for financial advice online, one of the first and most basic suggestions is often that if you're not already taking advantage of any 401k match your employer offers, you definitely should be. It's free money! Similarly, taking advantage of the paid paternity leave is a no-brainer. This isn't a challenge run in a video-game, so there aren't any prizes for making things needlessly harder for oneself.
Edit: Also, I bloody hate it when people say shit like this:
"Most men are hard working and want to support their families so they don't take leave".
The subtext they're saying here is "I don't acknowledge parenting (and other caring labour) as being hard work, and I certainly don't acknowledge how critical essential this labour is for the world to function. I assume that this work is primarily for women, because this allows me to ignore it and the people who do it, which allows me to feel more important. The only way I can maintain my self identity as 'hardworking' is if I implicitly demean others' hard work".
It's bullshit, and your instincts are right to flag this shit as weird. Parenting is bloody difficult, and anyone who makes comments like this are actively reinforcing old systems that led to many fathers not being given the opportunity to be active fathers.
Anyway, rant finished. I'll finish this edit with something I forgot to say in my main comment: congratulations, and good luck in the weeks to come. And well done on taking this paternity leave, because that helps to disrupt the existing, outdated systems of traditional family structure that make everyone miserable. The impact of one person's choice is only small, but if enough people opt for their family over slaving over the altar of capitalism, I hope that we can build a world where a father wanting to actively be a father is treated like the normal thing it is.