You can set Vim as your pager. If you are on Neovim, its very easy, just set this variable:
export PAGER='nvim +Man!'
export MANPAGER="${PAGER}"
Then try man grep. These variables could be set in your .bashrc in example. If you use the "original" Vim, it's a bit more complicated to setup. I did that before too, so it's definitely doable.