Video container types (.mkv, .mp4) are different from encoding types (x264, 265)
For both, the “best” or “better” is gonna depend on your use case. .mkv is nice because they bundle multiple video tracks, audio tracks, and subtitle tracks all into one file, but .mp4 is pretty much guaranteed to be playable on anything (although the number of devices that don’t support .mkv is pretty small at this point)
x265 is nice because the quality:file size ratio is much better than x264, but again, compatibility is an issue because x265 is still somewhat new, so older devices probably won’t support it.
As you can probably tell, I prefer .mkv + x265 whenever possible, but that’s only because every device I own can play them