I recall this happening on my higher refreshrate and resolution screen on X11 and nvidia (iirc a 2070 super). Moving to wayland fixed it.
I mainly noticed frame drops in videos in forefox, but I'm sure scrolling was also affected.
It probably manifests on a per usecase basis since it depends on how exactly the gpu or x11 is loaded. Whatever it is, it's some consequence of x11 being flawed at its core abd just not scaling well with modern usecases.
In my case I was forced onto wayland after I got my new better monitor due to how choppy x11 felt.
I had 2 1080p 60Hz and got a 4k 120/144Hz screen.