I would imagine a clean install, over just an update from x11 to wayland, might work better, since applications might still expect x11, and fail to render or work properly.

I used gnome with wayland and an Nvidia gpu maybe a couple years back, and it worked pretty well. I’d give it another go. mutter, gnome’s wayland compositor is actually pretty good compared to most others.