Maybe I got it narrowed down a bit.
Everything was fine, then I played around with Ventoy and booted cachyos-desktop-linux-241110.iso
It worked while it was booted there, did a reboot - And mouse stopped working right after boot.
For other people finding this later the fix is easy: Boot a Windows Install ISO, unplug the dongle and plug it back in. Reboot.