Yep. It depends on the soundcard and its support, as well as the DE. The DE could sometimes be a bottleneck, especially when you want to run session managers.
You also may want to try a rt kernel in some cases (you are often forced to switch back to alsa then), but pipewire should work around that. With pw-jack you even can take up most of the advatages from the jack server.
But honestly I am not heading for extra latency, because I am not singing along my stuff, which is
very much appreciated by my neighbours.
I use an old Behringer UMC 404 via USB and can get as low as 256 k if I want on Cachy, but 512 is more stable and works best with most of the plug-ins I use.
Sure, the best latency I got still was using Windows. I could go for 64k and 96 kHz, but with a cpu load, that made working with multiple tracks kind of useless. Followed by Ubuntu Studio with the same setup as on Cachy using 128k, but this was very very instable.