Significant performance drop / slow checkpoint loading in ComfyUI

Issues & Assistance
1

Hi everyone,

I’ve noticed a major performance regression regarding checkpoint loading (moving the model into GPU VRAM) in ComfyUI after updating to recent kernels.

The Issue:
The very first generation in ComfyUI, which involves loading the checkpoint into the GPU, takes an extremely long time.

Benchmark comparison:

  • Kernel 7.1.rc7-1-cachyos: Bug persists (extremely slow)
  • Kernel 7.0.12-1-cachyos: 405.26 seconds
  • Kernel 6.18.35-1-cachyos-lts: 25.56 seconds

kernel_compare
kernel_compare1038×560 263 KB

(Note: I think It was working fine on the previous 7.0.11 kernel, but since it’s no longer in the repositories (i tried to install it but it keep’s installing 7.0.12), I couldn’t re-test it directly right now, so I tested against the LTS kernel where the performance is normal).

Has anyone else experienced this, or is there a known workaround for the 7.0.12+ and 7.1.rc series?

Thanks for any help!

2

Thanks to the new kernel version 7.1.1-1, the problem seems to be fixed.

kernel_7.1.1-1-cachyos-bore
kernel_7.1.1-1-cachyos-bore1321×245 57.4 KB

3

Seems like I was celebrating too soon. While the model loading speed is fixed in 7.1.1 and up, switching to a different checkpoint (model) on the fly completely freezes the PC. The CPU load spikes to 100% and it starts thrashing the system RAM until the system locks up. The ComfyUI “Unload Models” menu just says “0 models unloaded”.

When I boot back into the older linux-cachyos-lts 6.18.38-2 kernel, model swapping works perfectly fine without any issues. It looks like a VRAM reallocation/garbage collection regression in the newer amdgpu driver or KFD layer with ROCm. Any tips on what is causing this or how to fix it would be greatly appreciated.