AW3423DWF using incorrect color format in HDR

I have an Alienware AW3423DWF with an Nvidia GPU running CachyOS KDE. When using HDR with 165hz, things start to look off and everything looks fringey/blurry, especially thin elements, text, and red icons. If I switch to 100hz or turn off HDR, things look normal again.

I found out that this might be because it’s running in YCbCr with chroma subsampling at 165hz. I believe this happens since the monitor doesn’t use DSC and therefore doesn’t support the full resolution at a 10 bit color depth and 165hz. On Windows higher refresh rates activates 8-bit with dithering instead, allowing me to stay at at full color resolution, but I haven’t figured out how to do the same on Linux, it goes to YCbCr chroma subsampling no matter what.

Is there any way to force 8-bit with dithering to allow me to use the full resolution and HDR without chroma subsampling, or does anyone with a similar monitor know of a solution for this?