Note this VisionFive 2 Debian Image 202308 Released (latest) - #24 by Tenkawa
My initial thoughts is that its kernel related with the driver handling although I need to do much more testing to isolate the individual calls causing it to create the artificial load. I have been able to recreate the condition and the fix on both the VF2 and Pine64 JH7110 SBCs however.