I had the same issue as long as I powered my VF2 from a common 3A USB-C power supply and also with the 4A for the Pi4. Since I use a 65W max USB-C Multi-Slot power supply, I’m rid of those messages.
I assume, that there are short spikes in power consumption exceeding the 3A/4A border of the simpler power supplies and that then disturbs the PICe protocol on the bus causing retries.
3 Likes
Hello, everyone! Is there already a solution to this problem? I have an OrangePi RV JH7110 and NVMe WD SN740. I’ve tried different power supplies, but nothing helps. I’m currently using vanilla kernel 6.19-rc1 (because DTS for my development board has appeared there).
You could try with a different SSD. Have a look at the Compatibilty List
I am running a Toshiba KXG50ZNV256G on the VF2 and a “Samsung Electronics Co Ltd NVMe SSD Controller 980 (DRAM-less)” on the VF lite (can’t tell to much about the compatibility of the later, as it is still running from the eMMC, haven’t installed anything to the NVMe yet).
A Union Memory AM620 256GB was also detected on the VF lite.
If you don’t want to hunt for one of the devices on the list, get something on the lower performance spectrum, they tend to use less power. Smaller capacity M.2 with 80mm length should be easily available both new and used, even with the current AI induced shortage.
2 Likes
Thanks a lot for your answer. By the way, people here are experimenting with various kernel startup parameters. For me, the situation is slightly better when the parameters “pcie_port_pm=off pcie_aspm.policy=performance” are used, but timeout messages are still present.
You could try with a different SSD.
I found and ordered one of the NVMe drives from the list of supported devices. When it arrives, I will post the results here.
“pcie_port_pm=off pcie_aspm.policy=performance”
Without these options, the system doesn’t boot at all and endless timeout errors appear.
What are you using as a power supply ?
I did a quick search on the Western Digital SN740 NVMe SSD and it has a maximum power draw listed that varies by capacity, generally ranging from about 4.6 watts to 6.0 watts under maximum load (active use).
The 8 GB VisionFive 2 under maximum load (CPU/GPU cores all stressed) will probably consuming about 5W to 6W watts in isolation, with no peripheral hardware attached.
The maximum power in to VF2 is specified to be 30 watts via the USB-C cable.
Even if a cheaper power supply is actually rated to supply enough power (VF2 + NVMe + USB peripherals), it might not be able to ramp up from low power to maximum power usage (CPU+SSD) fast enough to match the change in power demand. Usually this delay is worked around by having large enough decoupling capacitors in modules or other peripherals, but to save money this can be skimped upon by using physically smaller lower capacitance capacitors that can not provide enough current for long enough to bridge any delays in the switched mode power supply ramping up the current (The assumption being made is that the SMPS will be high end to handle the problem with more expensive higher capacitance lower ESR capacitors). And that could be why the timeouts are happening. The easy solution is to use an older NVMe that do not have the same extremely rapid changes in power demands and/or use a higher quality power supply.
1 Like
What are you using as a power supply ?
I tried Zenwire YMS-608 100W and something called “Raspberry Pi 5 5.1V 5A Power Supply PD 27W USB Type C” from AliExpress. No difference at all, same behavior. I also found information that some ARM-based SBCs also have similar behaviour, and they also recommend installing an older NVMe, which consumes less power and, apparently, solves the problem. I also have an OrangePi RV2 (Spacemit K1), and there are no problems at all with either the power supplies or the NVMe.
Some NVMe SSDs actually can’t be used on the VF2; the XPG SX8200 Pro is one of them. It works fine on my mini PC (N6000), by the way.
Today I received a Kingston NV1 NVMe M.2 2280 SSD 250GB and copied the root partition to it. After booting, I ran find /gnu -type f | xargs md5sum (I use Guix System) and immediately got a timeout message. After I removed the pcie_port_pm=off pcie_aspm.policy=performance from kernel boot options and rebooted, the timeout errors stopped occurring. To be more precise, they appear one-two times during kernel boot and do not appear again even when actively working with the disk. This is true with every power adapter I own, no matter.
1 Like
There are no miracles – a few days later, timeouts returned.
TL;DR: anything related to power supplies, ASPM, or tuning the various nvme_core kernel parameters can be set aside — the real issue is elsewhere. The interrupt does fire, but the completion is missed and only recovered later by the timeout poll. This is a timing/race issue in the interrupt handling path, not a power problem.
Evidence that it is NOT power/PM/external:
Using the Ubuntu image provided by StarFive (Linux 6.12.5 with StarFive’s custom patches), I get no timeout at all — not at boot, not later. Zero. Better: I recompiled dozens of Gentoo packages of varying size inside a chrooted F2FS partition on a Kingston NVMe, using that very same StarFive image → 0 NVMe timeout messages. The exact same workload under a vanilla 6.12.5 kernel is a nightmare: timeouts over timeouts.
Same board, same PSU, same drive, same everything external — only the kernel changed. So power supply / power management / external factors are out from the start.
“Same” kernel version, “same” config, different result → the culprit is the kernel code.
What is missing in the Linux vanilla kernel?
Diffing the vanilla 6.12.5 sources against StarFive’s shows essentially no difference in the NVMe/PCIe path except this (in drivers/nvme/host/pci.c, function nvme_poll_cq(), called from the IRQ handler):
/*
* In some cases, such as udev trigger, cqe status may update
* a little bit later than MSI, which cause an irq handle missing.
* To workaound, here we will prefetch the status first, and wait
* 1us if we get nothing.
*/
if (!nvme_cqe_pending(nvmeq))
udelay(1);
(plus #include <linux/delay.h>).
What it means: on the JH7110, the completion MSI can be seen by the CPU before the CQE (DMA write) has landed in host memory. nvme_irq() → nvme_poll_cq() then finds the queue empty, returns without reaping anything, and the command is only completed when the 30 s I/O timeout polls the queue → hence “completion polled”. It is not a lost interrupt — the data simply isn’t visible yet when the handler runs. Under continuous load the next completion’s IRQ rescues it; it only becomes a visible timeout when the queue goes idle right after an isolated, cold completion.
Result: integrating that snippet into a vanilla kernel makes a real difference as ~99% of the NVMe timeouts are now gone.
The only remaining variable is the filesystem: StarFive’s Ubuntu image uses ext4, whereas my Gentoo stage 4 uses F2FS, which is far more I/O-intensive (log-structured, lots of small scattered metadata reads) and therefore much more likely to trigger the residual. I’d expect 0 timeouts on ext4 (still to be validated). The rare residual is consistent with the fixed 1 µs occasionally being too short when the CQE is very late (cold cache); a slightly longer/adaptive wait should close that gap.
Bottom line: this is a StarFive BSP patch that never made it upstream, which is why every mainline kernel on the VF2 hits this and every StarFive-image user does not. It has nothing to do with power supplies or ASPM.
That udelay hack from starfive is never going to make it upstream. The actual problem lies in the SoC, or rather the PCIe controller integration of it. There was a discussion back in 2024: ref: Re: [PATCH v15,RESEND 22/23] PCI: starfive: Offload the NVMe timeout workaround to host drivers. where I asked starfive to root cause it, but never got any real progress since then. The problem might be originated in either PCIe controller, interconnect fabric, or the risc-v core, or a combination of all of them. The design of JH7110 SoC is based on Sifive U74 core. It’s a risc-v core using the traditional CLINT+PLIC, without native MSI support. The PCIe controller has to translate a MSI into wired interrupt. What should happen is that the PCIe controller must forward the posted write DMA from the NVMe drive to the fabric, ensure it’s acked by the fabric, and only after that, trigger the wired interrupt. Maybe the PCIe controller wasn’t doing that? Or maybe the fabric was giving false ack’s? Or maybe somehow the writes went stuck in the fabric and became visible to the risc-v core only after the wired interrupt was delivered? I can only guess. This is a tricky scenario, because using the out-of-band wired interrupt sort of breaks all the nice properties of MSI, and it needs to be handled carefully to “restore” the ordering mandated by the MSI standard.
Overall it seems to me it’s the lack of system-level validation that lead to such failure, as there’re several components involved. NVMe device → PCIe controller → Interrupt controller (PLIC) → risc-v core. Everyone needs to work coherently along that path. Given the lack of transparency and follow up from Starfive on this issue, I would treat the PCIe on JH7110 as unstable and avoid putting mission critical device on it, such as NVMe with useful data, in fear of data corruption. For other peripherals, I haven’t seen issues like this so far.
patch for 7.2
1 Like