NVME Performance Drive Comparison

[WillR]

With this year’s cratering of NVME drive prices, I’ve picked up a handful to play with on my VF2 board. Some high end and some that were even originally more budget drives. I have a VF2 rev 1.3B board powered by a 65W GaN PSU running cwt14’s ArchLinux booted from a Micro SD card. All drives had at least 1 partition with an ext4 fs (empty on most drives b/c they’re new) which was mounted but not doing anything. After changing out the SSD and booting, I mounted the fs, changed the governor to performance for more consistent results, and ran a simple hdparm -t-direct test 100 times then ran the Storage.sh script from pibenchmarks.com.

The drives I tested were a Crucial P3 Plus 2TB, an Intel P1600X 118GB Optane, a Mushkin Helix-L 1TB, a Patriot P300 1TB, a Samsung 970 Evo Plus 2TB, a Solidigm P44 Pro 1TB, and a Western Digital Black SN770 1TB. Not all of my most recent pibench results got uploaded correctly it seems. I do not know why, but I only used the most recent tests for some graphs.

Running the 100x hdparm read test, while odd, showed some even odder inconsistent speeds out of 1 drive, the WD 770. I may reinsert the drive to see if it pulls an unusual amount of power compared to some of the other drives. It saw a difference between its min and max throughput speeds of 68.26 MB/s (334.71 - 226.45). The most consistent drive, while also the slowest, was the Samsung 970 Evo.

All the drives were virtually identical on their write speeds during the single run of the PiBench Storage test.

And no drive really stands out as the best at all the tests on the VF2.

Being limited to 1 PCI-e 2.0 lane seems to hold them all back equally so the question of drive choice is almost moot imo. Use the size you need or want and if you’re buying new then just get whatever is on clearance. The $60 Solidigm is definitely not twice the performance of the $30 Mushkin (Newegg had a promo code on it a couple weeks ago). It’s 11% faster at Cached reads and 34% faster at IOZone’s 4k Random Read. Everything else is <=10% faster or in a few cases slower.

Ignoring the Intel drive, the PiBench generated “scores” of the fastest drive are only 11% higher than the slowest.

Finally some plain data.

If anyone is interested in the really raw data, I can attach my shell outputs or clean up my spreadsheets into 1 and upload it.

If anyone wants to see others comparisons of the drives, feel free to ask.

Oh, last thought on the drives. The Mushkin stands out for having an orange LED on it… and a sticky label.

[RiscVFan]

We’ve known the single-lane experience on these was sure to keep them out of high-performance use. (I’d imagine that attaching a GPU via an external cage would be similarly frustrating.) It is a welcome alternative to the speed (and self-destructive nature) of SD cards, though.

However, the (imaginary) 4Tbps of a single lane is within a stone’s throw of USB3’s (more imaginary) 4.8Tbps. Adding an additional packetization/serialization/locking/bus arbitrarion to use an SD card over USB instead of over “raw” PCI isn’t likely to help, but do you have any similar feeling of how well an external USB SSD (a real SSD; not an SD card) is likely to perform in these hosts?

[WillR]

No idea, but I’m curious enough to find out. So I just ordered a USB to NVME enclosure to try some of the drives out and see how much overhead USB adds. I didn’t have one already, but it seems like something handy to have around. I’ll be out of town this weekend and possibly next week, so it’ll probably be a while before I have any data to chart and compare.

[RiscVFan]

Cool. Thank you!

FWIW, even being spotted .8Tbps, I don’t expect it to win. The question is more “did they nerf the USB interface like they nerfed the SD interface?”

[WillR]

I have no expectation of anything connected to the USB ports or controller being faster. They run over and share a same PCIe 2.0 x1 lane like the NVME does. If anything the bandwidth should be lower, but I expect close to the same as mounting the drive natively. The IO could be a different story completely. I wouldn’t be surprised to see small reads and writes plummet due to the USB and CPU overhead. We’ll see.

As for the SD card adapter, I have no idea. The performance there is awful. The drive I currently boot the board from sees 70MB/s when in my own adapter in a USB port but 11MB/s when using the built in card reader. I think my best card gets 190MB/s using the USB port and just over 20MB/s using the VF2’s reader. I’ve never explored why.

[RiscVFan]

Ah, I didn’t know it was on PCI. I assume it was a grown-up HCI that could do bus arbitration and mastering on its own. I’d agree with your general guesses, then.

See stat_crabhead’s posts starting around Question about sd-card speed - #8 by Stat_headcrabed The higher clock rates on that chip are only available at 1.8V and they nailed it to the 3.3V rails and it’s shared with several other chips. You can’t just trivially cut that pin loose and re-anchor it and then remove the software lock on setting the clock speed to “fast”.

Hardware limitation or hardware bug? Tough to say, but it’s pretty likely to get picked up if they ever decide to lay the board out again for a respin.

SD being awful is one reason so many people were so anxious to go to PCIe…before discovering IT was disappointing, but at least faster than SD.

[WillR]

I tested all the drives that I could. Most actually would not work in the SSK Aluminum M.2 to USB NVMe SATA SSD Enclosure that I ordered. The enclosure uses a RTL9210B. The 3 drives (out of 8) that worked were the Crucial P3 Plus 2TB, Mushkin Helix-L 1TB, and Patriot P300 1TB. The first drive I tested was the Mushkin and it actually surprised me with hdparm being faster while in the USB enclosure (by about 33MB/s or 11%). It was repeatable but for only that drive. That was the only drive and only test that was faster from the enclosure. The other 2 drives saw 8-9% slower hdparm results. All drives had 6-16% lower results from most of the write tests, but the read performance due to USB overhead on the 4k tests suffered losses from 43-60% of the drive’s speed when mounted to the board.

The most painful “successful” test was on the Optane drive that I immediately pulled after seeing 1 hdparm run.

root@ArchVF2 [~/] # hdparm -t-direct /dev/sda4

/dev/sda4:
Timing O_DIRECT disk reads: 2 MB in 92.15 seconds = 22.22 kB/sec
root@ArchVF2 [~/] #

Never seen that before. And the drive probably disappeared and wouldn’t populate sda or sdb just like the Samsung 970 Evo, 990 Pro, Solidigm P44, and Western Digital SN770 behaved.

So drives would be usable this way, but there is no benefit and they will run quite hot depending on the enclosure. It would be fine for dumping images, but I’m not sure I’d even want to use it just for storage long term as hot as everything gets.

Looking at this as a test of the USB system, I’d conclude it works quite well.

[WillR]

Tried a couple more 2TB drives. Samsung 990 Pro and a Patriot Viper VP4300 Lite.