NVME Boot using VisionFive2 Software v2.11.5

[kprasadvnsi]

I managed to boot kernel directly from nvme ssd in VF2. Here is the steps for anyone trying to do the same.

Note: These are some bleeding edge stuff and can potentially brick your VF2. You have been warned.

Step-1: Get the latest sdcard.img file from here. after booting this new sdcard.img, we need to update the bootloader in QSPI Flash.

Step-2: Update the QSPI Flash bootloader.
Either scp or use wget to download the required bootloader files into the running system.

wget https://github.com/starfive-tech/VisionFive2/releases/download/VF2_v2.11.5/u-boot-spl.bin.normal.out

wget https://github.com/starfive-tech/VisionFive2/releases/download/VF2_v2.11.5/visionfive2_fw_payload.img

Now using the flashcp command we will burn these files into the QSPI Fash

flashcp -v u-boot-spl.bin.normal.out /dev/mtd0
flashcp -v visionfive2_fw_payload.img /dev/mtd1

Step-3: Prepare NVME SSD with VF2 Debian Image. For this step I just followed a very good blog post by James A. Chambers StarFive VisionFive 2 Official Debian SSD Boot Guide

We don’t need to following the complete blog post. just the parts where he copied the debian to nvme SSD.

Step-4: Make sure the boot mode is set to QSPI Flash. Remove the microSD card an power up the board. we are looking for the u-boot prompt.

For some unknown reason interrupting the u-boot at Hit any key to stop autoboot: results into PCI device not getting detected. One workaround is to let u-boot fall back to the prompt due to not having a valid boot configuration.

Step-5: look for a PCI device using the pci command.

Ones we have a PCI device detected, we can try for scan for a nvme device by using the nvme scan command.

Now we will check the partitions on this nvme SSD by using the nvme part command.

I know that its the 2nd partition that have kernel image and DTB for our board. we will check this files using the fatls command.

Setp-6: We now have all thing required to boot linux from nvme SSD. Execute the following command in sequence.

fatload nvme 0:2 ${kernel_addr_r} /boot/vmlinuz-5.15.0-starfive

fatload nvme 0:2 ${ramdisk_addr_r} /boot/initrd.img-5.15.0-starfive

fatload nvme 0:2 ${fdt_addr_r} /boot/dtbs/starfive/jh7110-visionfive-v2.dtb

setenv bootargs 'root=/dev/nvme0n1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0'

setenv kernel_comp_addr_r 0x50000000

setenv kernel_comp_size 0x04000000

booti $kernel_addr_r $ramdisk_addr_r:$filesize $fdt_addr_r

This will let you boot the kernel.

Note: Make sure to have nothing connected to the board other then power and serial port. I found my kernel panic when HDMI cable is plugged in.

[coltree]

flashed the new firmware, switches set to Flash
followed instructions for a successful manual boot, then reboot
failed to find nvme when booting from flash
probes pcie
pci fails to autoconfig

Do I have to type this for each nvme boot or can the configuration be saved to uboot ?

U-Boot SPL 2021.10 (Mar 24 2023 - 01:42:56 +0800)
DDR version: dc2e84f0.
Trying to boot from SPI

OpenSBI v1.2

---

/ __ \ / | _ _ |
| | | | __ ___ _ __ | ( | |) || |
| | | | '_ \ / _ \ '_ \ ___ | _ < | |
| || | |) | __/ | | |) | |) || |
_/| ./ _|| ||/|/___|
| |
|_|

Platform Name : StarFive VisionFive V2
Platform Features : medeleg
Platform HART Count : 5
Platform IPI Device : aclint-mswi
Platform Timer Device : aclint-mtimer @ 4000000Hz
Platform Console Device : uart8250
Platform HSM Device : —
Platform PMU Device : —
Platform Reboot Device : pm-reset
Platform Shutdown Device : pm-reset
Platform Suspend Device : —
Firmware Base : 0x40000000
Firmware Size : 392 KB
Firmware RW Offset : 0x40000
Runtime SBI Version : 1.0

Domain0 Name : root
Domain0 Boot HART : 1
Domain0 HARTs : 0*,1*,2*,3*,4*
Domain0 Region00 : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()
Domain0 Region01 : 0x0000000040000000-0x000000004003ffff M: (R,X) S/U: ()
Domain0 Region02 : 0x0000000040040000-0x000000004007ffff M: (R,W) S/U: ()
Domain0 Region03 : 0x0000000000000000-0xffffffffffffffff M: (R,W,X) S/U: (R,W,X)
Domain0 Next Address : 0x0000000040200000
Domain0 Next Arg1 : 0x0000000042200000
Domain0 Next Mode : S-mode
Domain0 SysReset : yes
Domain0 SysSuspend : yes

Boot HART ID : 1
Boot HART Domain : root
Boot HART Priv Version : v1.11
Boot HART Base ISA : rv64imafdcbx
Boot HART ISA Extensions : none
Boot HART PMP Count : 8
Boot HART PMP Granularity : 4096
Boot HART PMP Address Bits: 34
Boot HART MHPM Count : 2
Boot HART MIDELEG : 0x0000000000000222
Boot HART MEDELEG : 0x000000000000b109

U-Boot 2021.10 (Mar 24 2023 - 01:42:56 +0800), Build: jenkins-github_visionfive2-13

CPU: rv64imacu
Model: StarFive VisionFive V2
DRAM: 8 GiB
MMC: sdio0@16010000: 0, sdio1@16020000: 1
Loading Environment from SPIFlash… SF: Detected gd25lq128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

StarFive EEPROM format v2

--------EEPROM INFO--------
Vendor : StarFive Technology Co., Ltd.
Product full SN: VF7110B1-2253-D008E000-00002190
data version: 0x2
PCB revision: 0xb2
BOM revision: A
Ethernet MAC0 address: 6c:cf:39:00:32:e5
Ethernet MAC1 address: 6c:cf:39:00:32:e6
--------EEPROM INFO--------

In: serial
Out: serial
Err: serial
Model: StarFive VisionFive V2
Net: eth0: ethernet@16030000, eth1: ethernet@16040000
Card did not respond to voltage select! : -110
Card did not respond to voltage select! : -110
bootmode flash device 0
Card did not respond to voltage select! : -110
Hit any key to stop autoboot: 0
Card did not respond to voltage select! : -110
Couldn’t find partition mmc 0:3
Can’t set block device
Importing environment from mmc0 …

Warning: Input data exceeds 1048576 bytes - truncated

Info: input data size = 1048578 = 0x100002

Error: “boot2” not defined

Card did not respond to voltage select! : -110
Couldn’t find partition mmc 0:3
Can’t set block device

Warning: defaulting to text format

Card did not respond to voltage select! : -110
starfive_pcie pcie@2C000000: Starfive PCIe bus probed.
PCI: Failed autoconfig bar 10
ethernet@16030000 Waiting for PHY auto negotiation to complete… done
BOOTP broadcast 1
BOOTP broadcast 2
BOOTP broadcast 3
BOOTP broadcast 4
*** Unhandled DHCP Option in OFFER/ACK: 125
*** Unhandled DHCP Option in OFFER/ACK: 125
DHCP client bound to address 192.168.1.120 (1861 ms)
*** ERROR: serverip' not set Cannot autoload with TFTPGET BOOTP broadcast 1 *** Unhandled DHCP Option in OFFER/ACK: 125 *** Unhandled DHCP Option in OFFER/ACK: 125 DHCP client bound to address 192.168.1.120 (139 ms) *** ERROR: serverip’ not set
Cannot autoload with TFTPGET
StarFive #

[kprasadvnsi]

@coltree

Do I have to type this for each nvme boot or can the configuration be saved to uboot ?

unfortunately yes, This is just to demonstrate that nvme boot works. I will a better way to do it that doesn’t require typing everything manually.

[mzs]

The env save or saveenv command saves the U-Boot environment in persistent storage.
To reset back to factory defaults is env default -a -f with a env save or saveenv

[coltree]

permanent fix here using mostly existing scripts and your kernel_comp_addr and size variables

Software 2.11.5 and nvme boot

in previous post, seems fdtfile has to be fixed in u-boot env because it is needed before uEnv.txt is read from /boot then missing kernel_comp_addr_r=0x50000000 kernel_comp_size=0x04000000 env edit these saveenv now gets through to systemd edit /etc/fstab, change to nvme0n1p3 and nvme0n1p4 forgot to mention changed /boot/extlinux/extlinux.conf to root=/dev/nvme0n1p4 that’s got it for booting from nvme BUT booting ssd is now broken unless ssd is selected with switches instead of flash …

[kprasadvnsi]

anybody tested it other then OP?

[strlcat]

Yeah, me. When I heard about this I immediately rushed to get git head then to build it.
Immediately I found that U-Boot does not enumerate PCI automatically, but I knew it can be scripted to do so before you even get dropped to it’s prompt.

U-Boot runs preboot each time it runs but before bootcmd or shell pops up. All you need is to properly instruct it:

Append pci enum; nvme scan to end of existing preboot contents: echo $preboot to get current contents, then add ; pci enum; nvme scan to the end. For example, I had run chipa_set_uboot;run mmcbootenv in $preboot, I just made it look like run chipa_set_uboot;run mmcbootenv; pci enum; nvme scan. Run saveenv to save changes (they are saved to same QSPI flash space at 0xf0000), and reset / power cycle your VF2. If your NVME is inserted, it will catch up automatically, otherwise there is no any harm if no any present, driver will just silently fail but will not prevent U-Boot from continuing.

Sometimes PCI enumeration gets stuck for about ten seconds, dunno why, it happens rarely and usually successfully.

If you don’t like it, you can remove the change. In worst case you can reflash the entire thing with recovery binary.

HTH

[strlcat]

I also made entire thing of booting more convenient by instructing U-Boot to read an extlinux.conf from my NVME boot partition.

1. Copy existing contents of bootcmd to a backup variable: env set bootcmd_b $bootcmd
2. Set bootcmd to sysboot nvme 0:1 any ${pxefile_addr_r} /extlinux.conf
3. Save environ: saveenv
4. Reset / power cycle

Prepare your {nvme0n1p1}/extlinux.conf with like:

menu title Slackware RISCV boot options
timeout 20
default linux64

label linux64
        kernel /linux64.gz
        fdt /jh7110-visionfive-v2.dtb
        append ro quiet console=ttyS0,115200 earlycon=sbi root=/dev/nvme0n1p2 rootwait stmmac.chain_mode=1
        initrd <optional if you need it>

idk, you might also need a /extlinux/extlinux.conf symlink to /extlinux.conf. Mine board does have it. If you have troubles booting, move file there or make this symlink. Consult U-Boot environment.

This way you can add boot targets, like working one and another with init=/bin/sh for recovery. And you’ll get text menu to choose from, with timeout to default.

[easytarget]

@strlcat

This way you can add boot targets, like working one and another with init=/bin/sh for recovery. And you’ll get text menu to choose from, with timeout to default.

You can make these changes permanent by editing /etc/default/uboot, eg this is what I have:

#U_BOOT_UPDATE="true"
#U_BOOT_ALTERNATIVES="default recovery"
#U_BOOT_DEFAULT="l0"
#U_BOOT_ENTRIES="all"
#U_BOOT_MENU_LABEL="Debian GNU/Linux"
#U_BOOT_PARAMETERS="ro quiet"
U_BOOT_ROOT="/dev/nvme0n1p4"
U_BOOT_TIMEOUT="200"
#U_BOOT_FDT=""
#U_BOOT_FDT_DIR="/usr/lib/linux-image-"
#U_BOOT_FDT_OVERLAYS=""
#U_BOOT_FDT_OVERLAYS_DIR="/boot/dtbo/"
U_BOOT_PARAMETERS="rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0"
U_BOOT_FDT_DIR="/dtbs/"

If you are upgrading kernels via packages (as I do) this helps update-uboot to set the defaults properly when dpkg -i installs the new kernel.

Note that I remove the root= line from the line parameters, U_BOOT_ROOT should be set automagically by DPKG based on the current root partition, but specifying it here makes sure it is correct. If you do not remove it from the default line you will have two root= entries in the extlinux commandline, which works but is ugly/confusing.

[strlcat]

I run Slarm64/riscv64 and I dont have these

[coltree]

bootcmd is now =bootcmd_distro
I’m using bootcmd=distro_bootcmd

with boot_targets=mmc0 nvme0 dhcp
bootcmd_nvme0=devnum=0; run nvme_boot
kernel_comp_addr_r=0x90000000
kernel_comp_size=0x10000000

and it now works for nvme or mmc
I haven’t got an emmc to test if it can alternatively work

the environment variables are stacking up with a whole lot of patches (cruft)

[coltree]

Unfortunately mmcbootenv doesn’t load uEnv.txt for nvme
much of the scripting is built around mmc

a lot of changes are required to use ${devtype} instead of mmc

in a number of places ${partnum} is hardcoded to 3
locking the u-boot to only boot a distro from partition 3

flash firmware should be more adaptable to different booting requirements

[DavidPlatt]

I’ve independently gone through much of the same exercise today as coltree did, modifying and/or duplicating the mmc-specific boot commands to make them suitable for use with nvme. I agree, a big cleanup of the boot scripts/commands would be very welcome - it’s clear that there’s a lot of “initial development and bringup” logic in there, and not as much generality as would be useful.

Ne’er the less, after a few hours of hacking, I’ve gotten my board so that it can boot directly from QSPI into NVME, with no mmc access at all being required (my SD card is now sitting happily in the storage box).

This took a bit of experimental hackery, since the released versions of the U-Boot payload for QSPI don’t have PCIe/NVMe support yet. I ended up extracting the versions of the SPL and U-Boot from the Wayland .img recently released (used loopback to access the .img file, dd to copy partitions 1 and 2), stripped off some of the redundant zeroes at the ends of each so that spflash didn’t complain about size, and flashed. Amazingly enough, this worked without a single episode of bricking

So, I now have the Wayland image copied to NVME, its SPL and U-boot copied to QSPI, a bunch of new commands and settings added to the QSPI environment, and it seems to boot NVME reliably every time.

My version of this QSPI stuff won’t auto-boot from MMC at all - if I want to do that again I have to either stop the boot process from the console and “run mmcboot” manually, or flip the boot-device switches so that the ROM loads the SPL and U-Boot from the MMC.

[coltree]

in the preboot stage, identify the boot order
then in bootcmd boot from first bootable partition found
proposed change pseudo code-

boot_targets=sd emmc nvme
preboot=for target in boot_targets;
do initialise target device; set envs ${devtype}, ${devnum};
find first boot partition; setenv ${devpart};
load and import environment variables; exit;
done
(exits preboot after loading first uEnv, etc in boot order)
leaves ${devtype} ${devnum} ${devpart} ready for bootcmd
bootcmd=load … ; boot … ;
(or fall through to)
initialise dhcp; boot dhcp

[coltree]

I can now boot from sd, emmc or nvme with a few small changes
not the major changes I feel are needed for a universal u-boot.
it now fails when booting SD sdcard.img from Releases · starfive-tech/VisionFive2 · GitHub
they have forked it…

[danboid]

I have had no luck booting the Starfive Debian ‘69’ image off my NVMe SSD yet by following the instructions in the first post. It starts to boot but it kernel panics before it loads Xorg or gets to the login screen.

I have updated my VF2 to the latest official u-boot and firmware, expanded the root partition to fill the SSD and I’ve adjusted the fstab and extlinux.conf files to use nvme0n1p3 for the root partition instead of the SD card.

Here’s a boot log of me ‘manually’ entering (copy/pasting) the above suggested U-boot commands via a serial connection followed by the kernel panic which seems to be HDMI firmware related?

I can successfully boot the Debian 69 image off SD card fine and I can also access my NVMe disk under Debian 69 when booted from SD card fine but I want to boot Debian directly from NVMe with no SD card boot partition.

This log is from when my VF2 was attached to a 1080p HDMI display that I have tested and know works under Debian 69 when it is booted from a SD card.

https://gist.github.com/danboid/c925494e99121f3df58bc1871df1f4fc

[coltree]

you can’t boot old images with a new u-boot and visa versa
get the new Debian image or Armbian, etc
the info is all up there somewhere
get the uboot prompt and good hunting
have to modify a couple of uboot scripts

[lgold1p]

I would like to try your method and I think others would also. Could you possibly present a step by step method for a learner with limited experience but eager to learn. Thanks much!

[danboid]

Almost there!

I’ve been able to successfully boot Debian sid directly from NVME on my VF2 but only by manually entering these commands into uboot.

I am running the official 2.11.5 u-boot and firmware from here:

Releases · starfive-tech/VisionFive2

I have not updated OpenSBI.

I burned this image onto an NVME disk:

Release 230321-02 - experimental debian sid image for the starfive...

and I expanded the root partition to fill the full SSD. I did not adjust the extlinux.conf or the fstab.

I entered the following commands in u-boot:

nvme scan
fatload nvme 0:3 ${kernel_addr_r} vmlinuz-5.15.0-vf2-104+
fatload nvme 0:3 ${ramdisk_addr_r} initrd.img-5.15.0-vf2-104+
fatload nvme 0:3 ${fdt_addr_r} dtb-5.15.0-vf2-104+/starfive/jh7110-visionfive-v2.dtb
setenv bootargs 'root=/dev/nvme0n1p4 rw console=tty0 console=ttyS0,115200 earlycon rootwait'
setenv kernel_comp_addr_r 0x50000000
setenv kernel_comp_size 0x04000000
booti $kernel_addr_r $ramdisk_addr_r:$filesize $fdt_addr_r

That booti command boots Debian off the NVME. Note I’ve omitted saveenv because these commands don’t work when saved, only entered manually at the uboot prompt. I’ve already tried defining the relevant env names ($bootfile, $ramdiskfile, $fdtfile), adding nvme0 to $boot_targets etc and a few other things but I’ve not hit the winning combo yet.

I’m hoping that someone who understands u-boot and the VF2 boot process better than me can tell me how to configure uboot to make this Debian sid image boot without manual intervention or adding extra boot partitions/disks. I doubt that a new u-boot or opensbi is required to fix this.

4K HDMI output works under sid and it survives apt upgrade too.

[danboid]

I have now got Debian sid automatically booting from my m.2 SSD.

Note that this is a hacky solution because it uses the booti command to load the kernel and initrd instead of using the u-boot sysboot command to parse the extlinux.conf but its a step in the right direction and proof that direct SSD boot is possible with a current image and the current official u-boot/firmware, 2.11.5.

I did not need to modify the default fstab nor the extlinux.conf included with sid. The only modifaction to the standard sid image was to resize the root btrfs partition to fill the disk using gparted.

You might want to run env default -a -f to reset your current u-boot config before running the following u-boot commands:

setenv boot_targets nvme0
setenv preboot 'run chipa_set_uboot;pci enum;nvme scan'
setenv bootargs 'root=/dev/nvme0n1p4 console=tty1 console=ttyS0,115200  debug rootwait  earlycon=sbi'
setenv kernel_comp_addr_r 0x50000000
setenv kernel_comp_size 0x04000000
setenv bootcmd 'fatload nvme 0:3 ${kernel_addr_r} vmlinuz-5.15.0-vf2-104+; fatload nvme 0:3 ${ramdisk_addr_r} initrd.img-5.15.0-vf2-104+; fatload nvme 0:3 ${fdt_addr_r} dtb-5.15.0-vf2-104+/starfive/jh7110-visionfive-v2.dtb; booti $kernel_addr_r $ramdisk_addr_r:$filesize $fdt_addr_r'
saveenv