EMMC boot though spi flash?

Stat_headcrabed · January 7, 2023, 12:06pm

Could we boot from emmc by copying system image data from sdcard to emmc?

dominik · January 26, 2023, 1:55pm

I’m also interested about that,
I burned same on sd card and eMMC and expected that it should start from eMMC selecting right jumper positions, but it could not start for both I’ve got this on UART:

dwmci_s: Response Timeout.
BOOT fail,Error is 0xffffffff

or just silence on other combinations.
Setting two jumpers to right it started successfuly from SD card (but not from eMMC).
I updated uboot and spl but same thing, it could start only from SD

When system is up lsblk shows eMMC as well as nvme drive. I guess that this last one may be able to boot in future, for now I would like to start from eMMC only.

part of uart log:

In:    serial@10000000
Out:   serial@10000000
Err:   serial@10000000
Model: StarFive VisionFive V2
Net:   eth0: ethernet@16030000, eth1: ethernet@16040000
Card did not respond to voltage select! : -110
switch to partitions #0, OK
mmc0(part 0) is current device
bootmode flash device 0
Can't set block device
Failed to load '/boot/uEnv.txt'
Hit any key to stop autoboot:  0
Can't set block device
Importing environment from mmc0 ...
## Warning: Input data exceeds 1048576 bytes - truncated
## Info: input data size = 1048578 = 0x100002
Card did not respond to voltage select! : -110
Couldn't find partition mmc 1:2
Can't set block device
## Warning: defaulting to text format
## Error: "boot2" not defined
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:2...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
823 bytes read in 4 ms (200.2 KiB/s)
U-Boot menu
1:      Debian GNU/Linux bookworm/sid 5.15.0-starfive
2:      Debian GNU/Linux bookworm/sid 5.15.0-starfive (rescue target)
Enter choice: 1:        Debian GNU/Linux bookworm/sid 5.15.0-starfive
Retrieving file: /boot/initrd.img-5.15.0-starfive
9684953 bytes read in 212 ms (43.6 MiB/s)
Retrieving file: /boot/vmlinuz-5.15.0-starfive
8015200 bytes read in 175 ms (43.7 MiB/s)
append: root=/dev/mmcblk1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0
Retrieving file: /boot/dtbs/starfive/starfive_visionfive2.dtb
Failed to load '/boot/dtbs/starfive/starfive_visionfive2.dtb'
kernel_comp_addr_r or kernel_comp_size is not provided!

Stat_headcrabed · February 5, 2023, 3:02am

An official reply about this…

dominik · February 5, 2023, 11:14pm

Ok, so for now latest official debian is not supported with eMMC, I did not expected that, because I never had any problems with other SBCs to boot from this (and it was always available at launch).
For now i copied system into nvme and modified fstabs to run it from there, that is fairy easy but just require some other boot medium (SD card). Maybe some day anything except nvme won’t be needed, for now sd card stays there.

gadams · February 6, 2023, 7:03am

I just worked on this a little this evening. I was able to get the Arch Linux image posted in Arch Linux Image for VisionFive 2 working. I had to adjust the paths in /boot/boot/extlinux/extlinux.conf and /etc/fstab, as expected. What was unexpected was that I had to do:

setenv fdtfile starfive/jh7110-visionfive-v2.dtb

at the uBoot prompt to get past the Failed to load '/boot/dtbs/starfive/starfive_visionfive2.dtb' error.

I’m baffled at why this isn’t needed to boot from micro-SD, but is from eMMC. Did I somehow not image the eMMC card cleanly?

Note that I did not change the boot mode switches on the board. If I do that, then I don’t see the messages on the serial console port.

gadams · February 13, 2023, 8:22am

Aha! I figured out why the fdtfile setting wasn’t working: the SPI flash’s u-boot only reads uEnv.txt from mmc1:2 (the SD card’s partition 2) out of the box. You need to change that in the u-boot env:

setenv fatbootpart 0:2
saveenv

This allows me to boot completely from eMMC. (Well, u-boot is still read from SPI flash.) And setting the fdtfile probably isn’t needed anymore, since it now reads uEnv.txt from the eMMC.

omac777 · February 13, 2023, 12:36pm

I’m trying to follow what you did. You’re talking booting from emmc. You affected only uEnv.txt from the sdcard /dev/mmcblk1p2

Did you add that setenv fatbootpart 0:2 command within the uEnv.txt?
Where did you apply that command?
uboot tool command, if I understood correctly,
implies you have a uart-usb connected to the board
to enter the uboot cmd shell
before the board boots onto the sdcard.
You entered the u-boot prompt: “=>” then ran these commands.

Unfortunately I don’t have the uart-usb dongle to talk in the serial console with the board and apply a similar change for the booting from the nvme.

 # blkid
/dev/nvme0n1p2: LABEL="_/" UUID="d9334329-e2ad-4b72-8f5f-b61406e9d461" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="922b24b0-dc95-441b-92cb-c24ff72c2522"
/dev/mmcblk1p3: LABEL="rootpart" UUID="e17e5f37-48b9-4648-a1de-85c504490ef5" UUID_SUB="f684fe2b-a6b1-4ce0-ba2c-3f23ca9b3f52" BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="f5d0ab68-14dd-5945-bc87-263c1dc052a9"
**/dev/mmcblk1p2**: LABEL_FATBOOT="BOOTPART" LABEL="BOOTPART" UUID="E68A-7FF7" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="2400026e-23d5-4948-b4cc-e2fbde3ac380"
/dev/mmcblk1p1: PARTUUID="442e8885-bcbf-6c4f-b6a4-ffe20e5b5ee5"

I’m trying to boot from nvme in a similar manner. I haven’t modified anything yet.

Here are my current relevant mountpoints:

 # mount  

/dev/mmcblk1p2 on /boot type vfat (rw,nosuid,nodev,relatime,fmask=0000,dmask=0000,allow_utime=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,flush,errors=remount-ro)

/dev/mmcblk1p3 on / type btrfs (rw,noatime,nodiratime,compress-force=zstd:3,ssd,space_cache=v2,subvolid=5,subvol=/)

/dev/nvme0n1p2 on /mnt type ext4 (rw,relatime)

Here’s my uEnv.txt:

 # cat /boot/boot/uEnv.txt 
fdt_high=0xffffffffffffffff
initrd_high=0xffffffffffffffff
kernel_addr_r=0x44000000
kernel_comp_addr_r=0x90000000
kernel_comp_size=0x10000000
fdt_addr_r=0x48000000
ramdisk_addr_r=0x48100000
# Move distro to first boot to speed up booting
boot_targets=distro mmc0 dhcp 
# Fix wrong fdtfile name
fdtfile=starfive/jh7110-visionfive-v2.dtb
# Fix missing bootcmd
bootcmd=run bootcmd_distro

Here is my extlinux.conf:

 # cat /boot/boot/extlinux/extlinux.conf
TIMEOUT 30

DEFAULT visionfive2
 
MENU TITLE starfive visionfive2 boot options

LABEL visionfive2
      MENU LABEL visionfive2
      LINUX ../../vmlinuz-5.15.0-vf2-260+
      INITRD ../../initrd.img-5.15.0-vf2-260+
      FDTDIR ../../dtb-5.15.0-vf2-260+
      APPEND root=LABEL=rootpart rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0 apparmor=0

gadams · February 14, 2023, 6:27am

Ah! Sorry; I should have been more clear.

Yes, I’m talking to the UART on pins 6, 8, and 10 of the GPIO header. (I highly recommend you pick up a USB-serial interface. They’re handy for talking to all sorts of devices. They’re dirt cheap on AliExpress, or in a hurry, Amazon carries AdaFruit’s version that the Chinese ones copy.)

You need to set in the U-Boot environment variables the location of the partition that contains the uEnv.txt file. The U-Boot variables are stored in the SPI flash (if I’m not mistaken).

There may be a way to write the U-Boot environment variables using some fancy flashcp incantations, but that seems pretty risky. (If you mess up, you’ll have to boot over UART, and that’ll definitely require a USB UART dongle.) I’m not sure how else to set these variables.

Here’s what I did:

Set up a bootable SD card. I used cwt’s Arch Linux image.
Set up the GPT partition table on the eMMC device using fdisk.
Copy the SD card’s partition 2 (/dev/mmcblk1p2) and 3 (/dev/mmcblk1p3) to the eMMC device (/dev/mmcblk0p2 and /dev/mmcblk0p2).
a. On the Arch image, there are multiple BTRFS subvolumes in partition 3, so there aren’t any more partitions. Nice and easy. Copy the rest if you have more.
b. I used dd to copy the EFI partition (partition 2) because I set them to identical sizes, and I used btrfs send / btrfs receive to copy partition 3 to save time and not to have to worry about the slightly smaller partition size on my eMMC device.
Edit the boot/extlinux/extlinux.conf in the eMMC /boot partition. (This will likely be mounted on /mnt/boot/boot, if you do it similarly to how I did.)
You do not need to edit uEnv.txt. You just need to arrange for it to be read from the eMMC device. That’s what step 8 does.
Reboot.
Remove the SD card after it’s synced but before it starts up. (You could just shut down, remove the SD card, and power up again.)
Using the UART port, interrupt the boot process by pressing any key early on. Then issue these commands to U-Boot:
a. setenv fatbootpart 0:2
b. saveenv
Continue booting. (Type boot; hit the reset button; whatever.)

Now, all that said, I haven’t tried getting it to work with the EFI partition on NVMe. This may or may not work with the U-Boot currently in SPI flash. It’s easy enough to have your root filesystem on NVMe though; just use the appropriate root path in extlinux.conf. I’m doing that now.

Your uEnv.txt looks identical to mine. It probably doesn’t need to change.

However, your extlinux.conf file is a little different. Mine comes from the Arch build, as I noted. It looks like this:

default Arch-cwt6-server
menu title U-Boot menu
prompt 0
timeout 50

label Arch-cwt6-server
	menu label Arch Linux 5.15.0-cwt6-server
	linux /boot/vmlinuz-5.15.0-cwt6-server
	initrd /boot/initrd.img-5.15.0-cwt6-server
	fdtdir /boot/dtbs/
	append root=/dev/nvme0n1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 rootflags=defaults,discard=async,compress=lzo,user_subvol_rm_allowed,subvol=arch-minimal

label Arch-cwt6-server-single
	menu label Arch Linux 5.15.0-cwt6-server (Single User Mode)
	linux /boot/vmlinuz-5.15.0-cwt6-server
	initrd /boot/initrd.img-5.15.0-cwt6-server
	fdtdir /boot/dtbs/
	append root=/dev/nvme0n1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 rootflags=defaults,discard=async,compress=lzo,user_subvol_rm_allowed,subvol=arch-minimal single

I wonder how those ../../ paths work in yours to locate the boot image files. In any event, I think you’ll need to change the root boot parameter to root=/dev/nvme0n1p3 to have, say, the EFI boot partition on eMMC and root on NVMe.

Oh, and I’m using the latest build of U-Boot (2021.10) in SPI flash. That may or may not matter.

gadams · February 14, 2023, 7:34am

This post suggests that there may be another way to write the U-Boot environment variables, but it seems like there might be issues with it. I haven’t looked into this at all; proceed with caution!

mzs · February 14, 2023, 8:38am

The problem I noticed is that the partition table on the QSPI NOR FLASH from Linux is seen as:

|device name|Device offset|Env. size |Flash sector size|Label|
|-----------|-------------|----------|-----------------|-----|
|/dev/mtd1  |0x0000000    |0x00020000|0x00020000       |SPL  |
|/dev/mtd2  |0x0100000    |0x00300000|0x00300000       |UBOOT|
|/dev/mtd3  |0x0F00000    |0x01000000|0x01000000       |DATA |

But the way that it is actually used by U-Boot according to the “JH7110Boot User Guide (page 8)” is:

|device name|Device offset|Env. size |Flash sector size|Contents                         |
|-----------|-------------|----------|-----------------|---------------------------------|
|           |0x0000000    |0x00080000|0x00080000       |SPL                              |
|           |0x00F0000    |0x00010000|0x00010000       |U-Boot environment variables     |
|           |0x0100000    |0x00400000|0x00400000       |fw_payload.img (OpenSBI + U-Boot)|
|           |0x0600000    |0x01000000|0x01000000       |Reseved                          |

So from Linux there is currently no raw partition to access the U-Boot environment variables using “libubootenv”. You could create a new partition that points to the U-Boot environment variables and create a /etc/fw_env.config file that point to that, but probably far safer to wait on the next image to arrive which will probably align the partitions with the documentation, although I just spotted a typo, but it is nothing important.

cobalt · February 14, 2023, 2:46pm

Hmm, but linux reports yet other sizes:

# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00020000 00001000 "spl"
mtd1: 00300000 00001000 "uboot"
mtd2: 00100000 00001000 "data"

Note mtd2 is smaller than mtd1 here. While your figures would allow to store a trimmed down kernel into the nor flash, it would not be possible, if /proc/mtd info above were right.

@msz, I would also love to get / set the environment via fw_printenv, but i’m not certain about the geometry, yet. Unfortunately, digging into the u-boot or kernel source does not help much, if the figures there are not right w.r.t. the last partition. One could look at the chips specs, though. Where did you get the sector sizes from, btw?

cobalt · February 14, 2023, 4:11pm

Hmm, i digged into the question of booting without SD earlier, too, but came to the conclusion, that this is not possible with the current u-boot. I have checked out the JH7110_VisionFive2_devel branch here.

The point is, that there is no PCI support configured there, which would then allow to configure the NVMe driver (PLDA XpressRICH3-AXI PCIe controller), which is simply not ported, both due to the sources and to the upstream plan, where the driver is listed “WIP”.

Now the “fatbootpart” above gets its meaning in the u-boot environment by bootcmd, which finally ends in sysboot mmc ${fatbootpart} fat .... Thus sysboot uses the mmc subsystem (i.e. SD), but to it should use the nvme subsystem, which is not configured and not configurable, until the driver is present.

So i simply do not understand, how, what you report, can technically be possible.

Hmm, i tried with u-boot command line:

StarFive # mmc list
sdio0@16010000: 0
sdio1@16020000: 1 (SD)

so yes, there is some device 0 in the mmc subsystem. But i was not able to get much further with it:

StarFive # mmc dev 0
Card did not respond to voltage select! : -110

SunWukong · February 14, 2023, 4:12pm

The RVspace Doc Center is a good place to look for information.
There you will also find the following, for example: Software SDK Developer Guide for U- Boot

cobalt · February 15, 2023, 7:03am

Hmm, i checked through the docs and found that the device has both an eMMC and a M.2 socket, which i were confusing above. I have no emmc device installed, so it is not listed.

cobalt · February 15, 2023, 1:45pm

I dug a into the issue and fortunately it can be handled without recompiling the kernel.

The partition table for mtd is defined in the device tree, originally in the linux kernel source in

/linux/arch/riscv/boot/dts/starfive/jh7110.dtsi

as

partitions {
     compatible = "fixed-partitions";
     #address-cells = <1>;
     #size-cells = <1>;

     spl@0 {
             reg = <0x0 0x20000>;
     };
     uboot@100000 {
             reg = <0x100000 0x300000>;
     };
     data@f00000 {
             reg = <0xf00000 0x100000>;
      };

Unfortunately, this is outdated or was never right.

@mzs posted hopefully better table based on information in JH7110 Boot User Guide and found a glitch there, too.

Now since the device table is separate (binary) file in the /boot directory, the table can be fixed there as a workaround. The procedure is as follows:

# apt-get install device-tree-compiler # get dtc
# cd /boot/boot/dtbs/starfive
# cp jh7110-visionfive-v2.dtb jh7110-visionfive-v2.dtb.orig
# dtc -I dtb -O dts jh7110-visionfive-v2.dtc > jh7110-visionfive-v2.dts # convert to source
# vi jh7110-visionfive-v2.dts # replacing the section as below
# dtc -I dts -O dtb jh7110-visionfive-v2.dts > jh7110-visionfive-v2.dtb # convert to binary
# reboot

The section in question need to be replaced by

partitions {
  compatible = "fixed-partitions";
  #address-cells = <0x01>;
  #size-cells = <0x01>;

  spl@0 {
    reg = <0x00 0x80000>;
  };

  ubootenv@0 {
    reg = <0xF0000 0x10000>;
  };

  payload@100000 {
    reg = <0x100000 0x400000>;
  };

  reserved@600000 {
    reg = <0x600000 0xa00000>;
  };
};

which is hopefully a bit better.

After the reboot, try

# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00080000 00001000 "spl"
mtd1: 00010000 00001000 "ubootenv"
mtd2: 00400000 00001000 "payload"
mtd3: 00a00000 00001000 "reserved"

(Note that the erase sizes shown differ from @mzs 's table.)

Now the u-boot environment should be accessible by fw_printenv. Install the utility using apt-get install libubootenv-tool. The utility must be configured. Create a file /etc/fw_env.config containing:

# MTD device name       Device offset   Env. size       Flash sector size       Number of sectors
/dev/mtd1               0x00000         0x10000         0x10000

Now as good as i understand, the u-boot environment is not persistent by default. Thus fw_printenv would show you an empty environment.

Being connected to the UART, reboot and enter the u-boot dialog:

StarFive # env save

Now /dev/mtd1 is populated and will be used by u-boot. Boot into linux and try:

# fw_printenv
baudrate=115200
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
...
ver=U-Boot 2021.10 (Jan 08 2023 - 18:04:54 +0800)
visionfive2_mem_set=fdt memory ${memory_addr} ${memory_size};

Parts of the configuration is also /etc/u-boot-initial-env, which can best be produced now:

# fw_printenv > /etc/u-boot-initial-env

If anything fails, the original configuration can be restored in u-boot using

StarFive # env default -a

Note that the utility contains a fw_setenv command, too.

Hope it helps, good luck!

Stat_headcrabed · February 15, 2023, 5:35pm

Since important information like MAC address are storaged inside external eeprom, I think we could completely re-arrange spi partition layout. I think this also might help solve problems happened on current opensuse image:https://en.opensuse.org/HCL:VisionFive2#Manual_Installation
Also, we could separate OpenSBI image and U-Boot to update them independently.

cobalt · February 15, 2023, 7:52pm

Hmm, while paging through the u-boot env and the related processes, i came over
https://u-boot.readthedocs.io/en/latest/develop/distro.html
I wonder how common u-boot related kernel installations and upgrades currently are. I’ve seen a u-boot-update script in Debian that hooks into the regular kernel installion. Does SuSE have a likely infrastructure or is this all new ground for everyone?

Perhaps this and the next entry in the blog below might be of interest for this topic,
https://blog.mobian.org/posts/2023/01/25/efficient-tooling/
since many distributions struggled to cope commonly with SPI and U-BOOT installations.
https://blog.mobian.org/posts/2022/03/30/universal-images/

dominik · March 1, 2023, 12:57pm

With new image of debian I just was able to boot without sd card
(but just once )

Michael.Zhu · March 1, 2023, 1:46pm

please make sure you press&hold more than 3 second

dominik · March 1, 2023, 2:38pm

this is just hard reset - correct?
I just reinstalled image on eMMC and after all it’s booting again, I’ll play now with that to see if it’s everything ok. Eventually I would like to start from m.2 slot only, for now sdcard-less booting is just better than earlier.