Debian - Move root partition to NVMe and keep boot partition

cool, could you share a bit more detail on this?

Hi @tommythorn,

to the point of using the stock debian-ports, i think the commands were:

# mount /dev/nvme0n1p1 /mnt
# debootstrap --keyring /usr/share/keyrings/debian-ports-archive-keyring.gpg unstable /mnt https://deb.debian.org/debian-ports

As you see, just using debian-ports and the corresponding keyring does the trick.

The keyring is from the debian-ports-archive-keyring package. Hmm, you may need to fetch and install it from https://deb.debian.org/debian-ports, before. Please see e.g. this and the following posts to be aware of some minor troubles ahead.

Apparently the kernel is still fresh enough to run debian-unstable. I did not try to get the X-server going, only a command line debian, which does operate just as usual. You should have a terminal (screen) attached to the a serial line of the device when booting into the partition. This figure shows how to attach to the serial line.

Please feel free to ask if you get stuck somewhere.

I also managed to compile u-boot on the VF2 but did not yet dare to install it.

2 Likes

Hmm, i dug a little into the u-boot sources and found, that a boot partition apparently needs to be kept on the sd-card for now even if a nvme device is present.

The reason is, that the mtd device (cat /proc/mtd) is too small to contain the linux kernel and the current u-boot cannot read the kernel from nvme. While u-boot does have a nvme subsystem, it may lack a suitable driver in the moment. Please correct me, if i’m wrong here.

This does not preclude to use the sd slot normally. One can savely unmount /boot if the sd slot is really needed, since the boot partition on the sd is only needed during the u-boot stage.

1 Like

Here are the details about my new nvme.

root@vf2-sid:/home/linux# nvme list
Node                  Generic               SN                   Model                                    Namespace Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
dev nvme0n1           dev ng0n1            OSC23010900000000014 OSC PCIe 256GB                           1         256.06  GB  256.06  GB    512   B +  0 B   T0918A0L
This is an OSCOO ON900 m.2 nvme with pcie Gen3 4-lanes. It has NCQ and SMART support and low power consumption with TRIM support.
https://www.aliexpress.com/item/1005004601159371.html

Working hard to get this nvme up and chrooted. Here are my details up to the debootstrap.

root@vf2-sid:/home/linux# apt-get install nvme-cli

root@vf2-sid:/home/linux# nvme list
Node                  Generic               SN                   Model                                    Namespace Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            OSC23010900000000014 OSC PCIe 256GB                           1         256.06  GB / 256.06  GB    512   B +  0 B   T0918A0L

root@vf2-sid:/home/linux# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0    31:0    0   128K  0 disk 
mtdblock1    31:1    0     3M  0 disk 
mtdblock2    31:2    0     1M  0 disk 
mmcblk1     179:0    0  29.5G  0 disk 
|-mmcblk1p1 179:1    0    16M  0 part 
|-mmcblk1p2 179:2    0   512M  0 part /boot
`-mmcblk1p3 179:3    0    29G  0 part /
nvme0n1     259:0    0 238.5G  0 disk 

root@vf2-sid:/home/linux# parted /dev/nvme0n1
GNU Parted 3.5
Using /dev/nvme0n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt                                                      
(parted) mkpart                                                           
Partition name?  []? /boot/efi                                            
File system type?  [ext2]? vfat                                           
parted: invalid token: vfat
File system type?  [ext2]? fat                                            
parted: invalid token: fat
File system type?  [ext2]? fat32                                          
Start? 100M                                                               
End? 700M                                                                 
(parted) mkpart                                                           
Partition name?  []? /boot                                                
File system type?  [ext2]? ext4                                           
Start? 798M                                                               
End? 2048M                                                                
(parted) mkpart                                                           
Partition name?  []? /                                                    
File system type?  [ext2]? ext4                                           
Start? 2056M                                                              
End? 254G                                                                 
(parted) print                                                            
Model: OSC PCIe 256GB (nvme)
Disk /dev/nvme0n1: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name       Flags
 1      99.6MB  700MB   601MB                /boot/efi  msftdata
 2      798MB   2048MB  1250MB  ext4         /boot
 3      2056MB  254GB   252GB   ext4         /

(parted) quit                                                             
Information: You may need to update /etc/fstab.

root@vf2-sid:/home/linux# mkfs.vfat /dev/nvme0n1p1
root@vf2-sid:/home/linux# mkfs.ext4 /dev/nvme0n1p2
root@vf2-sid:/home/linux# mkfs.ext4 /dev/nvme0n1p3
root@vf2-sid:/home/linux# sync
root@vf2-sid:~# mount /dev/nvme0n1p3 /mnt
root@vf2-sid:~# apt-get remove debootstrap
cd /root/
git clone https://salsa.debian.org/installer-team/debootstrap.git
cd debootstrap/
export DEBOOTSTRAP_DIR=`pwd`
echo $DEBOOTSTRAP_DIR
./debootstrap --verbose --keyring /usr/share/keyrings/debian-ports-archive-keyring.gpg sid /mnt https://deb.debian.org/debian-ports

#ok get into the chroot
chroot /mnt

wget https://deb.debian.org/debian-ports/pool/main/d/debian-ports-archive-keyring/debian-ports-archive-keyring_2023.02.01_all.deb
dpkg -i debian-ports-archive-keyring_2023.02.01_all.deb

apt-get update
apt-get upgrade

#get out of the chroot
exit

The chroot /mnt is now behaving well enough to build stuff with on the nvme. It’s certainly faster than my old sdcard for sure.

cat /boot/boot/extlinux.conf
cat: /boot/boot/extlinux.conf: No such file or directory

Solution:

cat /boot/boot/extlinux/extlinux.conf 
TIMEOUT 30

DEFAULT visionfive2
 
MENU TITLE starfive visionfive2 boot options

LABEL visionfive2
      MENU LABEL visionfive2
      LINUX ../../vmlinuz-5.15.0-vf2-260+
      INITRD ../../initrd.img-5.15.0-vf2-260+
      FDTDIR ../../dtb-5.15.0-vf2-260+
      APPEND root=LABEL=rootpart rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0 apparmor=0
1 Like

Congratulation. And yes, the system feels much faster with nvme. :smiley:

Since you write “behaving well enough”, my question is, whether you get I/O problems with the nvme? Unfortunately, for now, i’m so plagued by it, that i cannot make the system self-hoisting.

Can you please try to git clone https://github.com/starfive-tech/VisionFive2.git?

Since there are only very few notes about NVMe I/O timeouts, i wonder, if it might is a local issue that only pesters me.

1 Like

I’ll give it a go to clone vf2 repo.

chroot /mnt
apt-get install git

The following error is because I didn’t mount the other stuff before chrooting. that’s what I meant by well-enough.
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)

All good for this second git clone. Please recall I did a git clone of debootstrap because the debian-ports debootstrap .deb wasn’t completing on the /mnt successfully. The debootstrap was done straight from sources.

git clone https://github.com/starfive-tech/VisionFive2.git
Cloning into 'VisionFive2'...
remote: Enumerating objects: 4401, done.
remote: Counting objects: 100% (525/525), done.
remote: Compressing objects: 100% (267/267), done.
remote: Total 4401 (delta 282), reused 491 (delta 258), pack-reused 3876
Receiving objects: 100% (4401/4401), 290.57 MiB | 13.49 MiB/s, done.
Resolving deltas: 100% (2408/2408), done.

within the chroot, I installed hdparm:
apt-get install hdparm

Then ran the test:
hdparm -tT /dev/nvme0n1
/dev/nvme0n1: No such file or directory LOL that’s because of those required pseudo filesystems mountpoints before chrooting. brb

BOTTOM LINE: mount those pseudo filesystems before chrooting.
IMPORTANT NOTE: the daemon-reload is necessary to mount in the specified order immediately. The order is important.
So as a point of reference: Repair the Bootloader - System76 Support
It does the following 2 step chroot:

for i in dev dev/pts proc sys run; do mount -B /$i /mnt/$i; systemctl daemon-reload; done
chroot /mnt

NOTE AFTER USING CHROOT: Please remember to unmount the pseudo filesystems.
IMPORTANT NOTE: the daemon-reload is necessary to unmount in the specified order immediately. The order is important.
for i in dev dev/pts proc sys run; do umount /mnt/$i; systemctl daemon-reload; done
You might need to run the unmount twice.

There. Now we won’t get any annoying issues for the pseudo filesystems in the chroot anymore.
Let’s try that hdparm test on the nvme drive again.

hdparm -tT /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   1352 MB in  2.00 seconds = 676.26 MB/sec
 Timing buffered disk reads: 494 MB in  3.01 seconds = 164.18 MB/sec

versus the sdcard:

hdparm -tT /dev/mmcblk1

/dev/mmcblk1:
 Timing cached reads:   1242 MB in  2.00 seconds = 620.39 MB/sec
 Timing buffered disk reads:  66 MB in  3.03 seconds =  21.81 MB/sec

unmount the pseudo filesystems
for i in dev dev/pts proc sys run; do umount /mnt/$i; systemctl daemon-reload; done
You might need to run the unmount twice.

Thank you, @omac777, for making the test. Absolutely impossible for me, to get the git clone of the kernel through, without the driver or nvme locking up at some point. So it must be some local reason.

Since you did not make any trickery with kernel options or anything, and i do not believe, that it is the particular brand of the nvme, my suspicion is the power supply. My power source claims to deliver up to 2.4 A, i.e. 10 W something, but that’s also a matter of power negotiation. I can well imagine, that running the nvme under high load draws more power.

Are you using a decent USB-C power supply, perhaps?

You are right. If you don’t have a usb type c charger that is powerful enough, then you will have issues.

I ordered and just received a power supply purposed exactly to provide for the nvme.

I also got this tiny fan that sticks on top of the cpu. Perfect size imho. I used and exacto knife to trim it on the sides to make it fit into the seat of the motherboard since I paid no attention to the connectors, but luckily it all worked out LOL

1 Like

I chose my SSD based on its low power consumption, as I feared that power consumption could be an issue.

That’s the Patriot P300. As expected, it works w/o issue. It’s also quite inexpensive.

Overall, Patriot’s P300 is fairly efficient, nearly matching the SX8200 in performance per watt. It consumes the least amount of power out of all other SSDs in our test pool, sipping just over 2.2W and peaking at 3.3W under concurrent small and large block sequential reading/writing.

The Patriot drive also supports APST, ASPM, and L1.2 power saving modes. On our desktop testbench, the SSD couldn’t hit its lowest idle state, but fell to a respectable 40mW when ASPM was enabled. When disabled or when active, P300 consumes about 10x the amount, lower than the rest of the pool once again.

24€ for 256GB, 120€ for 2TB (got this).

My supply is a 20w usb-pd.

1 Like

Positively. I found a 18W USB-C power supply in my space junk and that was it.

Hmm, i asked for the wrong git repository, btw. it was the linux kernel, really, that caused the problems. But now (running in a root partition on nvme):

root@starfive:/opt# git clone https://github.com/starfive-tech/linux.git
Cloning into 'linux'...
remote: Enumerating objects: 9214172, done.
remote: Counting objects: 100% (133027/133027), done.
remote: Compressing objects:  25% (11283/44939)
remote: Compressing objects: 100% (44939/44939), done.
Receiving objects:  11% (1087157/9214172), 475.80 MiB | 5.59 MiB/s
remote: Total 9214172 (delta 98043), reused 107310 (delta 87780), pack-reused 9081145
Receiving objects: 100% (9214172/9214172), 2.08 GiB | 7.93 MiB/s, done.
Resolving deltas: 100% (7811060/7811060), done.
Checking objects: 100% (33554432/33554432), done.
Updating files: 100% (78785/78785), done.

I have no heat sink attached yet and sensors reported a maximum of +72.4°C. I think is was perhaps more the cpus than the nvme device, that demanded the power in this case.

A too weak power supply this is really a mean trap, as first, everything looks good and the problem then shows up as something different. :smiling_imp:

I’ve seen weirdness like this around dust/static on old motherboards/gpus.
Make sure you power dust your sbc every once in a while as well.

The /etc/network/interfaces wasn’t being used since my network is on end0.

cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
#source-directory /etc/network/interfaces.d

auto lo
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet dhcp

NetworkManager does everything now.

 # nmcli -f NAME,DEVICE,FILENAME connection show
NAME                DEVICE  FILENAME                                                               
Wired connection 1  end0    /run/NetworkManager/system-connections/Wired connection 1.nmconnection 
Ifupdown (eth0)     --      --                                                                     
Wired connection 2  --      /run/NetworkManager/system-connections/Wired connection 2.nmconnection
 # cat "/run/NetworkManager/system-connections/Wired connection 1.nmconnection"
[connection]
id=Wired connection 1
uuid=04d0195f-616e-3f7c-a28f-5e74de071df9
type=ethernet
autoconnect-priority=-999
interface-name=end0
timestamp=1676149361

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=default
method=auto

[proxy]

[.nmmeta]
nm-generated=true

This one is off but is good for the second ethernet port:

 # cat "/run/NetworkManager/system-connections/Wired connection 2.nmconnection"
[connection]
id=Wired connection 2
uuid=e077f27d-633b-38db-a2e8-85508f86ac6b
type=ethernet
autoconnect-priority=-999
interface-name=end1
timestamp=1676149361

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=default
method=auto

[proxy]

[.nmmeta]
nm-generated=true

I’m looking forward to getting that usb-to-uart thingy because I can’t connect via ssh after changing the /boot/boot/extlinux.conf

The network config are identical sdcard/nvme. The /etc/fstab is fine. ntpsec and openssh-server/client are installed.
I’m stumped.

Also journalctl is showing me logs from Jan 30th, but nothing afterwards. I looked at my conf, but everything is set to defaults.

I have had the least trouble with cp2102 and ft232r based boards, relative to other cheap chips (ch340, pl23xx). They are quite inexpensive in aliexpress, and are good up to 1mbit/s.

The fancy alternative is ft232h or ft2323h. These can do 12mbit/s and have some useful extra functionality (jtag, i2c, spi, bitbang).

Well, yes, that might be the problem. I renamed ‘eth*’ to ‘end*’, and activated the previously outcommented ‘iface’ lines. That was all. I do not have the network manager installed, but never had much problems with it.

Otherwise the only suspicious line i see in what your post above, is in nmcli -f ..., where the network manager says ‘end0’ but ‘Ifupdown (eth0)’. Thus the network manager is unsure about the device name at least when printing the line.

Yes, get one. This will allow to find the problem in any case. I also use it to ask for ip addr. Please make sure you ssh to the right address as this might vary between the images. Try to see the dhcp negotion using tcpdump on your work station while booting the board. If there is none, no ‘ifup’ on the interface has happened.

I think you should change (or even duplicate) the names in ‘/etc/network/interfaces’ on the nvme image. Make sure, the iface lines are active. Hmm, i have copied the file over from the sd image to nvme.

(Hmm, renaming away from ‘eth0’ is suppressed somewhere in the mmc image, but i did not figure out where.)

Hope it helps, good luck.

1 Like

Using a 32G microsd,
expand sdx3 to full size
put starfive…img file in /root
boot
dd if=starfive…img of=/dev/nvme0n1 bs=4M
expand nvme0n1p3 to full size
edit /boot/boot/extlinux/extlinux.conf
change the append line to nvme0n1p3
mount /dev/nvme0n1p3 /mnt
edit /mnt/etc/fstab
/dev/nvme0n1p3 / ext4 …
reboot into nvme drive
rescue boot still boots microsd

3 Likes

I haven’t gotten it to boot from nmve either, but gadams succeeded to boot off of emmc.

Maybe this could work, but to tweak this part within uboot, you need a usb-uart dongle which I don’t have yet.
I’m hopeful it will help.

1 Like