Kernel build ok but systemd fails to load

I have successfully built the kernel following the instructions in the release note for 202306, but then it fails to mount the filesystems for systemd and doesn’t complete the boot.

I have followed the instructions by the book, without any change, and namely:

apt-get install build-essential linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev dwarves bison git
git clone https://github.com/starfive-tech/linux
git checkout v5.1.3
cd linux/
cp arch/riscv/configs/starfive_visionfive2_defconfig .config
make ARCH=riscv olddefconfig
make ARCH=riscv -j4 bindeb-pkg
cd ..
dpkg -i *.deb

I run the build on the VF2 directly, no cross-build, on an NVME ssd.

There is a problem with the generated ‘/boot/extlinux/extlinux.conf’, which lists 2 ‘root=’ attributes for each kernel cmdline, eg:
append root=/dev/nvme0n1p4 root=/dev/mmcblk1p4 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0 single
and this made the boot fail. So I have removed the second one with:

sudo sed -e 's|root=/dev/mmcblk1p4 ||' /boot/extlinux/extlinux.conf -i

then the boot started properly:

Retrieving file: /extlinux/extlinux.conf
1341 bytes read in 4 ms (327.1 KiB/s)
U-Boot menu
1:      Debian GNU/Linux bookworm/sid 5.15.0gci001
2:      Debian GNU/Linux bookworm/sid 5.15.0gci001 (rescue target)
3:      Debian GNU/Linux bookworm/sid 5.15.0-starfive
4:      Debian GNU/Linux bookworm/sid 5.15.0-starfive (rescue target)
Enter choice: 1:        Debian GNU/Linux bookworm/sid 5.15.0gci001
Retrieving file: /initrd.img-5.15.0gci001
13826124 bytes read in 77 ms (171.2 MiB/s)
Retrieving file: /vmlinuz-5.15.0gci001
8489069 bytes read in 59 ms (137.2 MiB/s)
append: root=/dev/nvme0n1p4 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0
Retrieving file: /dtbs/starfive/jh7110-visionfive-v2.dtb
50235 bytes read in 5 ms (9.6 MiB/s)
   Uncompressing Kernel Image
Moving Image from 0x44000000 to 0x40200000, end=419b2000
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Using Device Tree in place at 0000000048000000, end 000000004800f43a

Starting kernel ...
...
[    0.000000] Linux version 5.15.0gci001 (user@starfive) (gcc (Debian 12.2.0-10) 12.2.0, GNU ld (GNU Binutils for Debian) 2.39.50.20221224) #2 SMP Tue Jul 4 22:00:24 CEST 2023

the entire initramfs part worked well, and then the control passed to systemd, where the problems appear:

Welcome to    12.644292] systemd[1]: Hostname set to <starfive>.
1mDebian GNU/Linux bookworm/sid!

[   13.050599] systemd[1]: Queued start job for default target Graphical Interface.
[   13.066845] systemd[1]: Created slice Slice /system/getty.
[  OK     13.073447] systemd[1]: Created slice Slice /system/modprobe.
0m] Created slice Slice /system/getty.
[  OK     13.087162] systemd[1]: Created slice Slice /system/serial-getty.
0m] Created slice Slice /system/modprob[   13.097993] systemd[1]: Created slice User and Session Slice.
e.
[   13.104283] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
m  OK  ] Cre[   13.113641] systemd[1]: Arbitrary Executable File Formats File System Automount Point was skipped because of an unmet condition check (ConditionPathExists=/proc/sys/fs/binfmt_misc).
ated slice    13.131558] systemd[1]: Reached target Local Integrity Protected Volumes.
;39mSlice /syste[   13.139271] systemd[1]: Reached target User and Group Name Lookups.
m/serial-getty   13.146730] systemd[1]: Reached target Remote File Systems.
0m.
[  [   13.153638] systemd[1]: Reached target Slice Units.
OK  ] Create[   13.159851] systemd[1]: Reached target Swaps.
d slice    13.165533] systemd[1]: Reached target Local Verity Protected Volumes.
mUser and Sessio[   13.173946] systemd[1]: Listening on initctl Compatibility Named Pipe.
n Slice.
[  OK  ] Started Forward Password R�…uests to Wall Directory Watch.
[  OK  ] Reached target[   13.193207] systemd[1]: Journal Audit Socket was skipped because of an unmet condition check (ConditionSecurity=audit).
 Local Integrity Protec[   13.205447] systemd[1]: Listening on Journal Socket (/dev/log).
ted Volumes.[   13.213896] systemd[1]: Listening on Journal Socket.

[  OK [   13.220220] systemd[1]: Listening on udev Control Socket.
 ] Reached t[   13.226754] systemd[1]: Listening on udev Kernel Socket.
arget User and Group Na[   13.234730] systemd[1]: Mounting Huge Pages File System...
me Lookups.
[  OK  [   13.243006] systemd[1]: Mounting POSIX Message Queue File System...
] Reached target Re[   13.252038] systemd[1]: Mounting Kernel Debug File System...
mote File System[   13.258907] systemd[1]: Kernel Trace File System was skipped because of an unmet condition check (ConditionPathExists=/sys/kernel/tracing).
s.
[  OK  ] Reached target    13.275781] systemd[1]: Starting Create List of Static Device Nodes...
1;39mSlice Units.
[  OK  ] Reac[   13.285945] systemd[1]: Starting Load Kernel Module configfs...
hed target Swaps.
[  OK    13.295761] systemd[1]: Starting Load Kernel Module drm...
[0m] Reached target Local Verity Protec[   13.305274] systemd[1]: Starting Load Kernel Module efi_pstore...
ted Volumes.
[  OK  ] Listening[   13.315548] systemd[1]: Starting Load Kernel Module fuse...
 on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on[   13.338161] systemd[1]: Starting Journal Service...
 udev Control Socket.
[  OK  ] Listening on udev Kernel Socket.
         Mounting Huge Pages File System...
         Mounting POSIX Message Queue File System   13.361523] systemd[1]: Starting Load Kernel Modules...
0m...
         Mounting Kernel Debug File System...
         Starting Create List of Static Device Nodes...
         Starting Load Kernel Module configfs...
         Starting    13.385062] systemd[1]: Starting Remount Root and Kernel File Systems...
0;1;39mLoad Kernel Module drm...
         Starting Load Kernel Module efi_pstore...
         Starting Load Kernel Module fuse...
         Starting Journal Service...
       [   13.406177] systemd[1]: Repartition Root Disk was skipped because no trigger condition checks were met.
  Starting Load Kernel Modules...
         Starting Remount Root and Kernel File Systems[   13.419052] systemd[1]: Starting Coldplug All udev Devices...
...
         Starting Coldplug All udev Devices...
[   13.457644] systemd[1]: dev-hugepages.mount: Mount process finished, but there is no mount.
[   13.457719] systemd[1]: dev-hugepages.mount: Failed with result 'protocol'.
[   13.458921] systemd[1]: Failed to mount Huge Pages File System.
[FAILED] Failed to mount Huge Pages File System.
[   13.507832] systemd[1]: dev-mqueue.mount: Mount process finished, but there is no mount.
[   13.507887] systemd[1]: dev-mqueue.mount: Failed with result 'protocol'.
[   13.508805] systemd[1]: Failed to mount POSIX Message Queue File System.
See 'systemctl status dev-hugepages.mount' for details.
[FAILED] Failed to mount POSIX M[   13.568917] systemd[1]: sys-kernel-debug.mount: Mount process finished, but there is no mount.
essage Queue File System.
[   13.568943] systemd[1]: sys-kernel-debug.mount: Failed with result 'protocol'.
[   13.569372] systemd[1]: Failed to mount Kernel Debug File System.
[   13.628667] systemd[1]: Started Journal Service.
See 'systemctl status dev-mqueue.mount' for details.
[FAILED] Failed to mount Kernel Debug File System.
See 'systemctl status sys-kernel-debug.mount' for details.
[  OK  ] Started Journal Service.
[  OK  ] Finished Create List of Static Device Nodes.
[  OK  ] Finished Load Kernel Module configfs.
[  OK  ] Finished Load Kernel Module drm.
[  OK  ] Finished Load Kernel Module efi_pstore.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished Load Kernel Modules.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Mounting FUSE Control File System...
         Mounting Kernel Configuration File System...
         Starting Flush Journal to Persistent Storage...
         Starting Load/Save Random Seed...
         Starting Apply Kernel Variables...
         Starting Create System Users...
[FAILED] Failed to mount FUSE Control File System.
See 'systemctl status sys-fs-fuse-connections.mount' for details.
[FAILED] Failed to mount Kernel Configuration File System.
See 'systemctl status sys-kernel-config.mount' for details.
[   13.848020] systemd-journald[259]: Received client request to flush runtime journal.
[   13.857567] systemd-journald[259]: File /var/log/journal/b0c15c2d3a1843d486fe35272f9f72ef/system.journal corrupted or uncleanly shut down, renaming and replacing.
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Preparation for Local File Systems.
[  OK  ] Set up automount EFI System Partition Automount.
[  OK  ] Reached target Local File Systems.
         Starting Tell Plymouth To Write Out Runtime Data...
         Starting Set Up Additional Binary Formats...
         Starting Rule-based Manage�…for Device Events and Files...
[  OK  ] Finished Tell Plymouth To Write Out Runtime Data.
[FAILED] Failed to start Rule-based�…r for Device Events and Files.
See 'systemctl status systemd-udevd.service' for details.
[  OK  ] Stopped Rule-based Manager for Device Events and Files.
         Starting Rule-based Manage�…for Device Events and Files...
[FAILED] Failed to start Set Up Additional Binary Formats.
See 'systemctl status systemd-binfmt.service' for details.
[FAILED] Failed to start Rule-based�…r for Device Events and Files.
See 'systemctl status systemd-udevd.service' for details.
[  OK  ] Stopped Rule-based Manager for Device Events and Files.
         Starting Rule-based Manage�…for Device Events and Files...
[FAILED] Failed to start Rule-based�…r for Device Events and Files.
See 'systemctl status systemd-udevd.service' for details.
[  OK  ] Stopped Rule-based Manager for Device Events and Files.
         Starting Rule-based Manage�…for Device Events and Files...
[FAILED] Failed to start Rule-based�…r for Device Events and Files.
See 'systemctl status systemd-udevd.service' for details.
[  OK  ] Stopped Rule-based Manager for Device Events and Files.
         Starting Rule-based Manage�…for Device Events and Files...
[  OK  ] Finished Coldplug All udev Devices.
         Starting Helper to synchronize boot up for ifupdown...
[FAILED] Failed to start Rule-based�…r for Device Events and Files.
See 'systemctl status systemd-udevd.service' for details.
[  OK  ] Stopped Rule-based Manager for Device Events and Files.
[FAILED] Failed to start Rule-based�…r for Device Events and Files.
See 'systemctl status systemd-udevd.service' for details.
         Starting Show Plymouth Boot Screen...
[  OK  ] Finished Helper to synchronize boot up for ifupdown.
         Starting Raise network interfaces...
[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Started Forward Password R�…s to Plymouth Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Path Units.
[  OK  ] Finished Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Raise network interfaces.
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Network Time Synchronization...
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[FAILED] Failed to start Network Time Synchronization.
See 'systemctl status systemd-timesyncd.service' for details.
[  OK  ] Stopped Network Time Synchronization.
         Starting Network Time Synchronization...
[FAILED] Failed to start Network Time Synchronization.
See 'systemctl status systemd-timesyncd.service' for details.
[  OK  ] Stopped Network Time Synchronization.
         Starting Network Time Synchronization...
[FAILED] Failed to start Network Time Synchronization.
See 'systemctl status systemd-timesyncd.service' for details.
[  OK  ] Stopped Network Time Synchronization.
         Starting Network Time Synchronization...
[FAILED] Failed to start Network Time Synchronization.
See 'systemctl status systemd-timesyncd.service' for details.
[  OK  ] Stopped Network Time Synchronization.
         Starting Network Time Synchronization...
[FAILED] Failed to start Network Time Synchronization.
See 'systemctl status systemd-timesyncd.service' for details.
[  OK  ] Stopped Network Time Synchronization.
[FAILED] Failed to start Network Time Synchronization.
See 'systemctl status systemd-timesyncd.service' for details.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target System Time Set.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Started Daily dpkg database backup timer.
[  OK  ] Started Periodic ext4 Onli�…ata Check for All Filesystems.
[  OK  ] Started Discard unused blocks once a week.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Daily man-db regeneration.
[  OK  ] Reached target Timer Units.
[   16.761536] random: avahi-daemon: uninitialized urandom read (4 bytes read)
[   16.777911] random: dbus-daemon: uninitialized urandom read (12 bytes read)
[  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Socket Units.
[  OK  ] Reached target Basic System.
         Starting Accounts Service...
         Starting Avahi mDNS/DNS-SD Stack...
[  OK  ] Started Regular background program processing daemon.
         Starting D-Bus System Message Bus...
         Starting Remove Stale Onli�…t4 Metadata Check Snapshots...
[   16.915613] random: dbus-daemon: uninitialized urandom read (12 bytes read)
         Starting Initialize hardware monitoring sensors...
         Starting Authorization Manager...
         Starting Power Profiles daemon...
         Starting Switcheroo Control Proxy service...
         Starting User Login Management...
         Starting Disk Manager...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Finished Initialize hardware monitoring sensors.
         Starting Network Manager...
         Starting WPA supplicant...
[  OK  ] Started Avahi mDNS/DNS-SD Stack.
[  OK  ] Started Authorization Manager.
         Starting Modem Manager...
[  OK  ] Started WPA supplicant.
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[DEPEND] Dependency failed for Network Manager Wait Online.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
         Starting OpenBSD Secure Shell server...
         Starting Permit User Sessions...
         Starting LSB: My shell script...
[  OK  ] Finished Permit User Sessions.
         Starting Hold until boot process finishes up...
[  OK  ] Stopped Network Manager.
         Starting Network Manager...
[FAILED] Failed to start Modem Manager.
See 'systemctl status ModemManager.service' for details.
[  OK  ] Started LSB: My shell script.
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[FAILED] Failed to start Remove Sta�…ext4 Metadata Check Snapshots.
See 'systemctl status e2scrub_reap.service' for details.
[   17.617516] random: crng init done
[   17.626048] random: 1 urandom warning(s) missed due to ratelimiting
[FAILED] Failed to start Switcheroo Control Proxy service.
See 'systemctl status switcheroo-control.service' for details.
[  OK  ] Stopped Network Manager.
         Starting Network Manager...
         Starting GNOME Display Manager...
[FAILED] Failed to start Power Profiles daemon.
See 'systemctl status power-profiles-daemon.service' for details.
[FAILED] Failed to start Load/Save Random Seed.
See 'systemctl status systemd-random-seed.service' for details.
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[  OK  ] Stopped Power Profiles daemon.
         Starting Power Profiles daemon...
[  OK  ] Started OpenBSD Secure Shell server.
[FAILED] Failed to start User Login Management.
See 'systemctl status systemd-logind.service' for details.
[  OK  ] Stopped User Login Management.
         Starting Load Kernel Module drm...
[  OK  ] Finished Load Kernel Module drm.
[  OK  ] Started Disk Manager.
[  OK  ] Started GNOME Display Manager.
[  OK  ] Stopped Network Manager.
         Starting Network Manager...
         Starting User Login Management...
[FAILED] Failed to start Accounts Service.
See 'systemctl status accounts-daemon.service' for details.
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[FAILED] Failed to start Power Profiles daemon.
[FAILED] Failed to start Network Manager.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start Network Manager.
[FAILED] Failed to start Power Profiles daemon.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start Power Profiles daemon.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start Power Profiles daemon.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start User Login Management.
[FAILED] Failed to start Power Profiles daemon.

I suppose that the late failures are consequences of the mounts failed:

[FAILED] Failed to mount Huge Pages File System.
[FAILED] Failed to mount POSIX Message Queue File System.
[FAILED] Failed to mount Kernel Debug File System.
[FAILED] Failed to mount FUSE Control File System.
[FAILED] Failed to mount Kernel Configuration File System.

how do I solve this, please?

I found the solution myself, by accident.
It turns out that the debian 202306 was cross-compiled from another system.
In my case, the working setup is with a ubuntu 22.04, with this cross-compiler:
riscv64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0

this setup generates 3 cross-built .deb packages that need also some rework to be in the right format.
I have done this with a script, that i copy here for reference:

#!/bin/bash
	
mkdir -p repack
mv *.deb repack
cd repack
for i in *.deb; do
	ar x $i
	rm $i
	zstd -d < control.tar.zst | xz > control.tar.xz
	zstd -d < data.tar.zst | xz > data.tar.xz
	ar -m -c -a sdsd $i debian-binary control.tar.xz data.tar.xz
	mv $i ..
done

so, apart from:

sudo apt install gcc-riscv64-linux-gnu
export CROSS_COMPILE=riscv64-linux-gnu-

the build instructions in the release note work just fine.

3 Likes