Open source speech recognition on VisionFive 2 with Next-gen Kaldi

Hi there,

we have managed to run speech recognition on VisionFive 2 with next-gen Kaldi.

GitHub repo: GitHub - k2-fsa/sherpa-ncnn: Real-time speech recognition using next-gen Kaldi with ncnn without Internet connection. Support iOS, Android, Raspberry Pi, etc.
Documentation: Embedded Linux (riscv64) — sherpa 1.2 documentation

The following is a screenshot of running sherpa-ncnn on VisionFive 2

Unfortunately, the board is unable to detect my USB microphone, which works perfectly on Raspberry Pi Model 4 B. However,
arecord -l on vision five 2 shows that there is no sound device available.

If you can record audio with the board, you can do real-time speech recognition with sherpa-ncnn on the board.

4 Likes

You never mentioned the make and model of your USB microphone. Does your USB microphone show up with lsusb (may require sudo apt install usbutils). If it is unbranded you can usually find additional information at the USB ID repository, using the Vendor ID and Product ID from the lsusb command. At a guess you probably just need to modprobe a module to get it working. And if that works then create a udev rule to run when that USB device is inserted.

If the module exists on the VF2, you may be able to just copy a working udev file from the RPi to the VF2 from either /usr/lib/udev/rules.d/ or /etc/udev/rules.d/, just grep for the VID and PID in all the scripts to identify the right one. (e.g. grep -E 'insert_VID_number|insert_PID_number' /usr/lib/udev/rules.d/* /etc/udev/rules.d/*)

2 Likes

@mzs Thanks for your reply.

After installing usbutils, I am able to use lsusb to detect my USB microphone:

user@starfive:~$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 006: ID 4c4a:4155 Jieli Technology UACDemoV1.0
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The second line is my USB microphone

Bus 001 Device 006: ID 4c4a:4155 Jieli Technology UACDemoV1.0

However, arecord -l still shows no sound devices are available.

The output of lsmod is given below

user@starfive:~$ lsmod
Module                  Size  Used by
venc                   28672  0
jpu                    24576  0
vdec                   32768  0
user@starfive:/lib/modules/5.15.0-starfive$ ls
build   modules.alias      modules.builtin.alias.bin  modules.dep      modules.order    modules.symbols.bin
extra   modules.alias.bin  modules.builtin.bin        modules.dep.bin  modules.softdep  v2.11.5.tar.gz
kernel  modules.builtin    modules.builtin.modinfo    modules.devname  modules.symbols
user@starfive:/lib/modules/5.15.0-starfive$ find . -name "*usb*"
./kernel/drivers/net/wireless/eswin/wlan_ecr6600u_usb.ko

I cannot find snd-usb-audio.ko on the board.

Have you succeeded in using a microphone with VisionFive 2?

1 Like

By the way,

git grep sound

in the following repo outputs nothing. Looks like Vision Five 2 does not support microphone at all.

1 Like

I did a search through the Linux source code for the VF2 and did not find any driver for the specific VendorID:ProductID of a 4c4a:4155 Jieli Technology UACDemoV1.0:

mzs@ding:~/src/risc-v/jh7110/VisionFive2/linux$ rgrep "Jieli" *
sound/usb/quirks.c:	case USB_ID(0x1224, 0x2a25):  /* Jieli Technology USB PHY 2.0 */
sound/usb/quirks.c:	DEVICE_FLG(0x1224, 0x2a25, /* Jieli Technology USB PHY 2.0 */
sound/usb/mixer.c:	case USB_ID(0x1224, 0x2a25): /* Jieli Technology USB PHY 2.0 */
mzs@ding:~/src/risc-v/jh7110/VisionFive2/linux$

And then I thought lets check the RPi linux kernel and see what they have, thinking that you probably only need a patch of few lines to add support for a Jieli Technology 4c4a:4155 device.

But the lines in the RPi linux kernel are exactly the same, which is strange, so then I started to wonder what it actually is and from what I can find it is not a USB sound card it is a Dual-Mode wireless bluetooth modem of some sort. So at a guess the reason it works on a RPi is because they would have installed by default all the required bluetooth packages because some of their boards ship with broadcom bluetooth chips. So you probably need to install and configure things like “bluetooth”, “bluez”, “bluez-firmware”, etc. before it will be able to function. To be honest I know next to nothing about bluetooth so you are probably better off consulting some online guides. But the reason your sound is not working is because you have no Bluetooth support (I think, I could be totally wrong).

4 Likes

You don’t mention which release you are running, but assuming it is a StarFive image you will need to recompile the kernel to get at many USB functions.

I had to do this to get my generic BT dongle working and using USBhid (Kbd and Mouse). You would need to configure the usb-sound driver, usb-bt and others…

This is what I needed to enable to get my BT keyboard and mouse working:

CONFIG_BT_HIDP=y
CONFIG_BT_HS=y
CONFIG_BT_INTEL=y
CONFIG_BT_BCM=y
CONFIG_BT_RTL=y
CONFIG_BT_HCIBTUSB=y
CONFIG_BT_HCIBTUSB_BCM=y	
CONFIG_BT_HCIBTUSB_MTK=y
CONFIG_BT_HCIBTUSB_RTL=y

The only sound related option enabled in the StarFive kernels is CONFIG_SOUND=y, so I guess you would need to enable USB sound and/or BT sound to get this working.

3 Likes