Orange pi 3b NVME problem

Troubleshooting
1

Creating a bug report/issue

[+] I have searched the existing open and closed issues

Required Information

  • DietPi version
    G_DIETPI_VERSION_CORE=9
    G_DIETPI_VERSION_SUB=2
    G_DIETPI_VERSION_RC=1
    G_GITBRANCH=‘master’

  • Distro version
    bookworm

  • Kernel version
    Linux DietPi 6.7.4-edge-rockchip64 #3 SMP PREEMPT Fri Feb 23 08:51:59 UTC 2024 aarch64 GNU/Linux

  • Architecture
    arm64

  • SBC model
    Orange Pi 3B (aarch64)

  • Power supply used
    5V 3,6A GeeekPi

  • SD card used
    SanDisk extreme

  • Can this issue be replicated on a fresh installation of DietPi?
    Yes

Steps to reproduce

  1. connect the m.2 nvme drive to the m.2 nvme slot on orange pi 3b
  2. log in using the lspci command (you need to download pciutils) the disk is displayed, but neither in fdisk -l nor in dietpi-drive_manager

Expected behaviour

  • Drive should be accessible

Actual behaviour

  • Drive isn’t accessible

Extra details

  • after connecting m.2 nvme drive to orange pi 3b it works much slower, and drive is getting pretty hot.
2

Do you mean you have the impression that the Orange Pi damages the NVMe drive? That would be bad indeed.

I am currently building new kernel and bootloader, in the hope it contains NVMe support for Orange Pi 3B.

3

what I meant was that the disk actually starts working or something after connecting it because it gets quite warm, you can hold it, but it’s not a pleasant heat for both the memory modules and the controller

4

Possible that the driver is doing something wrong, powering the device (of course, as it shows up in lspci) and probably does some detection loop, who knows.

Could you test with the new kernel:

cd /tmp
wget https://dietpi.com/downloads/binaries/testing/linux-{image,dtb}-edge-rockchip64.deb
dpkg -i linux-{image,dtb}-edge-rockchip64.deb
reboot

This should be Linux 6.8.4:

uname -a
5

I tried this solution and it’s still the same, it keeps getting warm

6

Okay, but it is finally detected (as block device/drive), can be formatted and/or mounted?

I am not sure how hot an NVMe drive is expected to become (on this SBC). It is also possible the other way round: That other PCB components heat up (contribute to higher idle temps of) the NVMe.

Is it possible to chance APM for it? Like

hdparm -B 1 /dev/nvme0n1
7

it was not detected by fdisk -l or dietpi-drive_manager, and regarding the command you suggested, I can’t check it now

8

I checked now, and the output of this command was: No such fille or directory

9

So it is not detected, sorry I misunderstood that it was finally working (but just getting unexpected warm).

Are there any kernel errors?

dmesg -l 0,1,2,3
11

there are errors
those with an m.2 nvme drive inserted:

[    1.409207] mmc1: tuning execution failed: -5                           
[    1.409238] mmc1: error -5 whilst initialising SD card                  
[    1.535144] mmc1: tuning execution failed: -5                           
[    1.906747] WCN_ERR: gpio chip_en request err: 27                       
[  121.080409] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
[  125.911131] systemd-fstab-generator[271]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?                                                           
[  125.911358] systemd-fstab-generator[271]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
[  126.062189] (sd-execut[267]: /usr/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 1.                                 
[  128.037520] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[  128.049523] wifi ini path = /lib/firmware/wifi_2355b001_1ant.ini        
[  128.053345] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[  128.067028] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[  128.084523] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[  128.092943] sprdwl:sprdwl_get_fw_info length mismatch: len_count=83, r_len=89                                                                      [  128.093008] sprdwl:sprdwl_get_fw_info, drv_version=1, fw_version=2, compat_ver=0                                                                   [  128.093034] sprdwl:chip_model:0x2355, chip_ver:0x0                      
[  128.093053] sprdwl:fw_ver:0, fw_std:0x7f, fw_capa:0x120f7f              
[  128.093073] sprdwl:mac_addr:e0:51:d8:20:e3:78                           
[  128.093094] sprdwl:credit_capa:TX_WITH_CREDIT                           
[  128.093110] sprdwl:ott support:0                                        
[  128.142231] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[  132.008376] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff
[  132.008438] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff

those without m.2 nvme disk:

[    2.377600] mmc1: tuning execution failed: -5                           
[    2.377631] mmc1: error -5 whilst initialising SD card                  
[    2.445830] WCN_ERR: gpio chip_en request err: 27                       
[    6.565318] systemd-fstab-generator[270]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?                                                           
[    6.565529] systemd-fstab-generator[270]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?                                                           
[    6.577887] (sd-execut[266]: /usr/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 1.                                 
[    8.496641] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[    8.503110] wifi ini path = /lib/firmware/wifi_2355b001_1ant.ini        
[    8.507881] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[    8.523079] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[    8.530794] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[    8.546082] sprdwl:sprdwl_get_fw_info length mismatch: len_count=83, r_len=89                                                                      [    8.546147] sprdwl:sprdwl_get_fw_info, drv_version=1, fw_version=2, compat_ver=0                                                                   [    8.546173] sprdwl:chip_model:0x2355, chip_ver:0x0                      
[    8.546191] sprdwl:fw_ver:0, fw_std:0x7f, fw_capa:0x120f7f              
[    8.546211] sprdwl:mac_addr:e0:51:d8:20:e3:78                           
[    8.546232] sprdwl:credit_capa:TX_WITH_CREDIT                           
[    8.546248] sprdwl:ott support:0                                        
[    8.591680] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0        
[   11.475952] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff   
[   11.476020] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff

And from what I noticed at first glance, from what I saw the only this line refers to the nvme disk and the fact that it is not ready:
[ 121.080409] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0

1 Like
12

https://www.reddit.com/r/archlinux/comments/15xbxeo/nvme_device_not_ready_aborting_initialisation/

Also looks similar (but not identical) to APST issue reported on Arch wiki: Solid state drive/NVMe - ArchWiki

Here is an indication that it might be a U-Boot issue instead of kernel issue: Nvme suddenly stopped working - Technical Assistance - Manjaro Linux Forum

Let’s try with updated U-Boot first:

cd /tmp
wget https://dietpi.com/downloads/binaries/testing/linux-u-boot-orangepi3b-edge.deb
dpkg -i linux-u-boot-orangepi3b-edge.deb
rm linux-u-boot-orangepi3b-edge.deb
source /usr/lib/u-boot/platform_install.sh
eval "$(declare -f write_uboot_platform | sed -e 's| > /dev/null 2>&1||g' -e 's|status=none|status=progress|g')"
write_uboot_platform "$DIR" "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"
13

didn’t help, just added a few more lines in
dmesg -l 0,1,2,3
from what I can see

[    1.410365] mmc1: tuning execution failed: -5
[    1.410394] mmc1: error -5 whilst initialising SD card
[    1.533597] mmc1: tuning execution failed: -5
[    1.904920] WCN_ERR: gpio chip_en request err: 27
[  121.074950] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
[  126.277375] systemd-fstab-generator[272]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
[  126.277570] systemd-fstab-generator[272]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
[  126.472819] (sd-execut[268]: /usr/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 1.
[  128.106763] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[  128.114523] wifi ini path = /lib/firmware/wifi_2355b001_1ant.ini
[  128.117207] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[  128.128004] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[  128.136783] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[  128.146567] sprdwl:sprdwl_get_fw_info length mismatch: len_count=83, r_len=89
[  128.146677] sprdwl:sprdwl_get_fw_info, drv_version=1, fw_version=2, compat_ver=0
[  128.146709] sprdwl:chip_model:0x2355, chip_ver:0x0
[  128.146729] sprdwl:fw_ver:0, fw_std:0x7f, fw_capa:0x120f7f
[  128.146749] sprdwl:mac_addr:e0:51:d8:20:e3:78
[  128.146770] sprdwl:credit_capa:TX_WITH_CREDIT
[  128.146786] sprdwl:ott support:0
[  128.187053] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[  132.163865] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff
[  132.163922] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff
[  239.722267] systemd-fstab-generator[1043]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
[  239.722382] systemd-fstab-generator[1043]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?

I checked the links you sent and the output of the command
journalctl -b | grep nvme
was:

kwi 07 20:47:35 DietPi kernel: nvme nvme0: pci function 0000:01:00.0
kwi 07 20:47:35 DietPi kernel: nvme 0000:01:00.0: enabling device (0000 -> 0002)
kwi 07 20:47:35 DietPi kernel: nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
kwi 07 20:50:07 DietPi systemd[1]: nvmefc-boot-connections.service - Auto-connect to subsystems on FC-NVME devices found during boot was skipped because of an unmet condition check (ConditionPathExists=/sys/class/fc/fc_udev_device/nvme_discovery).

I tested another m.2 disk (previously it was not detected by lspci) and there are slightly different logs with
dmesg -l 0,1,2,3
logs:

[    1.276383] nvme 0000:01:00.0: of_irq_parse_pci: failed with rc=134
[    1.277015] nvme 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[    1.616555] mmc1: tuning execution failed: -5
[    1.616584] mmc1: error -5 whilst initialising SD card
[    1.741030] mmc1: tuning execution failed: -5
[    1.914558] WCN_ERR: gpio chip_en request err: 27
[    7.748324] systemd-fstab-generator[291]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
[    7.748545] systemd-fstab-generator[291]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
[    7.911370] (sd-execut[287]: /usr/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 1.
[    9.554034] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[    9.566811] wifi ini path = /lib/firmware/wifi_2355b001_1ant.ini
[    9.569357] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[    9.580379] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[    9.587546] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[    9.600670] sprdwl:sprdwl_get_fw_info length mismatch: len_count=83, r_len=89
[    9.600732] sprdwl:sprdwl_get_fw_info, drv_version=1, fw_version=2, compat_ver=0
[    9.600759] sprdwl:chip_model:0x2355, chip_ver:0x0
[    9.600779] sprdwl:fw_ver:0, fw_std:0x7f, fw_capa:0x120f7f
[    9.600798] sprdwl:mac_addr:e0:51:d8:20:e3:78
[    9.600819] sprdwl:credit_capa:TX_WITH_CREDIT
[    9.600835] sprdwl:ott support:0
[    9.639699] sprdwl:__sprdwl_cmd_getbuf cant't get vif, ctx_id: 0
[   13.868517] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff
[   13.868590] sprdwl:ctx_id = 0, sm_state = 4, bssid =ff:ff:ff:ff:ff:ff
[  129.096772] systemd-fstab-generator[1107]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
[  129.096911] systemd-fstab-generator[1107]: Failed to create unit file '/run/systemd/generator/swapfile.swap', as it already exists. Duplicate entry in '/etc/fstab'?
14

Is there an issue with swap spaces?

grep swap /etc/fstab
swapon

However, yes, the issue with the aborted NVMe initialisation is still the same. Let’s test whether disabling APST helps. First of all, does this kernel module exist?

modinfo nvme_core

I am sure it is builtin. In this case, add nvme_core.default_ps_max_latency_us=0 to the end of extraargs (kernel command-line parameters) in /boot/dietpiEnv.txt:

sed -i '/^extraargs=/$/ nvme_core.default_ps_max_latency_us=0/' /boot/dietpiEnv.txt
reboot
15

the swap error did not occur before the kernel update, so this error probably occurred after the update, and as for these commands:

grep swap /etc/fstab
/swapfile none swap defaults 0 0
/swapfile none swap defaults 0 0
/swapfile none swap defaults 0 0
swapon
NAME      TYPE SIZE USED PRIO
/swapfile file  15G   0B   -2

modinfo nvme_core
name:           nvme_core
filename:       (builtin)
description:    NVMe host core framework
version:        1.0
license:        GPL
file:           drivers/nvme/host/nvme-core
....

the command you provided to edit the file returns an error that $ is unrecognised command

16

you should remove at least 2 of them

1 Like
17

After this command, this is what the line should look like?

extraargs=net.ifnames=0 nvme_core.default_ps_max_latency_us=0

if so, it didn’t help, it just caused one of the disks to not be detected at all now

18

One of the other (non-NVMe) disks?

Right, those are no entries from our swap space script. If you did not add them manually, probably you used some other setup script repetitively, which does not check for existing swap space entries before adding new ones. This also has nothing to do with kernel upgrades. No native kernel or even Debian userland or DietPi tool touches /etc/fstab like this.

19

both are nvme drives, one from WD, the other from Adata, and the files disappeared by themselves after reboot/drive_manager

20

How do you attach multiple NVMe drives to the Orange Pi 3B? And I thought they were never detected (as block devices)?

21

I tried changing disks, i.e. I turn off Orange Pi, replace the disk, turn on Orange Pi and check if it works