NanoPi NEO running unstable since latest update

Creating a bug report/issue

I have searched the existing open and closed issues

Required Information

  • DietPi version |
    G_DIETPI_VERSION_CORE=9
    G_DIETPI_VERSION_SUB=3
    G_DIETPI_VERSION_RC=0
    G_GITBRANCH=‘master’
    G_GITOWNER=‘MichaIng’
  • Distro version | bookworm
  • Kernel version | Linux NPiNeo 6.6.16-current-sunxi #1 SMP Fri Feb 23 08:25:28 UTC 2024 armv7l GNU/Linux
  • Architecture | armhf
  • SBC model | NanoPi NEO (armv7l)
  • Power supply used | 5V/2A phone charger
  • SD card used | SanDisk 4GB

Additional Information (if applicable)

  • Software title | Pi-hole, VPN, vaultwarden, …
    but irrelevant, since the issues also occur on a freshly installed system
  • Was the software title installed freshly or updated/migrated?
  • Can this issue be replicated on a fresh installation of DietPi?
    YES
  • Bug report ID | 16dba308-5a28-4dc3-b3a3-39d0ddc7db26

Steps to reproduce

  • download latest NanoPi NEO image, write to SD card and finish the first boot installation process
    or
    update a running NanoPi NEO to the latest version

    and

  • let the system run some time

Expected behaviour

  • low system load and low CPU temp, especially if no additional software is installed
  • system and installed services to be available without extremly delays

Actual behaviour

  • The load average regularly increases significant and causes the system to become slow or unresponsive for a while.
  • Some programmes (e.g. dietpi-software) then do not start until an additional key (e.g. space bar) is pressed.
  • CPU temperature then also rising up
  • can’t establish ssh connection sometimes
  • system crashes on shutdown (loses ssh connection, but sill answering to ping)

Extra details

I have this issues on all three systems, but only since the last update to v9.3.0.

The above information and the bug id are from a new installation without any additional software installed.

The following is often found in the protocols (journalctl) on all the systems:

kernel: rcu: INFO: rcu_sched self-detected stall on CPU
kernel: rcu:         3-....: (5250 ticks this GP) idle=0c14/1/0x40000002 softirq=210140/210146 fqs=2622
kernel: rcu:         (t=5254 jiffies g=276169 q=26 ncpus=4)
kernel: CPU: 3 PID: 5838 Comm: dropbear Tainted: G         C         6.6.16-current-sunxi #1
kernel: Hardware name: Allwinner sun8i Family
kernel: PC is at stmmac_get_stats64+0x26/0x128
kernel: LR is at 0xc371b000
kernel: pc : [<c078c5ce>]    lr : [<c371b000>]    psr: a00f0033
kernel: sp : e2389c68  ip : c3718000  fp : 00000001
kernel: r10: e2389e68  r9 : c9e52ce8  r8 : c3718000
kernel: r7 : 00000000  r6 : 00000001  r5 : 00000000  r4 : 80000000
kernel: r3 : 00005f2f  r2 : c371ae48  r1 : e2389d28  r0 : c3718000
kernel: Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
kernel: Control: 50c5387d  Table: 4481006a  DAC: 00000051
kernel:  stmmac_get_stats64 from dev_get_stats+0x27/0xd0
kernel:  dev_get_stats from dev_seq_printf_stats+0x21/0x124
kernel:  dev_seq_printf_stats from dev_seq_show+0x11/0x24
kernel:  dev_seq_show from seq_read_iter+0x281/0x35c
kernel:  seq_read_iter from seq_read+0x61/0x84
kernel:  seq_read from proc_reg_read+0x71/0x90
kernel:  proc_reg_read from vfs_read+0x75/0x1e4
kernel:  vfs_read from ksys_read+0x45/0x9c
kernel:  ksys_read from ret_fast_syscall+0x1/0x5c
kernel: Exception stack(0xe2389fa8 to 0xe2389ff0)
kernel: 9fa0:                   00001000 00000003 00000003 bebb4a0c 00001000 00000001
kernel: 9fc0: 00001000 00000003 00000000 00000003 bebb4a0c 00000fff 00000000 0050b84c
kernel: 9fe0: 00000003 bebb49f0 b6dae2bb b6d27616

The latest image file is at v9.2.1, but gets updated automatically to v9.3.0 at the first login.
Is there a way to skip the first boot update process and finish the installer to stay at v9.2.1?
Cause I would like to get my server applications back running.

Usually not, theoretically you could try to boot without internet connection as this will stop install. Probably some Debian packages that will get updated. I doubt it’s related to the DietPi scripts directly.

Would be good to check which Debian packages are pending to be updated. Maybe a kernel package.

You don’t have a backup done before running the DietPi update?

Ping @MichaIng . Maybe he could have a look.

I tried that now, but the first boot setup seems to need an internet connection to finish.
After some ‘network connectivity … unreachable’ and ‘setup failed’ messages the first boot setup quits and I am in the shell, but the setup restarts on a new login.

However, I’ve noticed that this v9.2.1 image already has the same kernel version 6.6.16-current-sunxi #1 SMP Fri Feb 23 08:25:28 UTC.

Even with a fresh install the system gets unreachable after a reboot.
Its now running for 2 hr and no rcu_sched messages yet.

I only have config file backups of the installed apps, but not a complete disk backup.

So it is not the kernel? With a custom pre-script (see dietpi.txt for details`), the packages else could be set on hold.

Actually there should nothing but the kernel lead to CPU stalls like that. I will let my M1 run to see whether I can replicate issues. Is there any load/software you installed before issues/CPU stalls appeared, or does it also happen on clean idle system?

I think this is related: Long-term load on Pi-Hole Instance - Help - Pi-hole Userspace
See also the linked thread in Armbian forum about high CPU temperature/usage on other boards with this kernel family.

EDIT: Here a new kernel package with Linux 6.6.29:

cd /tmp
wget https://dietpi.com/downloads/binaries/testing/linux-{image,dtb}-current-sunxi.deb
dpkg -i linux-{image,dtb}-current-sunxi.deb
rm linux-{image,dtb}-current-sunxi.deb
reboot

Thanks,
with this package the issues seem to be gone and the systems are back running stable for now.

1 Like

Great. I added the packages to our APT repo. The respective APT component will be added to all 32-bit Allwinner SBCs with next DietPi update: Index of /apt/dists/all/nanopineo/binary-armhf

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.