Odroid C4 won't boot after updates

devon · 18 December 2022 02:51

Creating a bug report/issue

Required Information

Since the SBC no longer boots, I can’t get a lot of the version information. I downloaded and imaged using “DietPi_OdroidC4-ARMv8-Bullseye.img” dated November 20th.

SBC model | Odroid C4
Power supply used | Ameridroid 12V/2A US power supply
SD card used | Samsung 32GB Evo Select

Steps to reproduce

Boot MicroSD with fresh image of DietPi
SSH to device and wait for initial updates

Expected behaviour

SSH to the device would still work after updates/reboot

Actual behaviour

Device is now inaccessible after updates/reboot

Extra details

If I connect an HDMI cable to the device, I can it is stopped after Running/scripts/local-premount...done
Error after “Starting version 247.3-7+deb11u1”: “Unable to handle kernel paging request at virtual address” followed by some more information and a call trace.

devon · 18 December 2022 22:02

I think I may have gotten this resolved. I was using a MicroSD to SD adapter to image my card. I’ve plugged a card reader in that supports MicroSD instead and re-imaged. I’m at least getting further than I was before.

devon · 19 December 2022 23:18

This is still flaky. Sometimes when I flash it, it will boot. (Many times it won’t, even after a fresh flash.) When it does boot, after updating and restarting, I end up with a kernel panic, whether it was imaged using the adapter or the card reader. Any help would be appreciated.

Joulinar · 19 December 2022 23:21

probably @MichaIng could have look

devon · 20 December 2022 01:23

Thank you all for the help. Here’s what happened in my latest attempt. Just flashed my card again. I’m trying a different card now: it’s a Kingston 64GB MicroSD.

This is one of the rare occasions it survived the reboot. Here’s the output in my SSH session before the reboot:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
 ─────────────────────────────────────────────────────
 DietPi v8.11.2 : 00:10 - Tue 12/20/22
 ─────────────────────────────────────────────────────
 - LAN IP : 192.168.1.129 (eth0)

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Checking for available DietPi update

[  OK  ] DietPi-Update | Checking IPv4 network connectivity
[  OK  ] DietPi-Update | Checking DNS resolver
[  OK  ] DietPi-TimeSync | systemctl stop systemd-timesyncd
[  OK  ] DietPi-TimeSync | mkdir -p /run/systemd/timesync
[ INFO ] DietPi-Update | Getting latest version from: https://raw.githubusercontent.com/MichaIng/DietPi/master/.update/version
[  OK  ] DietPi-Update | Got valid latest version: 8.12.1
[  OK  ] DietPi-Update | Update available:
[ INFO ] DietPi-Update | Current version : v8.11.2
[ INFO ] DietPi-Update | Latest version  : v8.12.1

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Checking for update pre-requirements

[  OK  ] DietPi-Update | DietPi-Userdata validation: /mnt/dietpi_userdata
[  OK  ] DietPi-Update | Free space check: path=/ | available=55703 MiB | required=100 MiB
[ SUB1 ] DietPi-Services > stop
[ INFO ] DietPi-Services | skip : cron (due to mask)

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Applying pre-patches

[  OK  ] DietPi-Update | Downloading pre-patches
[  OK  ] DietPi-Update | Applying execute permission
[  OK  ] DietPi-Update | Successfully applied pre-patches

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Upgrading APT packages

[ INFO ] DietPi-Update | APT update, please wait...
Get:1 https://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 https://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:3 https://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:4 https://deb.debian.org/debian bullseye-backports InRelease [49.0 kB]
Get:5 https://mirror-us-sea2.armbian.airframes.io/apt bullseye InRelease [36.6 kB]
Get:6 https://deb.debian.org/debian bullseye/contrib arm64 Packages [41.0 kB]
Get:7 https://deb.debian.org/debian bullseye/main arm64 Packages [8072 kB]
Get:8 https://deb.debian.org/debian bullseye/non-free arm64 Packages [73.0 kB]
Get:9 https://deb.debian.org/debian bullseye-updates/main arm64 Packages [12.0 kB]
Get:10 https://deb.debian.org/debian-security bullseye-security/main arm64 Packages [206 kB]
Get:11 https://deb.debian.org/debian bullseye-backports/non-free arm64 Packages [9864 B]
Get:12 https://deb.debian.org/debian bullseye-backports/main arm64 Packages [365 kB]
Get:13 https://deb.debian.org/debian bullseye-backports/contrib arm64 Packages [3844 B]
Get:14 https://mirror-us-sea2.armbian.airframes.io/apt bullseye/main arm64 Packages [592 kB]
Fetched 9668 kB in 7s (1352 kB/s)
Reading package lists...
[  OK  ] DietPi-Update | APT update
[ INFO ] DietPi-Update | APT upgrade, please wait...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  armbian-firmware base-files libtasn1-6 linux-dtb-current-meson64
  linux-image-current-meson64 linux-u-boot-odroidc4-current nano
7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 67.5 MB of archives.
After this operation, 137 MB disk space will be freed.
Get:1 https://deb.debian.org/debian bullseye/main arm64 base-files arm64 11.1+deb11u6 [70.1 kB]
Get:2 https://deb.debian.org/debian bullseye/main arm64 libtasn1-6 arm64 4.16.0-2+deb11u1 [53.3 kB]
Get:3 https://deb.debian.org/debian bullseye/main arm64 nano arm64 5.4-2+deb11u2 [647 kB]
Get:7 https://mirror-us-sea2.armbian.airframes.io/apt bullseye/main arm64 linux-u-boot-odroidc4-current arm64 22.11.1 [761 kB]
Get:5 https://mirror-us-sea1.armbian.airframes.io/apt bullseye/main arm64 linux-dtb-current-meson64 arm64 22.11.2 [130 kB]
Get:4 https://armbian.lv.auroradev.org/apt bullseye/main arm64 armbian-firmware all 22.11.2 [9443 kB]
Get:6 https://mirror-us-sea1.armbian.airframes.io/apt bullseye/main arm64 linux-image-current-meson64 arm64 22.11.2 [56.4 MB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 67.5 MB in 10s (6774 kB/s)
(Reading database ... 17886 files and directories currently installed.)
Preparing to unpack .../base-files_11.1+deb11u6_arm64.deb ...
Unpacking base-files (11.1+deb11u6) over (11.1+deb11u5) ...
Setting up base-files (11.1+deb11u6) ...
Installing new version of config file /etc/debian_version ...
(Reading database ... 17886 files and directories currently installed.)
Preparing to unpack .../libtasn1-6_4.16.0-2+deb11u1_arm64.deb ...
Unpacking libtasn1-6:arm64 (4.16.0-2+deb11u1) over (4.16.0-2) ...
Setting up libtasn1-6:arm64 (4.16.0-2+deb11u1) ...
(Reading database ... 17886 files and directories currently installed.)
Preparing to unpack .../nano_5.4-2+deb11u2_arm64.deb ...
Unpacking nano (5.4-2+deb11u2) over (5.4-2+deb11u1) ...
Preparing to unpack .../armbian-firmware_22.11.2_all.deb ...
Unpacking armbian-firmware (22.11.2) over (22.08.6) ...
Preparing to unpack .../linux-dtb-current-meson64_22.11.2_arm64.deb ...
Unpacking linux-dtb-current-meson64 (22.11.2) over (22.08.4) ...
Preparing to unpack .../linux-image-current-meson64_22.11.2_arm64.deb ...
Unpacking linux-image-current-meson64 (22.11.2) over (22.08.4) ...
Removing obsolete initramfs images
Preparing to unpack .../linux-u-boot-odroidc4-current_22.11.1_arm64.deb ...
Unpacking linux-u-boot-odroidc4-current (22.11.1) over (22.08.6) ...
Setting up linux-image-current-meson64 (22.11.2) ...
Removing obsolete initramfs images
removed '/boot/initrd.img-5.10.147-meson64'
removed '/boot/uInitrd-5.10.147-meson64'
update-initramfs: Generating /boot/initrd.img-6.0.13-meson64
update-initramfs: Converting to U-Boot format
Image Name:   uInitrd
Created:      Tue Dec 20 00:12:06 2022
Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
Data Size:    12444304 Bytes = 12152.64 KiB = 11.87 MiB
Load Address: 00000000
Entry Point:  00000000
'/boot/uInitrd' -> 'uInitrd-6.0.13-meson64'
Setting up linux-dtb-current-meson64 (22.11.2) ...
Setting up linux-u-boot-odroidc4-current (22.11.1) ...
Setting up armbian-firmware (22.11.2) ...
Setting up nano (5.4-2+deb11u2) ...
Processing triggers for libc-bin (2.31-13+deb11u5) ...
[  OK  ] DietPi-Update | APT upgrade

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Installing new DietPi code

[  OK  ] DietPi-Update | Downloading update archive
[  OK  ] DietPi-Update | Unpacking update archive
[  OK  ] DietPi-Update | Removing unused files
[  OK  ] DietPi-Update | Hardening update archive mode
[  OK  ] DietPi-Update | Installing new DietPi scripts
[  OK  ] DietPi-Update | Installing new DietPi system files
[ SUB1 ] DietPi-Set_software > verify_dietpi.txt
[  OK  ] DietPi-Set_software | Downloading current dietpi.txt
[  OK  ] verify_dietpi.txt  | Completed

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Applying incremental patches

[ INFO ] DietPi-Update | Current version : v8.11.2
[ INFO ] DietPi-Update | Latest version  : v8.12.1
[ INFO ] DietPi-Patch | Patching to DietPi v8.12...
[  OK  ] DietPi-Patch | Patched to DietPi v8.12
[ INFO ] DietPi-Update | APT autopurge, please wait...
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[  OK  ] DietPi-Update | APT autopurge
[  OK  ] DietPi-Update | Incremental patching to v8.12.1 completed
[ INFO ] DietPi-Update | Checking for new available live patches
[  OK  ] DietPi-Update | eval echo 1 > /boot/dietpi/.install_stage

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Completed

[ INFO ] DietPi-Update | Current version : v8.12.1
[ INFO ] DietPi-Update | Latest version  : v8.12.1
[  OK  ] DietPi-Survey | Setting in /boot/dietpi.txt adjusted: SURVEY_OPTED_IN=0
[  OK  ] DietPi-Survey | Purging survey data
[ INFO ] DietPi-Update | A reboot is done to finalise the kernel upgrade

I was able to SSH back in and start installing software. That session died. Here’s a Gist containing the output: Odroid C4 kernel panic while installing software via DietPi · GitHub

On the monitor I’ve got connected directly to the device, I see kernel panic - not syncing: stack-protector: kernel stack is corrupted in perf_event_mmap+0x448/0x448. Here’s the full trace:

1181.002418] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted In: perf_event_mmap+0×448/0×448
1181.034827] CPU: 1 PID: 9759 Comm: systemd-veritys Tainted: G C 6.0.13-meson64 #22.11.2
1181.034842] Hardware name: Hardkernel ODROID-C4 (DT)
1181.034846] Call trace:
1181.034849] dump_backtrace+0x12c/0×138
1181.034865] show_stack+0x18/0×50
1181.034872] dump_stack_Iv1+0x68/0×84
1181.034879] dump_stack+0×18/0×34
1181.034884] panic+0x174/0×334
1181.034891] _ktime_get_real_seconds+0x0/0×18
1181.034899] perf_event_aux_event+0x0/Oxe0
1181.034908] mmap_region+0x1c4/0×540
1181.034917] do_mmap+0×310/0×490
1181.034923] vm_mmap_pgoff+0xdc/0×108
1181.034930] ksys_mmap_pgoff+ox1b4/0×200
1181.034936] __arm64_sys_mmap+0×2c/0×40
1181.034943] Invoke_syscall+0×44/0x108
1181.034950] e10_svc_common.constprop.4+0x94/0xf8
1181.034956] do_e10_svc+0×2c/0xb8
1181.106572] e10_svc+0×20/0×50
1181.109588] e10t_64_sync_handler+0×98/0xc0
1181.113728] e10t_64_sync+0X170/0×174
1181.117354] SMP: stopping secondary CPUs
1181.121237] Kernel Offset: disabled
1181.124682] CPU features: 0x6000, 00000028, 19801c86
1181.129425] Memory Limit: none
1181.132458] ---[end Kernel panic - not syncing: stack-protector: Kernel stack Is corrupted in: perf_event _mmap+0x448/0×448 ]---

Now, if I try to power back up, the device will not boot. It tries to recover jornal on the SD card. Runs /scripts/local-bottom but fails running /scripts/init-bottom with Failed to send exit request: Connection refused. It ultimately finishes with No init found. Try passing init= bootarg.

Joulinar · 20 December 2022 08:15

I know we had 1 or 2 SBC who were extremely sensitive on type and quality of SD card. Can’t rememver which one. But @MichaIng knows it better than I.

MichaIng · 20 December 2022 08:43

Looks like an instability caused by the recent switch of kernel to v6.0. Was done just yesterday, if I see it correctly, so we probably need to wait for more reports on Armbian forum. The crash probably caused a filesystem corruption which breaks boot now (/sbin/init not being found).

Initial cause of the crash might have been the load of a specific kernel module. You could retry with installing everything but PHP (as this is where it crashed) and its dependencies. Then check whether it works stable so far, then in case install PHP and see whether it crashes at the same step php-common package configuration. But not sure how to know which kernel module it is, if not contained in the logs.

devon · 20 December 2022 15:28

I’ll try installing everything but PHP. I’m not installing PHP directly, so I’m guessing it’s a dependency of something else. Will I need to avoid installing anything with a web UI? Does that whole web stack get installed whenever another app needs to install a web server?

Joulinar · 20 December 2022 16:33

Yes dependency’s w IP will be installed whenever needed. Like Nextcloud, there we install web server, PHP, database and Redis as dependency automatically.

devon · 22 December 2022 02:59

Sorry. Just had an opportunity to circle back to this. Is there a convenient way to see which software doesn’t depend on PHP so that I can install just those apps? I think just about everything I want to run has a web interface, so there may not be any.

MichaIng · 22 December 2022 04:35

What did you actively select for install? Sonarr is one, I guess, which does not depend on PHP or a dedicated webserver. And some media player/server? Looks like one driven by Python, hence it should work without PHP as well. Might be Nextcloud only, which pulls in PHP.

devon · 22 December 2022 04:49

Yeah, Sonarr is one.

Radarr
qBittorrent
Bazarr
Prowlarr
Jellyfin
Fail2ban
OpenSSH server

That’s all I can remember right now. Maybe Nextcloud is the only one I need to leave out?

MichaIng · 22 December 2022 05:11

Indeed, Nextcloud is the only one .

devon · 23 December 2022 02:56

Remembered a few more as I was going back through :

Navidrome
FreshRSS
Pi-hole
Vaultwarden
Neovim (assuming this one is safe)

Also, what will I need to keep watch on to know if and when it’s safe to try PHP again?

Thanks again for all the help!

devon · 23 December 2022 03:20

OK, without PHP, I’m hitting a different issue when DietPi tries to enable SSH.

[FAILED] DietPi-Software | systemctl enable ssh
[FAILED] DietPi-Software | systemctl enable ssh
[ INFO ] DietPi-BugReport | Generating informative command outputs, please wait...
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory

Bug report: 576cf086-c9d0-469e-be15-b09b67d43798

MichaIng · 23 December 2022 04:29

Those two pull PHP as well. However, I think PHP is unrelated, resp. not the cause.

From your kernel logs:

[ 1622.011496] Unable to handle kernel paging request at virtual address 000ffbffeff75f90
[ 1622.016486] Mem abort info:
[ 1622.019317]   ESR = 0x0000000096000004
[ 1622.023018]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1622.028266]   SET = 0, FnV = 0
[ 1622.031306]   EA = 0, S1PTW = 0
[ 1622.034402]   FSC = 0x04: level 0 translation fault
[ 1622.039234] Data abort info:
[ 1622.042061]   ISV = 0, ISS = 0x00000004
[ 1622.045864]   CM = 0, WnR = 0
[ 1622.048741] [000ffbffeff75f90] address between user and kernel address ranges
[ 1622.055885] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 1622.061333] Modules linked in: snd_soc_hdmi_codec snd_soc_meson_axg_sound_card snd_soc_meson_card_utils snd_soc_meson_axg_tdm_interface snd_soc_meson_axg_frddr snd_soc_meson_axg_fifo snd_soc_meson_g12a_tohdmitx snd_soc_meson_codec_glue snd_soc_meson_axg_tdmout snd_soc_meson_axg_tdm_formatter snd_soc_core ir_nec_decoder snd_compress ac97_bus polyval_ce snd_pcm_dmaengine rc_odroid snd_pcm polyval_generic meson_gxbb_wdt meson_ir rc_core snd_timer snd meson_saradc meson_rng mc soundcore dw_hdmi_i2s_audio ip_tables x_tables autofs4 meson_gxl realtek dwmac_generic reset_meson_audio_arb dwmac_meson8b axg_audio rtc_meson_vrtc sclk_div clk_phase
[ 1622.117316] CPU: 1 PID: 14243 Comm: Prowlarr Not tainted 6.0.13-meson64 #22.11.2
[ 1622.124643] Hardware name: Hardkernel ODROID-C4 (DT)
[ 1622.129561] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1622.136460] pc : __mod_memcg_lruvec_state+0x30/0xf0
[ 1622.141286] lr : __mod_memcg_lruvec_state+0x30/0xf0
[ 1622.146115] sp : ffff80000a6e3920
[ 1622.149392] x29: ffff80000a6e3920 x28: ffff000001cec960 x27: ffff80000a6e3be8
[ 1622.156470] x26: 0000ffff05c00000 x25: 0000ffff05b08000 x24: ffff80000a6e3b58
[ 1622.163543] x23: 0020000092297bc3 x22: 00000000ffffffff x21: 0000000000000012
[ 1622.170615] x20: ffffffffffffffff x19: ffffffffffffffff x18: 0000000000000000
[ 1622.177687] x17: 00000000be8b6ea9 x16: 0000000011dc6402 x15: 0000000000000001
[ 1622.184761] x14: 0000000000000002 x13: 000000000004bf83 x12: 0000000000000000
[ 1622.191833] x11: 0000000000000000 x10: 0000000000000a80 x9 : ffff0000e47b5318
[ 1622.198905] x8 : fffffc00005858c8 x7 : ffff8000db194000 x6 : ffffffffffffffff
[ 1622.205977] x5 : ffff0000010ec000 x4 : ffff8000096211a3 x3 : ffff00000106c000
[ 1622.213051] x2 : 00107bff14de1f90 x1 : 0000000000000090 x0 : ffff8000db194000
[ 1622.220120] Call trace:
[ 1622.222534]  __mod_memcg_lruvec_state+0x30/0xf0
[ 1622.227026]  __mod_lruvec_state+0x40/0x58
[ 1622.230987]  __mod_lruvec_page_state+0x80/0xd0
[ 1622.235391]  page_remove_rmap+0x240/0x5c0
[ 1622.239358]  unmap_page_range+0x45c/0xaa0
[ 1622.243323]  unmap_single_vma+0x4c/0xc0
[ 1622.247116]  unmap_vmas+0xf4/0x118
[ 1622.250485]  exit_mmap+0x80/0x148
[ 1622.253757]  mmput+0x6c/0x138
[ 1622.256689]  do_exit+0x278/0x970
[ 1622.259881]  do_group_exit+0x34/0xa0
[ 1622.263420]  get_signal+0x778/0x7d8
[ 1622.266867]  do_notify_resume+0x218/0x4a8
[ 1622.270834]  el0_da+0x50/0x58
[ 1622.273772]  el0t_64_sync_handler+0x68/0xc0
[ 1622.277907]  el0t_64_sync+0x170/0x174
[ 1622.281533] Code: f9420405 f94788a2 8b010042 97ffeac9 (f8626804) 
[ 1622.287572] ---[ end trace 0000000000000000 ]---
[ 1622.292255] note: Prowlarr[14243] exited with preempt_count 1
[ 1622.298030] Fixing recursive fault but reboot is needed!
[ 1622.303165] ------------[ cut here ]------------
[ 1622.307664] Voluntary context switch within RCU read-side critical section!
[ 1622.307678] WARNING: CPU: 1 PID: 14243 at kernel/rcu/tree_plugin.h:318 rcu_note_context_switch+0x348/0x3d0
[ 1622.324141] Modules linked in: snd_soc_hdmi_codec snd_soc_meson_axg_sound_card snd_soc_meson_card_utils snd_soc_meson_axg_tdm_interface snd_soc_meson_axg_frddr snd_soc_meson_axg_fifo snd_soc_meson_g12a_tohdmitx snd_soc_meson_codec_glue snd_soc_meson_axg_tdmout snd_soc_meson_axg_tdm_formatter snd_soc_core ir_nec_decoder snd_compress ac97_bus polyval_ce snd_pcm_dmaengine rc_odroid snd_pcm polyval_generic meson_gxbb_wdt meson_ir rc_core snd_timer snd meson_saradc meson_rng mc soundcore dw_hdmi_i2s_audio ip_tables x_tables autofs4 meson_gxl realtek dwmac_generic reset_meson_audio_arb dwmac_meson8b axg_audio rtc_meson_vrtc sclk_div clk_phase
[ 1622.381863] CPU: 1 PID: 14243 Comm: Prowlarr Tainted: G      D            6.0.13-meson64 #22.11.2
[ 1622.390658] Hardware name: Hardkernel ODROID-C4 (DT)
[ 1622.395574] pstate: 404000c9 (nZcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1622.402473] pc : rcu_note_context_switch+0x348/0x3d0
[ 1622.407390] lr : rcu_note_context_switch+0x348/0x3d0
[ 1622.412306] sp : ffff80000a6e3500
[ 1622.415580] x29: ffff80000a6e3500 x28: ffff8000082a9fac x27: ffff80000933bd18
[ 1622.422656] x26: ffff80000933bd10 x25: 0000000000000000 x24: 0080000000000000
[ 1622.429727] x23: ffff8000080c64f8 x22: ffff800009482598 x21: 0000000000000000
[ 1622.436794] x20: ffff000002764b00 x19: ffff0000e47b1c00 x18: ffff80000989b648
[ 1622.443867] x17: 0000000000000020 x16: 0000000000000000 x15: 00000000000001ce
[ 1622.450941] x14: ffff80000a6e32b0 x13: 00000000ffffffea x12: ffff80000990b640
[ 1622.458017] x11: 0000000000000003 x10: 0000000000000001 x9 : ffff8000098f3658
[ 1622.465090] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : ffff8000098f3600
[ 1622.472158] x5 : ffff0000e47a9a00 x4 : 0000000000000000 x3 : ffff8000db194000
[ 1622.479235] x2 : ffff000002764b00 x1 : 828779c47fd3c800 x0 : 0000000000000000
[ 1622.486308] Call trace:
[ 1622.488723]  rcu_note_context_switch+0x348/0x3d0
[ 1622.493290]  __schedule+0xac/0x6e8
[ 1622.496659]  do_task_dead+0x48/0x50
[ 1622.500103]  make_task_dead+0xd0/0xf8
[ 1622.503732]  die+0x1fc/0x230
[ 1622.506573]  die_kernel_fault+0x26c/0x35c
[ 1622.510540]  __do_kernel_fault+0xfc/0x180
[ 1622.514508]  do_bad_area+0x64/0x78
[ 1622.517876]  do_translation_fault+0x38/0x68
[ 1622.522016]  do_mem_abort+0x40/0x90
[ 1622.525465]  el1_abort+0x44/0x68
[ 1622.528657]  el1h_64_sync_handler+0x64/0xe8
[ 1622.532797]  el1h_64_sync+0x64/0x68
[ 1622.536247]  __mod_memcg_lruvec_state+0x30/0xf0
[ 1622.540732]  __mod_lruvec_state+0x40/0x58
[ 1622.544700]  __mod_lruvec_page_state+0x80/0xd0
[ 1622.549099]  page_remove_rmap+0x240/0x5c0
[ 1622.553067]  unmap_page_range+0x45c/0xaa0
[ 1622.557034]  unmap_single_vma+0x4c/0xc0
[ 1622.560827]  unmap_vmas+0xf4/0x118
[ 1622.564194]  exit_mmap+0x80/0x148
[ 1622.567468]  mmput+0x6c/0x138
[ 1622.570401]  do_exit+0x278/0x970
[ 1622.573594]  do_group_exit+0x34/0xa0
[ 1622.577130]  get_signal+0x778/0x7d8
[ 1622.580580]  do_notify_resume+0x218/0x4a8
[ 1622.584547]  el0_da+0x50/0x58
[ 1622.587481]  el0t_64_sync_handler+0x68/0xc0
[ 1622.591621]  el0t_64_sync+0x170/0x174
[ 1622.595245] ---[ end trace 0000000000000000 ]---
[ 1624.792082] systemd-journald[785]: Failed to send WATCHDOG=1 notification message: Connection refused
[ 1682.158136] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1682.160401] rcu: 	Tasks blocked on level-0 rcu_node (CPUs 0-3): P14243/2:b..l
[ 1682.167474] 	(detected by 3, t=15004 jiffies, g=114725, q=7326 ncpus=4)
[ 1719.281094] systemd-journald[785]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected

A kernel bug, resp. hardware instability hit more likely with recent kernel.

So far I couldn’t find a related report at Armbian forum. Testing a different SD card and PSU might be worth it. Ah you did switch SD card already. E.g. here a report about failing reboot, solved with different SD card: Odroid C4 will not reboot after any sort of kernel update - have tried running nand-sata-install - Amlogic meson - Armbian Community Forums

Otherwise, reverting to Linux 5.10:

cd /tmp
curl -Lo linux.deb 'https://apt.armbian.com/pool/main/l/linux-5.10.147-meson64/linux-image-current-meson64_22.08.4_arm64.deb'
curl -Lo dtb.deb 'https://apt.armbian.com/pool/main/l/linux-5.10.147-meson64/linux-dtb-current-meson64_22.08.4_arm64.deb'
dpkg -i *.deb
apt-mark hold linux-image-current-meson64 linux-dtb-current-meson64
rm *.deb
reboot

devon · 23 December 2022 22:36

I don’t have a different PSU to try, but I have tried a couple of SD cards as you noted. I also tried the kernel downgrade, and after rebooting, I have this:

×5: 8888088080888064 ×4 : ffff000084aaf506
×5: 0000000000000064 ×4: 0800008900000000
*3: 00000808ffffffff *2 : ffffBB@Bab29a000
xI: ffff880BBBdbldBe ×8 : 80BffdffbffaaB68
Call trace:
mnt_drop_urite+8×54/0x128
do_unlinkat+0x1a8/0×2a8
_arm64_sys_unlinkat+8x40/BxB0
ell_svc-common, constprop. 4+6x6c/Bx188
do_e10_svc+0×24/0×90
elf_svc+0×14/0×20
e10_sync_handler+0×98/0xb8
el0_sync+0x160/0×180
Code: 12880083 9401840 d53cd042 86020000 (bB23001f)
---[end trace 2927f4f95f673d0 1-
note: systend-udevdt15611 exited with preempt_count 2
Unable to handle kernel paging request at virtual address ffffbdffbffc8280
Men abort info:
ESR = 0×96000004
EC = 0x25; DABT (current EL), IL = 32 bits
Started Network Time Synchronization. [4.9774023] SET = 0, FnV = 0
EA = 0, SIPT = 8
Data abort info:
ISV - 9, ISS - 8x000000B4
CM = 0, AhR = 0
swapper pgtable: 4k pages, 48-bit VAs, pgdp-0000000035423000
[ffffbdffbffc8280] pgd=0000000000000000 pg4d=0000000000000000
Internal error: Oops: 96000004 [#2] PREEMPT SMP
Modules linked in: display connector Ip-tables X_tables autofs4 axg_audio spidev sclk_div clk_phase rtc_meson_vrtc reset_meson_audio_arb
CPU: 2 PID: 1540 Comm: systend-random- Tainted; G D 5.10.147-meson64 #22.08.4
Hardware name: Hardkernel ODROID-C4 (DT)
pstate: 48400089 (nZcv daIf +PAN -UAO - TCO BTYPE=--)
pc  : __cgroup_account_cputime+0x20/0x40
Ir :  _coroup_account_cputIme+0x18/0x40
sp ffff8000121ebc10
×29: ffff8000121ebc10 ×20: ffff000001149d00
×27: ffff000001142c8 ×26:ffff9000bc7eZec0
×25: ffff80001134cb80 ×24: ffff0000bc7e2e40
×23: ffff0000bc72c0 ×22; ffff8000121ebdeB
x21: ffff000003e74600 ×20: 0000800880066248
×19: ffff800001149d00 ×18: 000000000000001c
×17: 00000006a59c914 ×16: 0000000048464323
×15: 00000000d43e1dB3 ×14: ffff900011967ade
×13: 00000000c08490d ×12: ffff800011867a60
×11: 8800008086cd31ef ×10: 0000000062ccd482
×9: 000000008430ad65 x8: 000000904646d33a
×7: 80080000c fer8605 хб:000000000857d4cc
×5: ffff000000030000 X4: 0000000000000248
x3: ffffbdffbffc8270 x2: ffff000001149d00
x1: ffff8000ab262000 x0: ffff900880830000
Call trace:
__cgroup_account_cputime+0x20/0×40
update_curr+0xe8/0x160
pick next_task_fair+0x108/0×378
__schedule+0x15c/0x768
schedule+0x70/0x108
wait_for_random_bytes.part.17+0x108/0x200
__arm64_sys_getrandom+0x9c/0xc8
e10_svc_common.constprop.4+0x8c/0x180
do_e10_svc+0x24/0x90
e10_svc+0x14/0x20
e10_sync_handler+0x90/0xb8
e10_sync+0x160/0x180
Code: 910003fd 97ffff03 aa0003e3 aa0503e0 (f9400862)
---[ end trace 2927ff4f95f673d1 ]---
note: systemd-random-[1540] exited with preempt_count 3

This was grabbed from a screen via OCR. Tried to go through and correct, but I can’t promise I got everything. Hopefully it’s enough.

MichaIng · 24 December 2022 01:59

I guess there was something broken with the last crash already. Probably downgrading the kernel after flashing a new image (and without installing any software at first) or on a backup from before crash works better.

I can also not get much out of the kernel errors. Only other thing I can suggest for now is to test an Armbian image: Odroid C4 – Armbian
Upgrade the kernel:

apt update
apt full-upgrade
reboot

Of course there is no dietpi-software, but probably you can install some parts manually or put some load onto the system in other ways to try triggering the crash, or failing boot with the first SD card. Armbian then has a bug tracking guideline which results in a forum post: Bug reporting – Armbian
Do not mention DietPi in it, they do not like us forwarding bug reports, even when obviously a kernel and/or bootloader issue.

I there is no fix until 2. January week, I’ll create an image with Linux 5.19 packages installed an set on hold.