First of all, it is not needed to do screen prints, you could copy everything from SSH terminal directly. 
Did the update finished? Did you tried to reboot?
Hey,
sorry for that. I won’t do screen prints next time.
Yes the update did finish but I didn’t reboot because I wanted to check whether to gather insights before the reboot or not for your analysis.
at least according your screen shot, it seems some part of MariaDB are still running. Therefore you are not able to start it again. You could try stopping the service and check log files
systemctl stop mariadb.service
journalctl -u mariadb
Stop doesn’t work. If I try to start it still says that there is running process. The stop doesn’t produce any logs that’s odd right?
Jul 04 15:07:41 xxx systemd[1]: mariadb.service: Found left-over process 18072 (mariadbd) in control group while starting unit. Ignoring.
Jul 04 15:07:41 xxx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 04 15:07:41 xxx systemd[1]: mariadb.service: Found left-over process 52092 (mariadbd) in control group while starting unit. Ignoring.
Jul 04 15:07:41 xxx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 04 15:07:41 xxx systemd[1]: mariadb.service: Will not start SendSIGKILL=no service of type KillMode=control-group or mixed while processes exist
Jul 04 15:07:41 xxx systemd[1]: mariadb.service: Failed to run ‘start-pre’ task: Device or resource busy
Jul 04 15:07:41 xxx systemd[1]: mariadb.service: Failed with result ‘resources’.
Jul 04 15:07:41 xxx systemd[1]: mariadb.service: Unit process 18072 (mariadbd) remains running after unit stopped.
Jul 04 15:07:41 xxx systemd[1]: mariadb.service: Unit process 52092 (mariadbd) remains running after unit stopped.
Jul 04 15:07:41 xxx systemd[1]: Failed to start MariaDB 10.5.15 database server.
The process is stuck because of some race condition.
Jul 04 12:14:01 xxx mariadbd[18072]: 2022-07-04 11:45:49 0 [Warning] InnoDB: A long semaphore wait:
Jul 04 12:14:01 xxx mariadbd[18072]: --Thread 281472669827328 has waited at srv0srv.cc line 1536 for 383.00 seconds the semaphore:
Jul 04 12:14:01 xxx mariadbd[18072]: X-lock on RW-latch at 0xaaaad8087d40 created in file dict0dict.cc line 1036
Jul 04 12:14:01 xxx mariadbd[18072]: a writer (thread id 281473040724224) has reserved it in mode exclusive
Jul 04 12:14:01 xxx mariadbd[18072]: number of readers 0, waiters flag 1, lock_word: 0
Jul 04 12:14:01 xxx mariadbd[18072]: Last time write locked in file dict0stats.cc line 2487
Jul 04 12:14:01 xxx mariadbd[18072]: 2022-07-04 11:49:59 0 [Note] InnoDB: A semaphore wait:
Jul 04 12:14:01 xxx mariadbd[18072]: --Thread 281472669827328 has waited at srv0srv.cc line 1536 for 838.00 seconds the semaphore:
Jul 04 12:14:01 xxx mariadbd[18072]: X-lock on RW-latch at 0xaaaad8087d40 created in file dict0dict.cc line 1036
Jul 04 12:14:01 xxx mariadbd[18072]: a writer (thread id 281473040724224) has reserved it in mode exclusive
Jul 04 12:14:01 xxx mariadbd[18072]: number of readers 0, waiters flag 1, lock_word: 0
Jul 04 12:14:01 xxx mariadbd[18072]: Last time write locked in file dict0stats.cc line 2487
Jul 04 12:14:01 xxx mariadbd[18072]: 2022-07-04 12:02:28 0 [Note] InnoDB: A semaphore wait:
Jul 04 12:14:01 xxx mariadbd[18072]: --Thread 281472678220032 has waited at dict0dict.cc line 938 for 1500.00 seconds the semaphore:
Jul 04 12:14:01 xxx mariadbd[18072]: Mutex at 0xaaaad8087d08, Mutex DICT_SYS created ./storage/innobase/dict/dict0dict.cc:1027, lock var 2
Jul 04 12:14:01 xxx mariadbd[18072]: InnoDB: Pending reads 0, writes 0
Jul 04 12:14:01 xxx mariadbd[18072]: 2022-07-04 12:11:28 0 [Warning] InnoDB: A long semaphore wait:
Jul 04 12:14:01 xxx mariadbd[18072]: --Thread 281472669827328 has waited at srv0srv.cc line 1536 for 1879.00 seconds the semaphore:
Jul 04 12:14:01 xxx mariadbd[18072]: X-lock on RW-latch at 0xaaaad8087d40 created in file dict0dict.cc line 1036
Jul 04 12:14:01 xxx mariadbd[18072]: a writer (thread id 281473040724224) has reserved it in mode exclusive
Jul 04 12:14:01 xxx mariadbd[18072]: number of readers 0, waiters flag 1, lock_word: 0
Jul 04 12:14:01 xxx mariadbd[18072]: Last time write locked in file dict0stats.cc line 2487
Jul 04 12:14:01 xxx mariadbd[18072]: 2022-07-04 12:12:40 0 [Warning] InnoDB: A long semaphore wait:
Jul 04 12:14:01 xxx mariadbd[18072]: --Thread 281472678220032 has waited at dict0dict.cc line 938 for 1804.00 seconds the semaphore:
Jul 04 12:14:01 xxx mariadbd[18072]: Mutex at 0xaaaad8087d08, Mutex DICT_SYS created ./storage/innobase/dict/dict0dict.cc:1027, lock var 2
Jul 04 12:14:01 xxx mariadbd[18072]: 2022-07-04 12:13:26 0 [Note] InnoDB: A semaphore wait:
Jul 04 12:14:01 xxx mariadbd[18072]: --Thread 281472669827328 has waited at srv0srv.cc line 1536 for 1978.00 seconds the semaphore:
Jul 04 12:14:01 xxx mariadbd[18072]: X-lock on RW-latch at 0xaaaad8087d40 created in file dict0dict.cc line 1036
Jul 04 12:31:49 xxx mariadbd[18072]: a writer (thread id 281473040724224) has reserved it in mode exclusive
Jul 04 12:31:49 xxx mariadbd[18072]: number of readers 0, waiters flag 1, lock_word: 0
Jul 04 12:31:49 xxx mariadbd[18072]: Last time write locked in file dict0stats.cc line 2487
Jul 04 12:31:49 xxx mariadbd[18072]: 2022-07-04 12:14:24 0 [Note] InnoDB: A semaphore wait:
Jul 04 12:31:49 xxx mariadbd[18072]: --Thread 281472678220032 has waited at dict0dict.cc line 938 for 1910.00 seconds the semaphore:
Jul 04 12:31:49 xxx mariadbd[18072]: Mutex at 0xaaaad8087d08, Mutex DICT_SYS created ./storage/innobase/dict/dict0dict.cc:1027, lock var 2
Jul 04 12:31:49 xxx mariadbd[18072]: InnoDB: Pending reads 0, writes 0
Jul 04 12:31:49 xxx mariadbd[18072]: 2022-07-04 12:16:02 0 [Warning] InnoDB: A long semaphore wait:
Jul 04 12:31:49 xxx mariadbd[18072]: --Thread 281472669827328 has waited at srv0srv.cc line 1536 for 2166.00 seconds the semaphore:
Jul 04 12:31:49 xxx mariadbd[18072]: X-lock on RW-latch at 0xaaaad8087d40 created in file dict0dict.cc line 1036
Jul 04 12:31:49 xxx mariadbd[18072]: a writer (thread id 281473040724224) has reserved it in mode exclusive
Jul 04 12:31:49 xxx mariadbd[18072]: number of readers 0, waiters flag 1, lock_word: 0
Jul 04 12:31:49 xxx mariadbd[18072]: Last time write locked in file dict0stats.cc line 2487
Jul 04 12:31:49 xxx mariadbd[18072]: 2022-07-04 12:19:34 0 [Warning] InnoDB: A long semaphore wait:
Jul 04 12:31:49 xxx mariadbd[18072]: --Thread 281472678220032 has waited at dict0dict.cc line 938 for 2279.00 seconds the semaphore:
Jul 04 12:31:49 xxx mariadbd[18072]: Mutex at 0xaaaad8087d08, Mutex DICT_SYS created ./storage/innobase/dict/dict0dict.cc:1027, lock var 2
Jul 04 12:31:49 xxx mariadbd[18072]: 2022-07-04 12:23:20 0 [Note] InnoDB: A semaphore wait:
Jul 04 12:31:49 xxx mariadbd[18072]: --Thread 281472669827328 has waited at srv0srv.cc line 1536 for 2711.00 seconds the semaphore:
Jul 04 12:31:49 xxx mariadbd[18072]: X-lock on RW-latch at 0xaaaad8087d40 created in file dict0dict.cc line 1036
Jul 04 12:31:49 xxx mariadbd[18072]: a writer (thread id 281473040724224) has reserved it in mode exclusive
Jul 04 12:31:49 xxx mariadbd[18072]: number of readers 0, waiters flag 1, lock_word: 0
Jul 04 12:31:49 xxx mariadbd[18072]: Last time write locked in file dict0stats.cc line 2487
Jul 04 12:31:49 xxx mariadbd[18072]: 2022-07-04 12:27:49 0 [Note] InnoDB: A semaphore wait:
Jul 04 12:31:49 xxx mariadbd[18072]: --Thread 281472678220032 has waited at dict0dict.cc line 938 for 2829.00 seconds the semaphore:
Jul 04 12:31:49 xxx mariadbd[18072]: Mutex at 0xaaaad8087d08, Mutex DICT_SYS created ./storage/innobase/dict/dict0dict.cc:1027, lock var 2
Jul 04 12:31:49 xxx mariadbd[18072]: InnoDB: Pending reads 0, writes 0
Jul 04 12:31:49 xxx mariadbd[18072]: 2022-07-04 12:31:49 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted 600 seconds. We intentionally crash the server because it appears to be hung.
Jul 04 12:31:49 xxx mariadbd[18072]: 220704 12:31:49 [ERROR] mysqld got signal 6 ;
Jul 04 12:31:49 xxx mariadbd[18072]: This could be because you hit a bug. It is also possible that this binary
Jul 04 12:31:49 xxx mariadbd[18072]: or one of the libraries it was linked against is corrupt, improperly built,
Jul 04 12:31:49 xxx mariadbd[18072]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jul 04 12:31:49 xxx mariadbd[18072]: To report this bug, see MariaDB Community Bug Reporting - MariaDB Knowledge Base
Jul 04 12:31:49 xxx mariadbd[18072]: We will try our best to scrape up some info that will hopefully help
Jul 04 12:31:49 xxx mariadbd[18072]: diagnose the problem, but since we have already crashed,
Jul 04 12:31:49 xxx mariadbd[18072]: something is definitely wrong and this may fail.
Jul 04 12:31:49 xxx mariadbd[18072]: Server version: 10.5.15-MariaDB-0+deb11u1
Jul 04 12:31:49 xxx mariadbd[18072]: key_buffer_size=134217728
Jul 04 12:31:49 xxx mariadbd[18072]: read_buffer_size=131072
Jul 04 12:31:49 xxx mariadbd[18072]: max_used_connections=10
Jul 04 12:31:49 xxx mariadbd[18072]: max_threads=153
Jul 04 12:31:49 xxx mariadbd[18072]: thread_count=2
Jul 04 12:31:49 xxx mariadbd[18072]: It is possible that mysqld could use up to
Jul 04 12:31:49 xxx mariadbd[18072]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467879 K bytes of memory
Jul 04 12:31:49 xxx mariadbd[18072]: Hope that’s ok; if not, decrease some variables in the equation.
Jul 04 12:31:49 xxx mariadbd[18072]: Thread pointer: 0x0
Jul 04 12:31:49 xxx mariadbd[18072]: Attempting backtrace. You can use the following information to find out
Jul 04 12:31:49 xxx mariadbd[18072]: where mysqld died. If you see no messages after this, something went
Jul 04 12:31:49 xxx mariadbd[18072]: terribly wrong…
Jul 04 12:31:49 xxx mariadbd[18072]: stack_bottom = 0x0 thread_stack 0x49000
Jul 04 12:31:50 xxx mariadbd[18072]: /usr/sbin/mariadbd(my_print_stacktrace+0x30)[0xaaaad784ee00]
Jul 04 12:31:51 xxx mariadbd[18072]: /usr/sbin/mariadbd(handle_fatal_signal+0x460)[0xaaaad738d114]
The database seems to be stuck, you could either kill the remaining process or restart your system. But I hope your database files are still ok in this status.
Everything works fine after a reboot. Can I somehow check if the database integrity is fine or is this done automatically via dietpi?
DietPi is not doing anything with the database, except to install the software. Rest is managed by the database software internally. But for more inside, how MariaDB is managing such situation, better to ask on a MariaDB specialised board/forum.
1 Like
