Hello,
I’ve been running an installation of Mariadb for quite some time now & a couple days ago, I (suspect) a power failure occurred, which may have corrupted a sector of my external drive, where data is located.
I’m in the process of running a badblocks check, but,
Since then, I performed a long smartctl scan:
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 2639
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 67888
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 087 087 000 Old_age Always - 5354
10 Spin_Retry_Count 0x0033 253 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 340
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 6
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 219
193 Load_Cycle_Count 0x0032 077 077 000 Old_age Always - 237303
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 48 (Min/Max 18/53)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 736
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 092 092 000 Old_age Always - 3564
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 175
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
journalctl gives me:
Sep 03 16:18:30 Grimlock systemd[1]: Starting MariaDB 10.5.15 database server...
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 430052 ...
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] InnoDB: Uses event mutexes
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] InnoDB: Number of pools: 1
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] InnoDB: Using ARMv8 crc32 instructions
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] InnoDB: Using Linux native AIO
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] InnoDB: Initializing buffer pool, total size = 8388608, chunk size = 8388608
Sep 03 16:18:31 Grimlock mariadbd[430052]: 2022-09-03 16:18:31 0 [Note] InnoDB: Completed initialization of buffer pool
Sep 03 16:18:35 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=3s
Sep 03 16:18:35 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current]
Sep 03 16:18:35 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0
Sep 03 16:18:35 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 43 82 9d 20 00 00 08 00
Sep 03 16:18:35 Grimlock kernel: blk_update_request: critical medium error, dev sda, sector 1132633376 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep 03 16:18:38 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=3s
Sep 03 16:18:38 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current]
Sep 03 16:18:38 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0
Sep 03 16:18:38 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 43 82 9d 20 00 00 08 00
Sep 03 16:18:38 Grimlock kernel: blk_update_request: critical medium error, dev sda, sector 1132633376 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep 03 16:18:38 Grimlock mariadbd[430052]: 2022-09-03 16:18:38 0 [Warning] InnoDB: 65536 bytes should have been read. Only 40960 bytes read. Retrying for the remaining bytes.
Sep 03 16:18:42 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=3s
Sep 03 16:18:42 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current]
Sep 03 16:18:42 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0
Sep 03 16:18:42 Grimlock kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 43 82 9d 20 00 00 08 00
Sep 03 16:18:42 Grimlock kernel: blk_update_request: critical medium error, dev sda, sector 1132633376 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep 03 16:18:42 Grimlock mariadbd[430052]: 2022-09-03 16:18:42 0 [Warning] InnoDB: Retry attempts for reading partial data failed.
Sep 03 16:18:42 Grimlock mariadbd[430052]: 2022-09-03 16:18:42 0 [ERROR] InnoDB: Operating system error number 5 in a file operation.
Sep 03 16:18:42 Grimlock mariadbd[430052]: 2022-09-03 16:18:42 0 [ERROR] InnoDB: Error number 5 means 'Input/output error'
Sep 03 16:18:42 Grimlock mariadbd[430052]: 2022-09-03 16:18:42 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
Sep 03 16:18:42 Grimlock mariadbd[430052]: 2022-09-03 16:18:42 0 [ERROR] InnoDB: File (unknown): 'read' returned OS error 205. Cannot continue operation
Sep 03 16:18:42 Grimlock mariadbd[430052]: 220903 16:18:42 [ERROR] mysqld got signal 6 ;
Sep 03 16:18:42 Grimlock mariadbd[430052]: This could be because you hit a bug. It is also possible that this binary
Sep 03 16:18:42 Grimlock mariadbd[430052]: or one of the libraries it was linked against is corrupt, improperly built,
Sep 03 16:18:42 Grimlock mariadbd[430052]: or misconfigured. This error can also be caused by malfunctioning hardware.
Sep 03 16:18:42 Grimlock mariadbd[430052]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Sep 03 16:18:42 Grimlock mariadbd[430052]: We will try our best to scrape up some info that will hopefully help
Sep 03 16:18:42 Grimlock mariadbd[430052]: diagnose the problem, but since we have already crashed,
Sep 03 16:18:42 Grimlock mariadbd[430052]: something is definitely wrong and this may fail.
Sep 03 16:18:42 Grimlock mariadbd[430052]: Server version: 10.5.15-MariaDB-0+deb11u1
Sep 03 16:18:42 Grimlock mariadbd[430052]: key_buffer_size=33554432
Sep 03 16:18:42 Grimlock mariadbd[430052]: read_buffer_size=131072
Sep 03 16:18:42 Grimlock mariadbd[430052]: max_used_connections=0
Sep 03 16:18:42 Grimlock mariadbd[430052]: max_threads=66
Sep 03 16:18:42 Grimlock mariadbd[430052]: thread_count=0
Sep 03 16:18:42 Grimlock mariadbd[430052]: It is possible that mysqld could use up to
Sep 03 16:18:42 Grimlock mariadbd[430052]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 178043 K bytes of memory
Sep 03 16:18:42 Grimlock mariadbd[430052]: Hope that's ok; if not, decrease some variables in the equation.
Sep 03 16:18:42 Grimlock mariadbd[430052]: Thread pointer: 0x0
Sep 03 16:18:42 Grimlock mariadbd[430052]: Attempting backtrace. You can use the following information to find out
Sep 03 16:18:42 Grimlock mariadbd[430052]: where mysqld died. If you see no messages after this, something went
Sep 03 16:18:42 Grimlock mariadbd[430052]: terribly wrong...
Sep 03 16:18:42 Grimlock mariadbd[430052]: stack_bottom = 0x0 thread_stack 0x49000
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(my_print_stacktrace)[0x555674ee00]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(handle_fatal_signal)[0x555628d114]
Sep 03 16:18:42 Grimlock mariadbd[430058]: addr2line: 'linux-vdso.so.1': No such file
Sep 03 16:18:42 Grimlock mariadbd[430052]: linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7f9cb0e788]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(gsignal)[0x7f9c1512ac]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(abort)[0x7f9c13dea0]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(std::unique_lock<std::mutex>::unlock())[0x5556592574]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(std::unique_lock<std::mutex>::unlock())[0x55565940a8]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x555657ad08]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x555657b318]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55565815e0]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x5556581770]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55565861ec]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55565868dc]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x5555fd59d8]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55565355c8]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(ha_initialize_handlerton(st_plugin_int*))[0x5556290008]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(sys_var_pluginvar::sys_var_pluginvar(sys_var_chain*, char const*, st_plugin_int*, st_mysql_sys_var*))[0x55560d5564]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(plugin_init(int*, char**, int))[0x55560d64b4]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(unireg_abort)[0x5556014094]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(mysqld_main(int, char**))[0x5556019c8c]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(__libc_start_main)[0x7f9c13e218]
Sep 03 16:18:42 Grimlock mariadbd[430052]: ??:0(_start)[0x555600e328]
Sep 03 16:18:42 Grimlock mariadbd[430052]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Sep 03 16:18:42 Grimlock mariadbd[430052]: information that should help you find out what is causing the crash.
Sep 03 16:18:42 Grimlock mariadbd[430052]: Writing a core file...
Sep 03 16:18:42 Grimlock mariadbd[430052]: Working directory at /mnt/HDD1/dietpi_userdata/mysql
Sep 03 16:18:42 Grimlock mariadbd[430052]: Resource Limits:
Sep 03 16:18:42 Grimlock mariadbd[430052]: Limit Soft Limit Hard Limit Units
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max cpu time unlimited unlimited seconds
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max file size unlimited unlimited bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max data size unlimited unlimited bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max stack size 8388608 unlimited bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max core file size 0 unlimited bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max resident set unlimited unlimited bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max processes 3567 3567 processes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max open files 32768 32768 files
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max locked memory 65536 65536 bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max address space unlimited unlimited bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max file locks unlimited unlimited locks
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max pending signals 3567 3567 signals
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max msgqueue size 819200 819200 bytes
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max nice priority 0 0
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max realtime priority 0 0
Sep 03 16:18:42 Grimlock mariadbd[430052]: Max realtime timeout unlimited unlimited us
Sep 03 16:18:42 Grimlock mariadbd[430052]: Core pattern: core
Sep 03 16:18:42 Grimlock systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Sep 03 16:18:42 Grimlock systemd[1]: mariadb.service: Failed with result 'signal'.
Sep 03 16:18:42 Grimlock systemd[1]: Failed to start MariaDB 10.5.15 database server.
Sep 03 16:18:48 Grimlock systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 30.
Sep 03 16:18:48 Grimlock systemd[1]: Stopped MariaDB 10.5.15 database server.
I’ve concluded, InnnoDB can’t read some part of disk & this won’t start. None of the other services I’ve got running (BItwarden, Jackett, etc.) seem affected.
Looking around, there seem to be a hundred ways to fix innoDB or mysql corruption, which I barely understand.
Is there a way to get back to a functioning system?
