Suddenly redis server breaks: Failed to start Advanced key-value store

Troubleshooting
1

Debian 11
Redis version=6.0.16
This is new:
After a reboot everything works, but then I get an odd error from redis server which breaks after a while:

journalctl -f -u redis-server
> -- Journal begins at Thu 2024-10-24 13:07:28 CEST. --
Okt 24 14:03:12 GLOIN systemd[1]: Stopped Advanced key-value store.
Okt 24 14:03:12 GLOIN systemd[1]: Starting Advanced key-value store...
Okt 24 14:03:22 GLOIN systemd[314571]: redis-server.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Host is down
Okt 24 14:03:22 GLOIN systemd[314571]: redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Host is down
Okt 24 14:03:22 GLOIN systemd[1]: redis-server.service: Main process exited, code=exited, status=226/NAMESPACE
Okt 24 14:03:22 GLOIN systemd[1]: redis-server.service: Failed with result 'exit-code'.
Okt 24 14:03:22 GLOIN systemd[1]: Failed to start Advanced key-value store.

If I comment out
ReadOnlyDirectories=/
in the redis systemd service file it would work. But I don’t like to do that.
Any idea?

2

pls fill the troubleshooting template

Required

  • DietPi version | cat /boot/dietpi/.version
  • Distro version | echo $G_DISTRO_NAME $G_RASPBIAN
  • Kernel version | uname -a
  • Architecture | dpkg --print-architecture
  • SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)
3

SNAG-0000
SNAG-00001453×612 130 KB

4

there is some output missing, did you used root?

5

No I used my user account. But it’s zsh

6

DietPi own functions like G_DISTRO_NAME require bash

7

But I told you it’s debian 11

8

I would like to know the device so that I can test it. For whatever reason, you have ignored the troubleshooting template and have not provided all the required information. The template is there for a reason and not just for fun.

But I assume you are using an RPI device?

9

I would like to know the device so that I can test it. For whatever reason, you have ignored the troubleshooting template and have not provided all the required information. The template is there for a reason and not just for fun.

But I assume you are using an RPI device?

10

Ah OK I thought I provided all the info neccessary. Yes it is a Rasberry Pi 4 Model B. I actually thought that that was selfexplaning, sorry.

11

tested on a demo system without issue

root@DietPi4:~# systemctl status redis
● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/redis-server.service.d
             └─dietpi.conf
     Active: active (running) since Thu 2024-10-24 17:55:49 CEST; 1min 36s ago
       Docs: http://redis.io/documentation,
             man:redis-server(1)
   Main PID: 6641 (redis-server)
     Status: "Ready to accept connections"
      Tasks: 5 (limit: 9352)
        CPU: 318ms
     CGroup: /system.slice/redis-server.service
             └─6641 /usr/bin/redis-server 127.0.0.1:6379

this is set on my demo system not causing issues

root@DietPi4:~# cat /lib/systemd/system/redis-server.service | grep ReadOnlyDirectories
ReadOnlyDirectories=/
root@DietPi4:~#

Can you share your service configuration file?

cat /lib/systemd/system/redis-server.service

And can you check if the directory exists

ls -ld /run/systemd/unit-root/

PS: no need to do screen prints, you should be able to copy past everything from terminal directly.

12

I know there is not a problem with the configuration per se. It worked for years without any issues. Also if I reboot the system the redis server would start normally. But after a while he stops.
This is new. I already checked the web where I found the tip to comment out ReadOnlyDirectories=/ but no real explanation what’s causing the problem.

Here is the service file

cat /lib/systemd/system/redis-server.service

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)
[Service]
Type=notify
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf --supervised systemd --daemonize no
PIDFile=/run/redis/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=2755
UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
#ReadOnlyDirectories=/
ReadWritePaths=-/var/lib/redis
ReadWritePaths=-/var/log/redis
ReadWritePaths=-/var/run/redis
NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCEMemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
ProtectSystem=true
ReadWriteDirectories=-/etc/redis
[Install]
WantedBy=multi-user.target
Alias=redis.service

❯ ls -ld /run/systemd/unit-root/

drwx------ 2 root root 40 23. Okt 12:40 /run/systemd/unit-root//
13

you could ask for assistance on Redis Github Maybe they could explain why this happen

14

Ok thank you Joulinar. You have no idea what

ReadOnlyDirectories=/

does and if it is ok to comment it out?

Maybe you can have a look at my other post as well at TigerVNC Server crashes regularly - #5 by Pausentee
and my latest update there. I am getting this error messages and .xsession-errors is growing by 100MB a day because of this.

15

This seems to be a systemd issue, not a Redis issue. The systemd unit limits the dirs the daemon has access to be internally mounting the system root read-only (what the option does you asked for) and mounting only the required dirs read-write via ReadWritePaths options. The first step seems to fail for whatever reason. Can you show the output of this, to see permission modes of all parent dirs of this internal root mount point?

sudo ls -dl / /run /run/systemd
16

drwxr-xr-x 18 root root 4096 14. Apr 2024 /
drwxr-xr-x 33 root root 1040 26. Okt 13:39 /run
drwxr-xr-x 20 root root 480 24. Okt 14:08 /run/systemd

17

That looks fine. The “host is down” message shows up if an NFS mount along those paths (?) or elsewhere is offline: debian - Why is radicale server crashing with `Host is down`-message? - Unix & Linux Stack Exchange
In this example weird is that the mountpoint does not look at all related to where systemd tries to setup to unit mounts :thinking:. However, do you maybe have an NFS or other remote/network mount that is not always online?

18

Yes that is possible. I have a OMV NAS in my network that is only online on demand for media requests and backups.

19

Can you try to temporarily remove or disable its mount? Maybe systemd gets stuck somehow trying to mount it, which affects other mounts as well then. I would consider this a bug, but there is quite some behaviour not seen as bug by systemd but as wrong usage. Probably having an offline NFS mount with auto or x-systemd.automount is seen as such as well.

20

The NAS is not mounted permanently only on demand for daily backups and via cifs.
I use the redis server for my owncloud. The owncloud data dir is symlinked to an attached ssd connected via usb. I thought that the issue maybe comes from there but couldn’t find anything.

❯ cat /etc/mtab

/dev/root / ext4 rw,lazytime,noatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1897628k,nr_inodes=474407,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=772992k,nr_inodes=819200,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
systemd-1 /mnt/usb autofs rw,relatime,fd=52,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
/dev/mmcblk0p1 /boot vfat rw,lazytime,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro 0 0
/dev/sda3 /mnt/usb ext4 rw,lazytime,noatime 0 0
tmpfs /tmp tmpfs rw,lazytime,nosuid,nodev,noatime,size=1048576k 0 0
tmpfs /run/user/1001 tmpfs rw,nosuid,nodev,relatime,size=386492k,nr_inodes=96623,mode=700,uid=1001,gid=1001 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0