AdguardHome + Unbound unstable connections and cannot load websites sometimes

we never tried to update as we did the opposite, a downgrade :wink:
The reinstall was a try to get back missing files after running AdGuardHome -s uninstal as I was not sure what exactly it will do.

Yes I got that. Seems AdGuardHome -s uninstall did not remove the executable, so the reinstall did not install a new one either.

So question remains what it actually does :sweat_smile:, and why extracting the v0.107.57 tarball into /mnt/dietpi_userdata/adguardhome (which should include the executable, of course) did not change something about v0.167.62 running. Maybe AdGuardHome -s install implies an update to the latest version?

Something to experiment with, and add to our docs whether/how to (not) use them. Reminds me of the mess tailscale -up is creating.

1 Like

These are my DNS-Settings/Encryption in AdguardHome:







Doesn’t looks like upstream DoH is enabled

What is this “Configured in …” entry in upstream DNS? Is this even valid? Also AGH should not use the upstream DNS servers used by Unbound, but should use Unbound. I would expect this entry instead, for AGH to use Unbound (only):

127.0.0.1:5335

This is whats inside that conf file:

 cat /mnt/dietpi_userdata/adguardhome/dietpi-unbound.conf
127.0.0.1:5335

If I uninstall Unbound for example it will appear there as that entry + Quad9. For example the upstream would have been:
127.0.0.1:5335
9.9.9.9

1 Like

Ah right makes sense. So this is how the GUI shows a drop-in config. Checked our code, so it is the upstream_dns_file setting which makes it look like this.

So if despite this setup, https://family.adguard-dns.com:443/dns-query is queried, then something is wrong with AdGuard Home indeed: GitHub · Where software is built

This sounds similar: High CPU Usage and Service Crash in v0.107.61 Under TLS Handshake Failures and DNS Upstream Timeouts · Issue #7800 · AdguardTeam/AdGuardHome · GitHub

They also have a family.adguard-dns query in the logs. Can you check whether you see high CPU load in htop as well? Usually a DNS server should use almost no CPU time at all. A Raspberry Pi Zero (1) is supposed to serve up to 250 clients without problems … at least this is the one, in the meantime old, Pi-hole case I still have in mind :sweat_smile:.




Check a few htop SS.

Okay, CPU usage is normal in your case. While you have >50% of RAM usable, the swap file is nearly full. Looks like there are some RAM usage peaks. Not sure whether related, but you could raise the swap file size and check back in a day or two whether that additional space has been used at all or not. E.g. for 1 GiB size (currently it is 149 MiB, filling the gab to 2 GiB overall memory):

/boot/dietpi/func/dietpi-set_swapfile 1024

I have also noticed a laggy ssh. Tried to ping from other machine inside network and response is fine. raspberry is connected through lan not wifi.

ping 10.0.0.150

Pinging 10.0.0.150 with 32 bytes of data:
Reply from 10.0.0.150: bytes=32 time=1ms TTL=63
Reply from 10.0.0.150: bytes=32 time<1ms TTL=63
Reply from 10.0.0.150: bytes=32 time<1ms TTL=63
Reply from 10.0.0.150: bytes=32 time<1ms TTL=63

Ping statistics for 10.0.0.150:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms

Ok. done. Will wait and see.

sudo /boot/dietpi/func/dietpi-set_swapfile 1024

 DietPi-Set_swapfile
─────────────────────────────────────────────────────
 Mode: Applying 1024 /var/swap

[ INFO ] DietPi-Set_swapfile | Disabling and deleting all existing swap files
[  OK  ] DietPi-Set_swapfile | swapoff -a
removed '/var/swap'
[  OK  ] DietPi-Set_swapfile | Free space check: path=/var/ | available=13804 MiB | required=1024 MiB
[  OK  ] DietPi-Set_swapfile | Generating new swap space
[ INFO ] DietPi-Set_swapfile | Size = 1024 MiB
[ INFO ] DietPi-Set_swapfile | Path = /var/swap
[  OK  ] DietPi-Set_swapfile | fallocate -l 1024M /var/swap
[  OK  ] DietPi-Set_swapfile | mkswap /var/swap
[  OK  ] DietPi-Set_swapfile | swapon /var/swap
[  OK  ] DietPi-Set_swapfile | chmod 0600 /var/swap
[  OK  ] DietPi-Set_swapfile | eval echo '/var/swap none swap sw' >> /etc/fstab
[  OK  ] DietPi-Set_swapfile | Setting in /boot/dietpi.txt adjusted: AUTO_SETUP_SWAPFILE_SIZE=1024
[  OK  ] DietPi-Set_swapfile | Desired setting in /boot/dietpi.txt was already set: AUTO_SETUP_SWAPFILE_LOCATION=/var/swap
[ INFO ] DietPi-Set_swapfile | Setting /tmp tmpfs size: 1461 MiB
[  OK  ] DietPi-Set_swapfile | Setting in /etc/fstab adjusted: tmpfs /tmp tmpfs size=1461M,noatime,lazytime,nodev,nosuid,mode=1777
[  OK  ] DietPi-Set_swapfile | systemctl daemon-reload
[  OK  ] DietPi-Set_swapfile | mount -o remount /tmp

I wonder where all the physical memory is being used. Maybe you can check following

ps -eo pid,comm,rss --sort=-rss | awk 'NR==1 {printf "%-8s %-20s %-10s\n", $1, $2, "RSS_MB"} NR>1 {printf "%-8s %-20s %-10.2f\n", $1, $2, $3/1024}' | head -n 10

Here is goes:

ps -eo pid,comm,rss --sort=-rss | awk 'NR==1 {printf "%-8s %-20s %-10s\n", $1, $2, "RSS_MB"} NR>1 {printf "%-8s %-20s %-10.2f\n", $1, $2, $3/1024}' | head -n 10
PID      COMMAND              RSS_MB
181081   AdGuardHome          185.71
828096   influxd              174.53
829109   grafana              124.34
591505   telegraf             55.83
830279   node                 43.61
828143   dockerd              41.89
828760   postgres             34.48
487      containerd           20.33
925635   postgres             20.25

Btw, what I missed yesterday: what is this container which seems to accumulate zombie curl processes for a daily health check or so? :smile:. Not a big problem, they consume no memory, but looks like the curl process is not terminated properly, hence remains in the process list as zombie, a new one every time it runs. Something the maintainers of this container should have a look into.

That’s probably one of the docker containers if I have to guess its immich as its the most new one and almost beta testing it:

sudo docker ps
CONTAINER ID   IMAGE                                      COMMAND                  CREATED         STATUS                                 PORTS                                            NAMES
46399afbce40   lscr.io/linuxserver/duckdns:latest         "/init"                  7 months ago    Up 15 hours                                                                             duckdns
dbab70baef6e   ghcr.io/immich-app/immich-server:release   "tini -- /bin/bash s…"   9 months ago    Up About a minute (health: starting)   0.0.0.0:2283->3001/tcp                           immich_server
fe6b3e39f6af   prodrigestivill/postgres-backup-local:14   "/init.sh"               9 months ago    Up 15 hours (healthy)                  5432/tcp                                         immich_db_dumper
70130235e812   tensorchord/pgvecto-rs:pg14-v0.2.0         "docker-entrypoint.s…"   9 months ago    Up 15 hours (healthy)                  5432/tcp                                         immich_postgres
7142af66acb1   jc21/nginx-proxy-manager:latest            "/init"                  10 months ago   Up 15 hours                            0.0.0.0:80-81->80-81/tcp, 0.0.0.0:443->443/tcp   nginxproxymanager

From the container ID, it is prodrigestivill/postgres-backup-local.

Here it is: docker-postgres-backup-local/debian.Dockerfile at 8181772cbe50ebaaa1950f585b915035a2c02128 · prodrigestivill/docker-postgres-backup-local · GitHub

So it uses the native Docker health check feature and basically a copy&paste from its official docs: Dockerfile reference | Docker Docs

In so far it is surprising that the native feature used with an example from the official docs with a standard tool leads to such zombies, in this case potentially one added every 5 minutes.

Also note that this contains is, same as tensorchord/pgvecto-rs marked as “unhealthy”, so that curl request fails or times out.

The only difference is the added port as variable … and that might be the issue:

  • In htop you seen the variable name in the call, while there should clearly be the port number 8080 shown.
  • Not sure whether it should show up expended as “8080” already, as then the URL would be currently invalid.
  • In any case, the container logs should contain some logs about the failed curl calls.
  • If really the variable is not expanded as expected, probably overriding things in the docker compose file fixes this:
        pgbackups:
            ...
            healthcheck:
              test: curl -f http://localhost:8080/ || exit 1
              interval: 5m
              timeout: 3s
    
    in the hope it overwrites of what is defined in the Docker file. Maybe @Joulinar knows better how this is supposed to work.

And do you think that’s what’s happening in the end, or the cause of the problem? Because Im sure everything was still as it is/was and the only thing I did was to update Adguardhome from the UI and then everything got slower in general… SSH console connection, DNS not processing well and having lots of disconnects, etc.

I am not sure if there is one single cause or if it is a combination of several factors. What we see, the RAM consumption is sometimes very high, so that the system also starts to swap. The use of swap has quite a high impact on performance and should generally be avoided. Since the swap is also completely occupied in the screenshots, a situation can arise in which the system is no longer able to answer requests and then simply runs out of memory. How does the system behave after the swap area has been increased to 1GB? Is it still slow? Maybe also try temporarily stopping the 3 Docker containers from immich. What is the duckdns container actually used for?

If for DDNS, better use dietpi-ddns, which should be much lighter than a dedicated Docker container for simple DDNS updates :wink:.

I thought so too, but wanted to be on the safe side so I asked. And I agree, you don’t need a container for that. A waste of system resources in that case.

yeah just did it… I was testing when I was using my docker compose… and I left it there… I think my problem was immich_server as it was consuming probably the whole resources of my raspberry. I’m still testing with all the dockers down and only nginx up… I think I need a better machine for immich