[Solved] SNMP suddenly stopped working to get free space left on a disc

Garret · 2 February 2020 17:25

I have two raspberry pis. One has Dietpi installed while the other has Hassio.

I am using SNMP installed on Dietpi to get the OID of a free space left on a specific disc on Dietpi. In this way, I could check on my Hassio installation how much disc space is left on Dietpi through the SNMP integration.

Since about a week I can see that I cannot retrieve anymore that information but I always get a Timeout: No Response from localhost

For instance:

root@DietPi:~# df | grep mnt
/dev/sda1      121169024  37773968  83378672  32% /mnt/data
/dev/mmcblk0p1     64366     53024     11342  83% /mnt/0095-112D
/dev/mmcblk0p5     30701      3249     25159  12% /mnt/a54b8bb4-b9ae-474d-9a87-3b0b590dc77f
/dev/sdb1      960379920 364306840 596056696  38% /mnt/media
overlay         15282608   2961432  11521816  21% /mnt/dietpi_userdata/docker-data/overlay2/d2521957b38eab1bb69b886a79e664e0979d89efb053d7c7a14a6c05c440beeb/merged
root@DietPi:~# snmpwalk -On  -v 2c -c public localhost .1.3.6.1.4.1.2021.9  | grep  /dev/sdb
Timeout: No Response from localhost
root@DietPi:~#

This is the snmpd status:

root@DietPi:~# service snmpd status
● snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
   Loaded: loaded (/lib/systemd/system/snmpd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-02-01 13:27:48 CET; 1 day 4h ago
  Process: 566 ExecStartPre=/bin/mkdir -p /var/run/agentx (code=exited, status=0/SUCCESS)
 Main PID: 576 (snmpd)
    Tasks: 1 (limit: 4915)
   Memory: 3.8M
      CPU: 1min 29.989s
   CGroup: /system.slice/snmpd.service
           └─576 /usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f

Feb 02 18:14:50 DietPi snmpd[576]: Connection from UDP: [192.168.40.100]:55149->[192.168.40.200]:161
Feb 02 18:14:50 DietPi snmpd[576]: Connection from UDP: [192.168.40.100]:55149->[192.168.40.200]:161
Feb 02 18:23:08 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /mnt/batocera: No such device
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /run/docker/netns/ad525e0f7c50: Permission denied
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
root@DietPi:~#

Would you know what I am doing wrong?

Joulinar · 2 February 2020 20:03

Hi,

not sure if this is related but you might need to check these 2 error messages

Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /mnt/batocera: No such device
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /run/docker/netns/ad525e0f7c50: Permission denied

Garret · 2 February 2020 20:29

Hi Joulinar, I had also noticed those errors but to me they don’t look important as they are related to partitions I am not interested (I am only interested in /mnt/sdb1).

I cannot understand why it has been working for me but it is just since a week or two that suddenly I get that error now.

Joulinar · 2 February 2020 21:06

well you receive a Timeout on your request. So it could be, that the system is not able to complete the request because of the 2 missing file systems bot being available.

What happen if you execute just a minimized snmpwalk command

snmpwalk -On  -v 2c -c public localhost

maybe there are some more information within journalctl

journalctl -u snmpd.service

Garret · 2 February 2020 21:24

Hi Joulinar, by giving the first command you mentioned I get a very long list of OIDs but in the end I can see there is still “Timeout: No Response from localhost”. I have cut the code because it was very long otherwise.

root@DietPi:~# snmpwalk -On  -v 2c -c public localhost
.1.3.6.1.2.1.1.1.0 = STRING: "Linux DietPi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l"
.1.3.6.1.2.1.1.2.0 = OID: .1.3.6.1.4.1.8072.3.2.10
.1.3.6.1.2.1.1.3.0 = Timeticks: (277734) 0:46:17.34
.1.3.6.1.2.1.1.4.0 = STRING: "Me <me@example.org>"
.1.3.6.1.2.1.1.5.0 = STRING: "DietPi"
.1.3.6.1.2.1.1.6.0 = STRING: "Sitting on the Dock of the Bay"
.1.3.6.1.2.1.1.7.0 = INTEGER: 72
.1.3.6.1.2.1.1.8.0 = Timeticks: (28) 0:00:00.28
.1.3.6.1.2.1.1.9.1.2.1 = OID: .1.3.6.1.6.3.11.3.1.1
.1.3.6.1.2.1.1.9.1.2.2 = OID: .1.3.6.1.6.3.15.2.1.1
.1.3.6.1.2.1.1.9.1.2.3 = OID: .1.3.6.1.6.3.10.3.1.1
.1.3.6.1.2.1.1.9.1.2.4 = OID: .1.3.6.1.6.3.1
.1.3.6.1.2.1.1.9.1.2.5 = OID: .1.3.6.1.6.3.16.2.2.1
.1.3.6.1.2.1.1.9.1.2.6 = OID: .1.3.6.1.2.1.49
.1.3.6.1.2.1.1.9.1.2.7 = OID: .1.3.6.1.2.1.4
.1.3.6.1.2.1.1.9.1.2.8 = OID: .1.3.6.1.2.1.50
.1.3.6.1.2.1.1.9.1.2.9 = OID: .1.3.6.1.6.3.13.3.1.3
.1.3.6.1.2.1.1.9.1.2.10 = OID: .1.3.6.1.2.1.92
.1.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB for Message Processing and Dispatching."
.1.3.6.1.2.1.1.9.1.3.2 = STRING: "The management information definitions for the SNMP User-based Security Model."

[..]

.1.3.6.1.2.1.11.10.0 = Counter32: 0
.1.3.6.1.2.1.11.11.0 = Counter32: 0
.1.3.6.1.2.1.11.12.0 = Counter32: 0
.1.3.6.1.2.1.11.13.0 = Counter32: 1118
.1.3.6.1.2.1.11.14.0 = Counter32: 0
.1.3.6.1.2.1.11.15.0 = Counter32: 0
.1.3.6.1.2.1.11.16.0 = Counter32: 1122
.1.3.6.1.2.1.11.17.0 = Counter32: 0
.1.3.6.1.2.1.11.18.0 = Counter32: 0
.1.3.6.1.2.1.11.19.0 = Counter32: 0
.1.3.6.1.2.1.11.20.0 = Counter32: 0
.1.3.6.1.2.1.11.21.0 = Counter32: 0
.1.3.6.1.2.1.11.22.0 = Counter32: 0
.1.3.6.1.2.1.11.24.0 = Counter32: 0
.1.3.6.1.2.1.11.25.0 = Counter32: 0
.1.3.6.1.2.1.11.26.0 = Counter32: 0
.1.3.6.1.2.1.11.27.0 = Counter32: 0
.1.3.6.1.2.1.11.28.0 = Counter32: 1132
.1.3.6.1.2.1.11.29.0 = Counter32: 1
.1.3.6.1.2.1.11.30.0 = INTEGER: 2
.1.3.6.1.2.1.11.31.0 = Counter32: 0
.1.3.6.1.2.1.11.32.0 = Counter32: 0
.1.3.6.1.2.1.25.1.1.0 = Timeticks: (278435) 0:46:24.35
.1.3.6.1.2.1.25.1.2.0 = Hex-STRING: 07 E4 02 02 16 14 30 00 2B 01 00
.1.3.6.1.2.1.25.1.3.0 = INTEGER: 393216
.1.3.6.1.2.1.25.1.4.0 = STRING: "coherent_pool=1M 8250.nr_uarts=0 bcm2708_fb.fbwidth=16 bcm2708_fb.fbheight=16 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.m"
.1.3.6.1.2.1.25.1.5.0 = Gauge32: 1
.1.3.6.1.2.1.25.1.6.0 = Gauge32: 121
.1.3.6.1.2.1.25.1.7.0 = INTEGER: 0
.1.3.6.1.2.1.25.2.2.0 = INTEGER: 999036
.1.3.6.1.2.1.25.2.3.1.1.1 = INTEGER: 1
.1.3.6.1.2.1.25.2.3.1.1.3 = INTEGER: 3
.1.3.6.1.2.1.25.2.3.1.1.6 = INTEGER: 6
.1.3.6.1.2.1.25.2.3.1.1.7 = INTEGER: 7
.1.3.6.1.2.1.25.2.3.1.1.8 = INTEGER: 8
.1.3.6.1.2.1.25.2.3.1.1.10 = INTEGER: 10
Timeout: No Response from localhost
root@DietPi:~#

This is instead the result with the second command:

root@DietPi:~# journalctl -u snmpd.service
-- Logs begin at Thu 2016-11-03 18:16:42 CET, end at Sun 2020-02-02 22:22:18 CET. --
Feb 02 21:34:03 DietPi systemd[1]: Starting Simple Network Management Protocol (SNMP) Daemon....
Feb 02 21:34:03 DietPi systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon..
Feb 02 21:34:04 DietPi snmpd[591]: pcilib: Cannot open /proc/bus/pci
Feb 02 21:34:04 DietPi snmpd[591]: pcilib: Cannot find any working access method.
Feb 02 21:34:04 DietPi snmpd[591]: pcilib: pci_init failed
Feb 02 21:36:01 DietPi snmpd[591]: Cannot statfs /mnt/batocera: No such device
Feb 02 21:36:01 DietPi snmpd[591]: /etc/snmp/snmpd.conf: line 145: Warning: Unknown token: defaultMonitors.
Feb 02 21:36:01 DietPi snmpd[591]: /etc/snmp/snmpd.conf: line 147: Warning: Unknown token: linkUpDownNotifications.
Feb 02 21:36:01 DietPi snmpd[591]: Turning on AgentX master support.
Feb 02 21:36:01 DietPi snmpd[591]: error finding row index in _ifXTable_container_row_restore
Feb 02 21:36:01 DietPi snmpd[591]: error finding row index in _ifXTable_container_row_restore
Feb 02 21:36:01 DietPi snmpd[591]: NET-SNMP version 5.7.3
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161

Joulinar · 2 February 2020 21:49

ok so snmp is working as it’s displaying all the information it has (1st command). it might be to many entries to display all of them.

The 2nd command is displaying all the connections you are doing.

Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161

An idea would be to increase the timeout value

Garret · 2 February 2020 21:55

Joulinar so in practice what could I do? Sorry but I am pretty noob with snmpd and honestly I don’t really like it as it seems very obscure to me. I am only using because it has been the only (easy?) way to check how much space I have left on my device and interface that information in Home assistant sitting in another raspberry pi.

Joulinar · 2 February 2020 22:07

pls can you try this command if it still going to timeout

snmpwalk -On -r 1 -t 20 -v 2c -c public localhost

Garret · 2 February 2020 22:09

Joulinar Yes, I still get the timeout error after waiting a while:

.1.3.6.1.2.1.11.26.0 = Counter32: 0
.1.3.6.1.2.1.11.27.0 = Counter32: 0
.1.3.6.1.2.1.11.28.0 = Counter32: 2289
.1.3.6.1.2.1.11.29.0 = Counter32: 1
.1.3.6.1.2.1.11.30.0 = INTEGER: 2
.1.3.6.1.2.1.11.31.0 = Counter32: 0
.1.3.6.1.2.1.11.32.0 = Counter32: 0
.1.3.6.1.2.1.25.1.1.0 = Timeticks: (560321) 1:33:23.21
.1.3.6.1.2.1.25.1.2.0 = Hex-STRING: 07 E4 02 02 17 07 2F 00 2B 01 00
.1.3.6.1.2.1.25.1.3.0 = INTEGER: 393216
.1.3.6.1.2.1.25.1.4.0 = STRING: "coherent_pool=1M 8250.nr_uarts=0 bcm2708_fb.fbwidth=16 bcm2708_fb.fbheight=16 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.m"
.1.3.6.1.2.1.25.1.5.0 = Gauge32: 1
.1.3.6.1.2.1.25.1.6.0 = Gauge32: 123
.1.3.6.1.2.1.25.1.7.0 = INTEGER: 0
.1.3.6.1.2.1.25.2.2.0 = INTEGER: 999036
.1.3.6.1.2.1.25.2.3.1.1.1 = INTEGER: 1
.1.3.6.1.2.1.25.2.3.1.1.3 = INTEGER: 3
.1.3.6.1.2.1.25.2.3.1.1.6 = INTEGER: 6
.1.3.6.1.2.1.25.2.3.1.1.7 = INTEGER: 7
.1.3.6.1.2.1.25.2.3.1.1.8 = INTEGER: 8
.1.3.6.1.2.1.25.2.3.1.1.10 = INTEGER: 10
Timeout: No Response from localhost
root@DietPi:~#

Joulinar · 2 February 2020 22:29

ok it stop at the same line. It seems snmp is not able to collect some data and that’s where it timeout. for me it still looks like, you would need to fix the 2 error messages we have seen above.

Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /mnt/batocera: No such device
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /run/docker/netns/ad525e0f7c50: Permission denied

Garret · 2 February 2020 22:37

Joulinar However, by quickly looking on Google, it seems it is normal to see that error and you cannot really do much about it

https://wiki.opennms.org/wiki/SNMP_spams_my_log (end of the page)

Joulinar · 2 February 2020 22:42

ok thats about the Permission denied
what about /mnt/batocera: No such device

Garret · 2 February 2020 23:04

Joulinar I think I solved the problem. With the function of dietpi drive manager I tried to see if I could delete this /mnt/batocera as I never created/used it. As soon as I opened the drive manager the partition disappeared. I guess for some reason the system was thinking that partition was existing and freezing when not finding it but now it looks like the registry was automatically “fixed” after accessing the drive manager. Still don’t know how all this mess created but I am happy got solved.

Joulinar · 2 February 2020 23:09

ok so finally we fixed it by removing the stale mount point

Garret · 2 February 2020 23:10

Joulinar just wanted to say big thank you for the continuous and fast support

Joulinar · 2 February 2020 23:13

yw. right in time for Super Bowl