Page 1 of 2

[Solved] SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 6:25 pm
by Garret
I have two raspberry pis. One has Dietpi installed while the other has Hassio.

I am using SNMP installed on Dietpi to get the OID of a free space left on a specific disc on Dietpi. In this way, I could check on my Hassio installation how much disc space is left on Dietpi through the SNMP integration.

Since about a week I can see that I cannot retrieve anymore that information but I always get a Timeout: No Response from localhost

For instance:

Code: Select all

root@DietPi:~# df | grep mnt
/dev/sda1      121169024  37773968  83378672  32% /mnt/data
/dev/mmcblk0p1     64366     53024     11342  83% /mnt/0095-112D
/dev/mmcblk0p5     30701      3249     25159  12% /mnt/a54b8bb4-b9ae-474d-9a87-3b0b590dc77f
/dev/sdb1      960379920 364306840 596056696  38% /mnt/media
overlay         15282608   2961432  11521816  21% /mnt/dietpi_userdata/docker-data/overlay2/d2521957b38eab1bb69b886a79e664e0979d89efb053d7c7a14a6c05c440beeb/merged
root@DietPi:~# snmpwalk -On  -v 2c -c public localhost .1.3.6.1.4.1.2021.9  | grep  /dev/sdb
Timeout: No Response from localhost
root@DietPi:~#
This is the snmpd status:

Code: Select all

root@DietPi:~# service snmpd status
● snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
   Loaded: loaded (/lib/systemd/system/snmpd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-02-01 13:27:48 CET; 1 day 4h ago
  Process: 566 ExecStartPre=/bin/mkdir -p /var/run/agentx (code=exited, status=0/SUCCESS)
 Main PID: 576 (snmpd)
    Tasks: 1 (limit: 4915)
   Memory: 3.8M
      CPU: 1min 29.989s
   CGroup: /system.slice/snmpd.service
           └─576 /usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f

Feb 02 18:14:50 DietPi snmpd[576]: Connection from UDP: [192.168.40.100]:55149->[192.168.40.200]:161
Feb 02 18:14:50 DietPi snmpd[576]: Connection from UDP: [192.168.40.100]:55149->[192.168.40.200]:161
Feb 02 18:23:08 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /mnt/batocera: No such device
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /run/docker/netns/ad525e0f7c50: Permission denied
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
Feb 02 18:24:38 DietPi snmpd[576]: Connection from UDP: [127.0.0.1]:59983->[127.0.0.1]:161
root@DietPi:~#
Would you know what I am doing wrong?

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 9:03 pm
by Joulinar
Hi,

not sure if this is related but you might need to check these 2 error messages

Code: Select all

Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /mnt/batocera: No such device
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /run/docker/netns/ad525e0f7c50: Permission denied

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 9:29 pm
by Garret
Hi @Joulinar, I had also noticed those errors but to me they don't look important as they are related to partitions I am not interested (I am only interested in /mnt/sdb1).

I cannot understand why it has been working for me but it is just since a week or two that suddenly I get that error now.

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 10:06 pm
by Joulinar
well you receive a Timeout on your request. So it could be, that the system is not able to complete the request because of the 2 missing file systems bot being available.

What happen if you execute just a minimized snmpwalk command

Code: Select all

snmpwalk -On  -v 2c -c public localhost
maybe there are some more information within journalctl

Code: Select all

journalctl -u snmpd.service

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 10:24 pm
by Garret
Hi @Joulinar, by giving the first command you mentioned I get a very long list of OIDs but in the end I can see there is still "Timeout: No Response from localhost". I have cut the code because it was very long otherwise.

Code: Select all

root@DietPi:~# snmpwalk -On  -v 2c -c public localhost
.1.3.6.1.2.1.1.1.0 = STRING: "Linux DietPi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l"
.1.3.6.1.2.1.1.2.0 = OID: .1.3.6.1.4.1.8072.3.2.10
.1.3.6.1.2.1.1.3.0 = Timeticks: (277734) 0:46:17.34
.1.3.6.1.2.1.1.4.0 = STRING: "Me <me@example.org>"
.1.3.6.1.2.1.1.5.0 = STRING: "DietPi"
.1.3.6.1.2.1.1.6.0 = STRING: "Sitting on the Dock of the Bay"
.1.3.6.1.2.1.1.7.0 = INTEGER: 72
.1.3.6.1.2.1.1.8.0 = Timeticks: (28) 0:00:00.28
.1.3.6.1.2.1.1.9.1.2.1 = OID: .1.3.6.1.6.3.11.3.1.1
.1.3.6.1.2.1.1.9.1.2.2 = OID: .1.3.6.1.6.3.15.2.1.1
.1.3.6.1.2.1.1.9.1.2.3 = OID: .1.3.6.1.6.3.10.3.1.1
.1.3.6.1.2.1.1.9.1.2.4 = OID: .1.3.6.1.6.3.1
.1.3.6.1.2.1.1.9.1.2.5 = OID: .1.3.6.1.6.3.16.2.2.1
.1.3.6.1.2.1.1.9.1.2.6 = OID: .1.3.6.1.2.1.49
.1.3.6.1.2.1.1.9.1.2.7 = OID: .1.3.6.1.2.1.4
.1.3.6.1.2.1.1.9.1.2.8 = OID: .1.3.6.1.2.1.50
.1.3.6.1.2.1.1.9.1.2.9 = OID: .1.3.6.1.6.3.13.3.1.3
.1.3.6.1.2.1.1.9.1.2.10 = OID: .1.3.6.1.2.1.92
.1.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB for Message Processing and Dispatching."
.1.3.6.1.2.1.1.9.1.3.2 = STRING: "The management information definitions for the SNMP User-based Security Model."

[..]

.1.3.6.1.2.1.11.10.0 = Counter32: 0
.1.3.6.1.2.1.11.11.0 = Counter32: 0
.1.3.6.1.2.1.11.12.0 = Counter32: 0
.1.3.6.1.2.1.11.13.0 = Counter32: 1118
.1.3.6.1.2.1.11.14.0 = Counter32: 0
.1.3.6.1.2.1.11.15.0 = Counter32: 0
.1.3.6.1.2.1.11.16.0 = Counter32: 1122
.1.3.6.1.2.1.11.17.0 = Counter32: 0
.1.3.6.1.2.1.11.18.0 = Counter32: 0
.1.3.6.1.2.1.11.19.0 = Counter32: 0
.1.3.6.1.2.1.11.20.0 = Counter32: 0
.1.3.6.1.2.1.11.21.0 = Counter32: 0
.1.3.6.1.2.1.11.22.0 = Counter32: 0
.1.3.6.1.2.1.11.24.0 = Counter32: 0
.1.3.6.1.2.1.11.25.0 = Counter32: 0
.1.3.6.1.2.1.11.26.0 = Counter32: 0
.1.3.6.1.2.1.11.27.0 = Counter32: 0
.1.3.6.1.2.1.11.28.0 = Counter32: 1132
.1.3.6.1.2.1.11.29.0 = Counter32: 1
.1.3.6.1.2.1.11.30.0 = INTEGER: 2
.1.3.6.1.2.1.11.31.0 = Counter32: 0
.1.3.6.1.2.1.11.32.0 = Counter32: 0
.1.3.6.1.2.1.25.1.1.0 = Timeticks: (278435) 0:46:24.35
.1.3.6.1.2.1.25.1.2.0 = Hex-STRING: 07 E4 02 02 16 14 30 00 2B 01 00
.1.3.6.1.2.1.25.1.3.0 = INTEGER: 393216
.1.3.6.1.2.1.25.1.4.0 = STRING: "coherent_pool=1M 8250.nr_uarts=0 bcm2708_fb.fbwidth=16 bcm2708_fb.fbheight=16 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.m"
.1.3.6.1.2.1.25.1.5.0 = Gauge32: 1
.1.3.6.1.2.1.25.1.6.0 = Gauge32: 121
.1.3.6.1.2.1.25.1.7.0 = INTEGER: 0
.1.3.6.1.2.1.25.2.2.0 = INTEGER: 999036
.1.3.6.1.2.1.25.2.3.1.1.1 = INTEGER: 1
.1.3.6.1.2.1.25.2.3.1.1.3 = INTEGER: 3
.1.3.6.1.2.1.25.2.3.1.1.6 = INTEGER: 6
.1.3.6.1.2.1.25.2.3.1.1.7 = INTEGER: 7
.1.3.6.1.2.1.25.2.3.1.1.8 = INTEGER: 8
.1.3.6.1.2.1.25.2.3.1.1.10 = INTEGER: 10
Timeout: No Response from localhost
root@DietPi:~#
This is instead the result with the second command:

Code: Select all

root@DietPi:~# journalctl -u snmpd.service
-- Logs begin at Thu 2016-11-03 18:16:42 CET, end at Sun 2020-02-02 22:22:18 CET. --
Feb 02 21:34:03 DietPi systemd[1]: Starting Simple Network Management Protocol (SNMP) Daemon....
Feb 02 21:34:03 DietPi systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon..
Feb 02 21:34:04 DietPi snmpd[591]: pcilib: Cannot open /proc/bus/pci
Feb 02 21:34:04 DietPi snmpd[591]: pcilib: Cannot find any working access method.
Feb 02 21:34:04 DietPi snmpd[591]: pcilib: pci_init failed
Feb 02 21:36:01 DietPi snmpd[591]: Cannot statfs /mnt/batocera: No such device
Feb 02 21:36:01 DietPi snmpd[591]: /etc/snmp/snmpd.conf: line 145: Warning: Unknown token: defaultMonitors.
Feb 02 21:36:01 DietPi snmpd[591]: /etc/snmp/snmpd.conf: line 147: Warning: Unknown token: linkUpDownNotifications.
Feb 02 21:36:01 DietPi snmpd[591]: Turning on AgentX master support.
Feb 02 21:36:01 DietPi snmpd[591]: error finding row index in _ifXTable_container_row_restore
Feb 02 21:36:01 DietPi snmpd[591]: error finding row index in _ifXTable_container_row_restore
Feb 02 21:36:01 DietPi snmpd[591]: NET-SNMP version 5.7.3
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 10:49 pm
by Joulinar
ok so snmp is working as it's displaying all the information it has (1st command). it might be to many entries to display all of them.

The 2nd command is displaying all the connections you are doing.

Code: Select all

Feb 02 22:20:47 DietPi snmpd[591]: Connection from UDP: [127.0.0.1]:56546->[127.0.0.1]:161
An idea would be to increase the timeout value

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 10:55 pm
by Garret
@Joulinar so in practice what could I do? Sorry but I am pretty noob with snmpd and honestly I don't really like it as it seems very obscure to me. I am only using because it has been the only (easy?) way to check how much space I have left on my device and interface that information in Home assistant sitting in another raspberry pi.

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 11:07 pm
by Joulinar
pls can you try this command if it still going to timeout

Code: Select all

snmpwalk -On -r 1 -t 20 -v 2c -c public localhost

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 11:09 pm
by Garret
@Joulinar Yes, I still get the timeout error after waiting a while:

Code: Select all

.1.3.6.1.2.1.11.26.0 = Counter32: 0
.1.3.6.1.2.1.11.27.0 = Counter32: 0
.1.3.6.1.2.1.11.28.0 = Counter32: 2289
.1.3.6.1.2.1.11.29.0 = Counter32: 1
.1.3.6.1.2.1.11.30.0 = INTEGER: 2
.1.3.6.1.2.1.11.31.0 = Counter32: 0
.1.3.6.1.2.1.11.32.0 = Counter32: 0
.1.3.6.1.2.1.25.1.1.0 = Timeticks: (560321) 1:33:23.21
.1.3.6.1.2.1.25.1.2.0 = Hex-STRING: 07 E4 02 02 17 07 2F 00 2B 01 00
.1.3.6.1.2.1.25.1.3.0 = INTEGER: 393216
.1.3.6.1.2.1.25.1.4.0 = STRING: "coherent_pool=1M 8250.nr_uarts=0 bcm2708_fb.fbwidth=16 bcm2708_fb.fbheight=16 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.m"
.1.3.6.1.2.1.25.1.5.0 = Gauge32: 1
.1.3.6.1.2.1.25.1.6.0 = Gauge32: 123
.1.3.6.1.2.1.25.1.7.0 = INTEGER: 0
.1.3.6.1.2.1.25.2.2.0 = INTEGER: 999036
.1.3.6.1.2.1.25.2.3.1.1.1 = INTEGER: 1
.1.3.6.1.2.1.25.2.3.1.1.3 = INTEGER: 3
.1.3.6.1.2.1.25.2.3.1.1.6 = INTEGER: 6
.1.3.6.1.2.1.25.2.3.1.1.7 = INTEGER: 7
.1.3.6.1.2.1.25.2.3.1.1.8 = INTEGER: 8
.1.3.6.1.2.1.25.2.3.1.1.10 = INTEGER: 10
Timeout: No Response from localhost
root@DietPi:~#

Re: SNMP suddenly stopped working to get free space left on a disc

Posted: Sun Feb 02, 2020 11:29 pm
by Joulinar
ok it stop at the same line. It seems snmp is not able to collect some data and that's where it timeout. for me it still looks like, you would need to fix the 2 error messages we have seen above.

Code: Select all

Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /mnt/batocera: No such device
Feb 02 18:24:38 DietPi snmpd[576]: Cannot statfs /run/docker/netns/ad525e0f7c50: Permission denied