I cant get promethus to scrape Navidrome info

abdulmalekA · October 26, 2023, 6:39am

Creating a bug report/issue

Required Information

DietPi version |

G_DIETPI_VERSION_CORE=8
G_DIETPI_VERSION_SUB=23
G_DIETPI_VERSION_RC=3
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
G_LIVE_PATCH_STATUS[0]='applied'

Distro version | bookworm 0
Kernel version | Linux DietPi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
Architecture | arm64
SBC model | `RPi4
Power supply used : 5V 1A Powersupply
SD card used | SanDisk ultra

Additional Information (if applicable)

Software title | Promethus , Grafan , node exporter
Was the software title installed freshly or updated/migrated? freshly
Can this issue be replicated on a fresh installation of DietPi? yes
← If you sent a “dietpi-bugreport”, please paste the ID here →
Bug report ID | echo $G_HW_UUID

Steps to reproduce

1.Installed Promethus server following official doc
2.Installed Grafana
3.Installed Node exporter

Expected behaviour

I should be able to see navidrome metric in Grafana dashboard

Actual behaviour

Unable to view navidrome data on Grafana dashboard

Deatils here :
wkL7fgn_d

Promethus.yml file

Df2NhRk_d

Navidrome.toml

83TmWbt_d

Promethus server status

vFkVFLI_d

Node exporter status
B3vp8Wd_d

Grafana server status

Jappe · October 26, 2023, 7:39am

Probably you need to change the lines /metrics_SOME_SECRET_KEY to something meaningful.
Is prometheus complaining about this path?
journalctl -u prometheus

abdulmalekA · October 26, 2023, 3:41pm

I tried using /metrics , and the same thing happen cant see any information in the dashboard

I cannot see navidrome as target in promethus . check here please :

promethustargets1915×478 24 KB

Jappe · October 26, 2023, 4:42pm

The navidrome docs have also no hint:
https://www.navidrome.org/docs/usage/monitoring/
What is this secret key? (I never used grafana/prometheus)

Jappe · October 26, 2023, 6:03pm

I got some more info from Discord:

the secret is key is something you would generate as you are exposing server metrics which you would want to keep private.

and it should be Prometheus.Enabled = true in toml file, to enable it.

its just a url path where the data is being sent
you should have some random string like metrics_sjhdf8h8923LF898 as your path to keep it secure, but leaving it as metrics_some_secret_key also works , just make sure you update it in both places

(see also https://www.navidrome.org/docs/usage/configuration-options/#available-options)

So this is what you already did. The only difference I can see now is the line Prometheus.Enabled = true without the ' ', which makes sense since it’s a boolean and not a string. You can also see in the link above which values are strings and which not. so probably a modification to Prometheus.Enabled = true will do the trick now and enable the metrics feature.
Probably this misconfig will also show up in the logs, which we could’ve checked in the first place
`journalctl -u navidrome.service’

btw you have set another 'true' in your config for EnableLogRedacting

abdulmalekA · October 27, 2023, 2:44am

Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000253 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000253
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000254 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000254
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50099” requestId=DietPi/ISRSpwLxo3-000255 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000255
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50098” requestId=DietPi/ISRSpwLxo3-000257 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000258 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000257
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000258
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000259 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000259
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50099” requestId=DietPi/ISRSpwLxo3-000260 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000260
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50096” requestId=DietPi/ISRSpwLxo3-000256 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000261 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000256
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000261
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000262 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000262
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50099” requestId=DietPi/ISRSpwLxo3-000264 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000264
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50098” requestId=DietPi/ISRSpwLxo3-000263 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000263
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50096” requestId=DietPi/ISRSpwLxo3-000267 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000267
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50098” requestId=DietPi/ISRSpwLxo3-000268 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000268
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000265 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000265
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50096” requestId=DietPi/ISRSpwLxo3-000269 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000269
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000270 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000266 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50099” requestId=DietPi/ISRSpwLxo3-000272 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000266
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000272
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000270
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50098” requestId=DietPi/ISRSpwLxo3-000271 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000271
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50096” requestId=DietPi/ISRSpwLxo3-000273 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000273
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000274 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000274
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50096” requestId=DietPi/ISRSpwLxo3-000278 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000275 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000278
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000279 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000275
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50098” requestId=DietPi/ISRSpwLxo3-000277 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000279
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50099” requestId=DietPi/ISRSpwLxo3-000276 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000276
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50096” requestId=DietPi/ISRSpwLxo3-000280 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000281 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000277
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000280
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000281
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000286 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000286
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50099” requestId=DietPi/ISRSpwLxo3-000282 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50098” requestId=DietPi/ISRSpwLxo3-000283 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000283
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50096” requestId=DietPi/ISRSpwLxo3-000284 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000284
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000282
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50097” requestId=DietPi/ISRSpwLxo3-000287 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50098” requestId=DietPi/ISRSpwLxo3-000288 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000287
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000288
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Invalid login” error=“invalid authentication” remoteAddr=“192. 168.100.4:50095” requestId=DietPi/ISRSpwLxo3-000285 username=null
Oct 27 02:44:19 DietPi navidrome[99649]: time=“2023-10-27T02:44:19+01:00” level= warning msg=“API: Failed response” endpoint=/rest/getCoverArt error=40 message=" Wrong username or password" requestId=DietPi/ISRSpwLxo3-000285

abdulmalekA · October 27, 2023, 2:48am


Oct 26 16:00:03 DietPi prometheus[87172]: ts=2023-10-26T15:00:03.313Z caller=com                                                                                                                                                             pact.go:519 level=info component=tsdb msg="write block" mint=1698321602765 maxt=                                                                                                                                                             1698328800000 ulid=01HDP7ZS3PP9R8CTWX1RSYKB3H duration=507.174703ms
Oct 26 16:00:03 DietPi prometheus[87172]: ts=2023-10-26T15:00:03.328Z caller=hea                                                                                                                                                             d.go:1219 level=info component=tsdb msg="Head GC completed" caller=truncateMemor                                                                                                                                                             y duration=11.759537ms
Oct 26 18:00:03 DietPi prometheus[87172]: ts=2023-10-26T17:00:03.374Z caller=com                                                                                                                                                             pact.go:519 level=info component=tsdb msg="write block" mint=1698328802765 maxt=                                                                                                                                                             1698336000000 ulid=01HDPEVGBSQK2XACNCERH3GAZF duration=565.149222ms
Oct 26 18:00:03 DietPi prometheus[87172]: ts=2023-10-26T17:00:03.383Z caller=hea                                                                                                                                                             d.go:1219 level=info component=tsdb msg="Head GC completed" caller=truncateMemor                                                                                                                                                             y duration=5.682185ms
Oct 26 18:00:03 DietPi prometheus[87172]: ts=2023-10-26T17:00:03.388Z caller=che                                                                                                                                                             ckpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=                                                                                                                                                             4 to_segment=5 mint=1698336000000
Oct 26 18:00:03 DietPi prometheus[87172]: ts=2023-10-26T17:00:03.785Z caller=hea                                                                                                                                                             d.go:1191 level=info component=tsdb msg="WAL checkpoint complete" first=4 last=5                                                                                                                                                              duration=397.673592ms
Oct 26 18:00:04 DietPi prometheus[87172]: ts=2023-10-26T17:00:04.784Z caller=com                                                                                                                                                             pact.go:460 level=info component=tsdb msg="compact blocks" count=3 mint=16983000                                                                                                                                                             02766 maxt=1698321600000 ulid=01HDPEVHAAFDWA2ZRDMN1BN3W1 sources="[01HDNKCR7N96N                                                                                                                                                             93JA4JWQ44523 01HDNT8FFQK6JVNBB80MCV07SG 01HDP141VNBN8MWA0MBQWZKNWQ]" duration=9                                                                                                                                                             98.898315ms
Oct 26 18:00:04 DietPi prometheus[87172]: ts=2023-10-26T17:00:04.804Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDP141VN                                                                                                                                                             BN8MWA0MBQWZKNWQ
Oct 26 18:00:04 DietPi prometheus[87172]: ts=2023-10-26T17:00:04.822Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDNKCR7N                                                                                                                                                             96N93JA4JWQ44523
Oct 26 18:00:04 DietPi prometheus[87172]: ts=2023-10-26T17:00:04.844Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDNT8FFQ                                                                                                                                                             K6JVNBB80MCV07SG
Oct 26 20:00:08 DietPi prometheus[87172]: ts=2023-10-26T19:00:08.316Z caller=com                                                                                                                                                             pact.go:519 level=info component=tsdb msg="write block" mint=1698336002765 maxt=                                                                                                                                                             1698343200000 ulid=01HDPNQCFNP78JXXKJ728Q6TYK duration=519.174092ms
Oct 26 20:00:08 DietPi prometheus[87172]: ts=2023-10-26T19:00:08.332Z caller=hea                                                                                                                                                             d.go:1219 level=info component=tsdb msg="Head GC completed" caller=truncateMemor                                                                                                                                                             y duration=12.144019ms
Oct 26 22:00:08 DietPi prometheus[87172]: ts=2023-10-26T21:00:08.324Z caller=com                                                                                                                                                             pact.go:519 level=info component=tsdb msg="write block" mint=1698343202765 maxt=                                                                                                                                                             1698350400000 ulid=01HDPWK3QRD5RNMVE8XCG6HYB8 duration=523.573907ms
Oct 26 22:00:08 DietPi prometheus[87172]: ts=2023-10-26T21:00:08.339Z caller=hea                                                                                                                                                             d.go:1219 level=info component=tsdb msg="Head GC completed" caller=truncateMemor                                                                                                                                                             y duration=11.716037ms
Oct 26 22:00:08 DietPi prometheus[87172]: ts=2023-10-26T21:00:08.350Z caller=che                                                                                                                                                             ckpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=                                                                                                                                                             6 to_segment=7 mint=1698350400000
Oct 26 22:00:08 DietPi prometheus[87172]: ts=2023-10-26T21:00:08.684Z caller=hea                                                                                                                                                             d.go:1191 level=info component=tsdb msg="WAL checkpoint complete" first=6 last=7                                                                                                                                                              duration=334.760241ms
Oct 27 00:00:08 DietPi prometheus[87172]: ts=2023-10-26T23:00:08.321Z caller=com                                                                                                                                                             pact.go:519 level=info component=tsdb msg="write block" mint=1698350402765 maxt=                                                                                                                                                             1698357600000 ulid=01HDQ3ETZPXQ9SFM8M6HE2W3DY duration=523.31711ms
Oct 27 00:00:08 DietPi prometheus[87172]: ts=2023-10-26T23:00:08.337Z caller=hea                                                                                                                                                             d.go:1219 level=info component=tsdb msg="Head GC completed" caller=truncateMemor                                                                                                                                                             y duration=11.787907ms
Oct 27 00:00:09 DietPi prometheus[87172]: ts=2023-10-26T23:00:09.155Z caller=com                                                                                                                                                             pact.go:460 level=info component=tsdb msg="compact blocks" count=3 mint=16983216                                                                                                                                                             02765 maxt=1698343200000 ulid=01HDQ3EVGYVB8T435ETM252K90 sources="[01HDP7ZS3PP9R                                                                                                                                                             8CTWX1RSYKB3H 01HDPEVGBSQK2XACNCERH3GAZF 01HDPNQCFNP78JXXKJ728Q6TYK]" duration=8                                                                                                                                                             05.974518ms
Oct 27 00:00:09 DietPi prometheus[87172]: ts=2023-10-26T23:00:09.169Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDP7ZS3P                                                                                                                                                             P9R8CTWX1RSYKB3H
Oct 27 00:00:09 DietPi prometheus[87172]: ts=2023-10-26T23:00:09.190Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDPEVGBS                                                                                                                                                             QK2XACNCERH3GAZF
Oct 27 00:00:09 DietPi prometheus[87172]: ts=2023-10-26T23:00:09.213Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDPNQCFN                                                                                                                                                             P78JXXKJ728Q6TYK
Oct 27 00:00:11 DietPi prometheus[87172]: ts=2023-10-26T23:00:11.109Z caller=com                                                                                                                                                             pact.go:460 level=info component=tsdb msg="compact blocks" count=3 mint=16982850                                                                                                                                                             47762 maxt=1698343200000 ulid=01HDQ3EWBZTF85QF8TR880ZX5E sources="[01HDNT8G2Q2FW                                                                                                                                                             N8DD75AMKHJR2 01HDPEVHAAFDWA2ZRDMN1BN3W1 01HDQ3EVGYVB8T435ETM252K90]" duration=1                                                                                                                                                             .894860961s
Oct 27 00:00:11 DietPi prometheus[87172]: ts=2023-10-26T23:00:11.129Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDNT8G2Q                                                                                                                                                             2FWN8DD75AMKHJR2
Oct 27 00:00:11 DietPi prometheus[87172]: ts=2023-10-26T23:00:11.167Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDPEVHAA                                                                                                                                                             FDWA2ZRDMN1BN3W1
Oct 27 00:00:11 DietPi prometheus[87172]: ts=2023-10-26T23:00:11.215Z caller=db.                                                                                                                                                             go:1541 level=info component=tsdb msg="Deleting obsolete block" block=01HDQ3EVGY                                                                                                                                                             VB8T435ETM252K90
Oct 27 02:00:06 DietPi prometheus[87172]: ts=2023-10-27T01:00:06.767Z caller=com                                                                                                                                                             pact.go:519 level=info component=tsdb msg="write block" mint=1698357602762 maxt=                                                                                                                                                             1698364800000 ulid=01HDQAAGPZJNBDDDY7G6PP6ZBA duration=527.926963ms
Oct 27 02:00:06 DietPi prometheus[87172]: ts=2023-10-27T01:00:06.782Z caller=hea                                                                                                                                                             d.go:1219 level=info component=tsdb msg="Head GC completed" caller=truncateMemor                                                                                                                                                             y duration=11.923592ms
Oct 27 02:00:06 DietPi prometheus[87172]: ts=2023-10-27T01:00:06.794Z caller=che                                                                                                                                                             ckpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=                                                                                                                                                             8 to_segment=9 mint=1698364800000
Oct 27 02:00:07 DietPi prometheus[87172]: ts=2023-10-27T01:00:07.188Z caller=hea                                                                                                                                                             d.go:1191 level=info component=tsdb msg="WAL checkpoint complete" first=8 last=9                                                                                                                                                              duration=394.717666ms

abdulmalekA · October 27, 2023, 2:56am

I have edited the config in both promethus and navidrome and still not show any mertics in grafana dashbaord

navidrome.toml

DataFolder = '/mnt/dietpi_userdata/navidrome/data'
MusicFolder = '/mnt/900C-3652/Streamripdownloads'
ImageCacheSize = '100MiB'

LogLevel = 'warn'
SessionTimeout = '24h'
AuthRequestLimit = '5'
AuthWindowLength = '20s'
ScanSchedule = 0
EnableLogRedacting = true
Scanner.Extractor = 'taglib'
CoverJpegQuality = '100'
CoverArtPriority = 'cover.*, folder.*, front.*, embedded, external'
Prometheus.Enabled = true
Prometheus.MetricsPath = /metrics # I tried using with ' ' and without ' '

promethus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    monitor: test

scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets:
    - XXX.XXX.XXX.X:9090
- job_name: node
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets:
    - XXX.XXX.XXX.X:9100
- job_name: navidrome
 honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets:
    - XXX.XXX.XXX.X:4533

Jappe · October 27, 2023, 5:33am

I have no other idea then. Did you restart navidrome after the changes?
edit: I will test this for myself this afternoon.

Joulinar · October 27, 2023, 6:15am

This seems to be important.

Jappe · October 27, 2023, 6:21am

But there is no login, you could use a “secret” metrics path to make it more secure.

Jappe · October 27, 2023, 3:52pm

edit:

Now I see what is wrong, in your prometheus config you have two times scrape_configs. I assume it can only be one block. My conf looks like this:

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped>
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: navidrome
    scheme: http
    static_configs:
      - targets: ["192.168.178.56:4533"]

Original post:

I got it working on my system now

Here is what I did:

In navidrome config /mnt/dietpi_userdata/navidrome/navidrome.toml I added

Prometheus.Enabled = true

and restarted navidrome systemctl restart navidrome.service

In prometheus config prometheus.yml I added:

 - job_name: navidrome
    scheme: http
    static_configs:
      - targets: ["192.168.178.56:4533"]

Then I just started prometheus (I used precompiled binary).
I did not set a metrics path at all, since my navidrome instance is not public.

I can also see values at http://192.168.178.56:4533/metrics
Maybe you can check if this is also the case for your navidrome instance, to make sure the endpoint is enabled.

abdulmalekA · October 27, 2023, 4:59pm

Do I need to add this to my promethus.yml too ?

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    monitor: test

Jappe · October 27, 2023, 5:15pm

Here is my full prometheus.yml:

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093


# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ["localhost:9090"]


  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]


  - job_name: navidrome
    scheme: http
    static_configs:
      - targets: ["192.168.178.56:4533"]

(the node is my rpi3 itself and the prometheus job was already in the file)

abdulmalekA · October 28, 2023, 3:01am

it does show metrics now in the dashboard , thank you so much sir you are the boss , my next one will be monitoring pihole , I might need your help if i ran into issues lol .

Jappe · October 28, 2023, 12:49pm

I checked this out right now.
I downloaded an exporter for pihole onto my Pihole device:

wget https://github.com/eko/pihole-exporter/releases/latest/download/pihole_exporter-linux-arm

I gave it execution permission:

chmod a+x pihole_exporter-linux-arm

Now I took my pihole web api from /etc/pihole/setupVars.conf (the value for WEBPASSWORDis the key) to start the exporter

./pihole_exporter-linux-arm -pihole_hostname 192.168.178.56 -pihole_api_token <here_the_key_from_step_before>

Then I just added a job to my prometheus.yaml:

- job_name: 'pihole'
    static_configs:
      - targets: ['192.168.178.56:9617']

(I can not use localhost since my pihole runs on another device then prometheus)
Then just fire up prometheus and you should see it udner targets in the web panel.

ressources:
https://github.com/eko/pihole-exporter
https://pi-hole.net/blog/2022/11/17/upcoming-changes-authentication-for-more-api-endpoints-required/#page-content
https://grafana.com/grafana/dashboards/10176-pi-hole-exporter/

addendum:

I was not able to get the dashboards, which were mentioned in the above links, working. But I can see the stats at http://192.168.178.56:9617/metrics.
I created some queries by myself inside grafana to make sure it’s working. Cheers