Custom Service file for custom program with auto communication between other apps

mail2rst · August 26, 2023, 6:28pm

I am using Dietpi as operating system & home -assistant, mqtt broker & zigbee2mqtt application on my nuc. In past i was using dietpi provided mqtt broker but recently i switched to rumqttd
" https://github.com/bytebeamio/rumqtt/tree/main/rumqttd "
i am now getting little trouble, in past when i was reboot the system home-assistant, zigbee2mqtt & mqtt broker all apps work after reboot but now my apps start but those are not communicate with each other. i have to give manually these commands systemctl restart zigbee2mqtt & then systemctl restart rumqttd then communication start working between all three of them.
I have write my rumqttd service file also the dietpi mosqitto service file. please guide me how i can fix this trouble & i want all three apps start working by itself every time after reboot.

[Unit]
Description=rumqttd
#After=network.target
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/mnt/dietpi_userdata/rumqttd/rumqttd_active -c /mnt/dietpi_userdata/rumqttd/rumqttd.toml
WorkingDirectory=/mnt/dietpi_userdata/rumqttd
#StandardOutput=inherit
StandardOutput=null
StandardError=inherit
Restart=always
RestartSec=10s
User=rumqttd
Group=rumqttd

[Install]
WantedBy=multi-user.target

[Unit]
Description=Mosquitto MQTT Broker (DietPi)
Documentation=man:mosquitto.conf(5) man:mosquitto(8)
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
NotifyAccess=main
User=mosquitto
ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Joulinar · August 26, 2023, 6:47pm

Do you use an external drive that is hosting your data?

mail2rst · August 26, 2023, 7:17pm

Thanks, No everything is on one ssd. this machine self include everything & work headless,

root@wyse:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           380M  5.3M  374M   2% /run
/dev/sda2        15G  5.4G  8.5G  39% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G  4.4M  2.9G   1% /tmp
tmpfs            50M  8.0K   50M   1% /var/log
/dev/sda1        63M   12M   52M  19% /boot/efi

only attached things zigbee2mqtt usb coordinator on USB, either-net wire & power wire. all data is written on self contain ssd. all dietpi standard installation.

Jappe · August 26, 2023, 7:39pm

After a reboot can you check the status of the services and the logs:

systemctl status zigbee2mqtt && systemctl status rumqttd
journalctl -u zigbee2mqtt && journalctl status rumqttd

mail2rst · August 26, 2023, 7:53pm

root@wyse:~# systemctl status zigbee2mqtt && systemctl status rumqttd
● zigbee2mqtt.service - zigbee2mqtt
     Loaded: loaded (/etc/systemd/system/zigbee2mqtt.service; enabled; preset: enabled)
     Active: active (running) since Sat 2023-08-26 13:52:10 MDT; 45s ago
   Main PID: 492 (npm start)
      Tasks: 31 (limit: 4496)
     Memory: 193.1M
        CPU: 9.420s
     CGroup: /system.slice/zigbee2mqtt.service
             ├─492 "npm start"
             ├─813 sh -c "node index.js"
             └─814 node index.js

Aug 26 13:52:10 wyse systemd[1]: Started zigbee2mqtt.service - zigbee2mqtt.

● rumqttd.service - rumqttd
     Loaded: loaded (/etc/systemd/system/rumqttd.service; enabled; preset: enabled)
     Active: active (running) since Sat 2023-08-26 13:52:10 MDT; 45s ago
   Main PID: 481 (rumqttd_active)
      Tasks: 4 (limit: 4496)
     Memory: 7.3M
        CPU: 42ms
     CGroup: /system.slice/rumqttd.service
             └─481 /mnt/dietpi_userdata/rumqttd/rumqttd_active -c /mnt/dietpi_userdata/rumqttd/rumqttd.toml

Aug 26 13:52:10 wyse systemd[1]: Started rumqttd.service - rumqttd.

mail2rst · August 26, 2023, 7:56pm

root@wyse:~# journalctl -u zigbee2mqtt && journalctl -u rumqttd
Aug 26 13:52:10 wyse systemd[1]: Started zigbee2mqtt.service - zigbee2mqtt.
Aug 26 13:52:10 wyse systemd[1]: Started rumqttd.service - rumqttd.

this time homeassistant not showing sensors values. All three apllications home-assistant, rumqttd, zigbee2mqtt working. but those are not communicated with each other yet.

Joulinar · August 27, 2023, 11:56am

looks like services are started correctly, no error message shown. Anything in HA logs?

mail2rst · August 27, 2023, 12:54pm

in all service files i made “StandardOutput=inherit” for more logs, still after just reboot sensors are not online

root@wyse:~# systemctl status zigbee2mqtt && systemctl status rumqttd && systemctl status home-assistant
● zigbee2mqtt.service - zigbee2mqtt
     Loaded: loaded (/etc/systemd/system/zigbee2mqtt.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-08-27 06:51:02 MDT; 2min 21s ago
   Main PID: 488 (npm start)
      Tasks: 31 (limit: 4496)
     Memory: 160.3M
        CPU: 10.450s
     CGroup: /system.slice/zigbee2mqtt.service
             ├─488 "npm start"
             ├─809 sh -c "node index.js"
             └─810 node index.js

Aug 27 06:51:02 wyse systemd[1]: Started zigbee2mqtt.service - zigbee2mqtt.
Aug 27 06:51:06 wyse npm[488]: > zigbee2mqtt@1.30.2 start
Aug 27 06:51:06 wyse npm[488]: > node index.js
Aug 27 06:51:07 wyse npm[810]:                 Zigbee2MQTT requires node version ^14 || ^16 || ^18 || ^19, you are running v20.5.1!

● rumqttd.service - rumqttd
     Loaded: loaded (/etc/systemd/system/rumqttd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-08-27 06:51:02 MDT; 2min 21s ago
   Main PID: 480 (rumqttd_active)
      Tasks: 4 (limit: 4496)
     Memory: 16.9M
        CPU: 50ms
     CGroup: /system.slice/rumqttd.service
             └─480 /mnt/dietpi_userdata/rumqttd/rumqttd_active -c /mnt/dietpi_userdata/rumqttd/rumqttd.toml

Aug 27 06:51:02 wyse systemd[1]: Started rumqttd.service - rumqttd.
Aug 27 06:51:02 wyse rumqttd_active[480]:
Aug 27 06:51:02 wyse rumqttd_active[480]:          ___ _   _ __  __  ___ _____ _____ ___
Aug 27 06:51:02 wyse rumqttd_active[480]:         | _ \ | | |  \/  |/ _ \_   _|_   _|   \
Aug 27 06:51:02 wyse rumqttd_active[480]:         |   / |_| | |\/| | (_) || |   | | | |) |
Aug 27 06:51:02 wyse rumqttd_active[480]:         |_|_\\___/|_|  |_|\__\_\|_|   |_| |___/
Aug 27 06:51:02 wyse rumqttd_active[480]:

● home-assistant.service - Home Assistant (DietPi)
     Loaded: loaded (/etc/systemd/system/home-assistant.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-08-27 06:51:02 MDT; 2min 21s ago
   Main PID: 479 (hass)
      Tasks: 22 (limit: 4496)
     Memory: 350.1M
        CPU: 21.915s
     CGroup: /system.slice/home-assistant.service
             └─479 /home/homeassistant/.pyenv/versions/3.11.4/bin/python3.11 /home/homeassistant/.pyenv/versions/3.11.4/bin/hass -c /mnt/dietpi_userdata/homeassistant

Aug 27 06:51:02 wyse systemd[1]: Started home-assistant.service - Home Assistant (DietPi).
Aug 27 06:51:11 wyse Home Assistant[479]: 2023-08-27 06:51:11.344 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
Aug 27 06:51:46 wyse Home Assistant[479]: 2023-08-27 06:51:46.464 WARNING (Thread-5) [pychromecast.socket_client] [Android TV(192.168.1.91):8009] Heartbeat timeout, resetting connection
Aug 27 06:51:47 wyse Home Assistant[479]: 2023-08-27 06:51:47.044 WARNING (Thread-6) [pychromecast.socket_client] [gini(192.168.1.71):8009] Heartbeat timeout, resetting connection

mail2rst · August 27, 2023, 1:00pm

root@wyse:~# journalctl -u zigbee2mqtt && journalctl -u rumqttd && journalctl -u home-assistant
Aug 27 06:51:02 wyse systemd[1]: Started zigbee2mqtt.service - zigbee2mqtt.
Aug 27 06:51:06 wyse npm[488]: > zigbee2mqtt@1.30.2 start
Aug 27 06:51:06 wyse npm[488]: > node index.js
Aug 27 06:51:07 wyse npm[810]:                 Zigbee2MQTT requires node version ^14 || ^16 || ^18 || ^19, you are running v20.5.1!
Aug 27 06:51:02 wyse systemd[1]: Started rumqttd.service - rumqttd.
Aug 27 06:51:02 wyse rumqttd_active[480]:
Aug 27 06:51:02 wyse rumqttd_active[480]:          ___ _   _ __  __  ___ _____ _____ ___
Aug 27 06:51:02 wyse rumqttd_active[480]:         | _ \ | | |  \/  |/ _ \_   _|_   _|   \
Aug 27 06:51:02 wyse rumqttd_active[480]:         |   / |_| | |\/| | (_) || |   | | | |) |
Aug 27 06:51:02 wyse rumqttd_active[480]:         |_|_\\___/|_|  |_|\__\_\|_|   |_| |___/
Aug 27 06:51:02 wyse rumqttd_active[480]:
Aug 27 06:51:02 wyse systemd[1]: Started home-assistant.service - Home Assistant (DietPi).
Aug 27 06:51:11 wyse Home Assistant[479]: 2023-08-27 06:51:11.344 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
Aug 27 06:51:46 wyse Home Assistant[479]: 2023-08-27 06:51:46.464 WARNING (Thread-5) [pychromecast.socket_client] [Android TV(192.168.1.91):8009] Heartbeat timeout, resetting connection
Aug 27 06:51:47 wyse Home Assistant[479]: 2023-08-27 06:51:47.044 WARNING (Thread-6) [pychromecast.socket_client] [gini(192.168.1.71):8009] Heartbeat timeout, resetting connection

in all logs all three apps started & working but sensor values not reflected on home-assistant, ok

mail2rst · August 27, 2023, 1:07pm

my working rumqtt.toml file is

id = 0

# A commitlog read will pull full segment. Make sure that a segment isn't
# too big as async tcp writes readiness of one connection might affect tail
# latencies of other connection. Not a problem with preempting runtimes
[router]
id = 0
instant_ack = true
max_segment_size = 104857600
max_segment_count = 10
max_read_len = 10240
max_connections = 10010
max_outgoing_packet_count = 200
# [bridge]
# name = "bridge-1"
# addr = "localhost:1883"
# qos = 0
# sub_path = "#"
# reconnection_delay = 5
# ping_delay = 5
# timeout_delay = 5
#     [bridge.connections]
#     connection_timeout_ms = 60000
#     max_client_id_len = 256
#     throttle_delay_ms = 0
#     max_payload_size = 20480
#     max_inflight_count = 500
#     max_inflight_size = 1024
#     dynamic_filters = true
#     [bridge.transport.tls]
#     ca = "ca.cert.pem"
#     client_auth = { certs = "test-1.cert.pem", key = "test-1.key.pem" }

# Configuration of server and connections that it accepts
[v4.1]
name = "v4-1"
listen = "0.0.0.0:1883"
next_connection_delay_ms = 1
    [v4.1.connections]
    connection_timeout_ms = 60000
    max_client_id_len = 256
    throttle_delay_ms = 0
#    max_payload_size = 20480
    max_payload_size = 50000
    max_inflight_count = 500
    max_inflight_size = 1024
    dynamic_filters = true
#    auth = { mosquitto = "dietpi", user2 = "password" }
       [v4.1.connections.auth]
       mosquitto = "dietpi"
#       user2 = "password"
#[v4.2]
#name = "v4-2"
#listen = "0.0.0.0:8883"
#next_connection_delay_ms = 10
    # tls config for rustls
#    [v4.2.tls]
#    capath = "/etc/tls/ca.cert.pem"
#    certpath = "/etc/tls/server.cert.pem"
#    keypath = "/etc/tls/server.key.pem"
    # settings for all the connections on this server
#    [v4.2.connections]
#    connection_timeout_ms = 60000
#    throttle_delay_ms = 0
#    max_payload_size = 20480
#    max_inflight_count = 100
#    max_inflight_size = 1024

[v5.1]
name = "v5-1"
listen = "0.0.0.0:1884"
next_connection_delay_ms = 1
    [v5.1.connections]
    connection_timeout_ms = 60000
    max_client_id_len = 256
    throttle_delay_ms = 0
    max_payload_size = 50000
    max_inflight_count = 500
    max_inflight_size = 1024
       [v5.1.connections.auth]
       mosquitto = "dietpi"



#[prometheus]
#listen = "127.0.0.1:9042"
#interval = 1

[ws]

[console]
listen = "0.0.0.0:3030"

# [metrics]
#     [metrics.alerts]
#     push_interval = 1
#     [metrics.meters]
#     push_interval = 1

Joulinar · August 27, 2023, 9:51pm

just out of interest, is there an order in which the services would need to be started?

mail2rst · August 28, 2023, 1:16am

As per my general sense, As rumqttd(mqtt broker) is standalone program. zigbee2mqtt & home assistant need to connect with username & password to mqtt broker. so i guess first mqtt broker need to start then zigbee2mqtt & home assistant on second position. this is my guess i am not expert in this. Is in dietpi software installed app there is any priority to start up? as mqtt broker install by dietpi works as zigbee2mqtt as custom app. so mqtt broker start before zigbee2mqtt. but now rumqttd & zigbee2mqtt comes on same custom app priority. thanks

Jappe · August 28, 2023, 6:46am

You can set dependencies in your service files. Systemd services has options like Requires=,After=, Wants= and Before=.
I think you can simply use Before= and After=, you can edit them via dietpi-services.

If unit foo.service contains the setting Before=bar.service and both units are being started, bar.service 's start-up is delayed until foo.service has finished starting up. After= is the inverse of Before= , i.e. while Before= ensures that the configured unit is started before the listed unit begins starting up, After= ensures the opposite, that the listed unit is fully started up before the configured unit is started.
https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=

mail2rst · August 28, 2023, 2:12pm

I edited & added before & after in my service files. even then after reboot sensors are not coming online. sensors are come online after “systemctl restart zigbee2mqtt” & then “systemctl restart rumqttd”

[Unit]
Description=zigbee2mqtt
#Wants=rumqttd.service
#After=network.target rumqttd.service
Wants=network-online.target
After=network-online.target rumqttd.service
#After=network-online.target rumqttd.service
[Service]
Environment=NODE_ENV=production
ExecStart=/usr/local/bin/npm start
WorkingDirectory=/mnt/dietpi_userdata/zigbee2mqtt
StandardOutput=inherit
#StandardOutput=null
StandardError=inherit
Restart=always
RestartSec=10s
User=zigbee2mqtt

[Install]
WantedBy=multi-user.target

#############################
[Unit]
Description=rumqttd
#After=network.target
Before=zigbee2mqtt.service home-assistant.service 
Wants=network-online.target
After=network-online.target

[Service]
##Type=notify
##NotifyAccess=main
##User=rumqttd
##ExecStart=/mnt/dietpi_userdata/rumqttd/rumqttd_active -c /mnt/dietpi_userdata/rumqttd/rumqttd.toml
##ExecReload=/bin/kill -HUP $MAINPID

ExecStart=/mnt/dietpi_userdata/rumqttd/rumqttd_active -c /mnt/dietpi_userdata/rumqttd/rumqttd.toml
#ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/mnt/dietpi_userdata/rumqttd
StandardOutput=inherit
#StandardOutput=null
StandardError=inherit
Restart=always
RestartSec=10s
User=rumqttd
Group=rumqttd

[Install]
WantedBy=multi-user.target

##########################
[Unit]
Description=Home Assistant (DietPi)
Wants=network-online.target
After=network-online.target mariadb.service rumqttd.service

[Service]
SyslogIdentifier=Home Assistant
User=homeassistant
ExecStart=/home/homeassistant/homeassistant-start.sh
RestartForceExitStatus=100
StandardOutput=inherit
#StandardOutput=null
StandardError=inherit

[Install]
WantedBy=multi-user.target

Joulinar · August 28, 2023, 8:45pm

I thought zigbee2mqtt would need to be started before rumqttd. Now, it seems exactly the other way around.

mail2rst · August 28, 2023, 9:38pm

even with these option system not picking sensors values after restart


[Unit]
Description=rumqttd
#After=network.target
#Before=zigbee2mqtt.service home-assistant.service
Wants=network-online.target
After=network-online.target zigbee2mqtt.service

[Service]
##Type=notify
##NotifyAccess=main
##User=rumqttd
##ExecStart=/mnt/dietpi_userdata/rumqttd/rumqttd_active -c /mnt/dietpi_userdata/rumqttd/rumqttd.toml
##ExecReload=/bin/kill -HUP $MAINPID

ExecStart=/mnt/dietpi_userdata/rumqttd/rumqttd_active -c /mnt/dietpi_userdata/rumqttd/rumqttd.toml
#ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/mnt/dietpi_userdata/rumqttd
StandardOutput=inherit
#StandardOutput=null
StandardError=inherit
Restart=always
RestartSec=10s
User=rumqttd
Group=rumqttd

[Install]
WantedBy=multi-user.target


#######################


Unit]
Description=zigbee2mqtt
#Wants=rumqttd.service
#After=network.target rumqttd.service
Before=rumqttd.service
Wants=network-online.target
After=network-online.target
#After=network-online.target rumqttd.service
[Service]
Environment=NODE_ENV=production
ExecStart=/usr/local/bin/npm start
WorkingDirectory=/mnt/dietpi_userdata/zigbee2mqtt
#StandardOutput=inherit
StandardOutput=null
StandardError=inherit
Restart=always
RestartSec=10s
User=zigbee2mqtt

[Install]
WantedBy=multi-user.target

mail2rst · August 29, 2023, 7:54pm

I gave up & go back to Dietpi Mqtt broker. its not having any issue & working seamless with home-assistant & zigbee2mqtt. thanks