Bullseye update fine but lighttpd is failing

mcnahum · November 9, 2021, 12:42pm

Hi,
I need some help on lighttpd troubleshooting.
after the bullseye update (without issue) I noticed that I cannot access my Pi-hole Webconsole anymore.
Pi-hole is running
diet-service show me that lighttpd failed to start and log is:

lighttpd.service - Lighttpd Daemon                                                                                 
                                                 │      Loaded: loaded (/lib/systemd/system/lighttpd.service; disabled; vendor preset: enabled)                         
                                                 │      Active: failed (Result: exit-code) since Tue 2021-11-09 12:22:00 CET; 1h 17min ago                              │ 
                                                 │     Process: 9326 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited,                   
                                                 │ status=255/EXCEPTION)                                                                                                │ 
                                                 │         CPU: 152ms                                                                                                   │ 
                                                 │                                                                                                                      │ 
                                                 │ Nov 09 12:22:00 DietPi systemd[1]: lighttpd.service: Scheduled restart job, restart counter is at 5.                 
                                                 │ Nov 09 12:22:00 DietPi systemd[1]: Stopped Lighttpd Daemon.                                                          │ 
                                                 │ Nov 09 12:22:00 DietPi systemd[1]: lighttpd.service: Start request repeated too quickly.                             │ 
                                                 │ Nov 09 12:22:00 DietPi systemd[1]: lighttpd.service: Failed with result 'exit-code'.

I need some clues to move the troubleshooting to the next step

(side note: Pi-hole has been installed manually not with diet-pi software…)

Joulinar · November 9, 2021, 12:49pm

You would like to check configuration file. Can be done as follow

/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf

mcnahum · November 9, 2021, 3:30pm

I got this … but when I’m editing “etc/lighttpd/lighttpd.conf” I don’t see duplicated information…

root@DietPi:~# /usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf 
Duplicate config variable in conditional 5 global / $SERVER["socket"] == "[::]:443": ssl.engine
2021-11-09 16:19:03: configfile.c.1970) source: /usr/share/lighttpd/use-ipv6.pl 443 line: 3 pos: 1 parser failed somehow near here: (EOL)
2021-11-09 16:19:03: configfile.c.1970) source: /etc/lighttpd/conf-enabled/10-ssl.conf line: 14 pos: 1 parser failed somehow near here: (EOL)
2021-11-09 16:19:03: configfile.c.1970) source: find /etc/lighttpd/conf-enabled -name '*.conf' -a ! -name 'letsencrypt.conf' -printf 'include "%p"\n' 2>/dev/null line: 6 pos: 8 parser failed somehow near here: (EOL)
2021-11-09 16:19:03: configfile.c.1970) source: /etc/lighttpd/lighttpd.conf line: 75 pos: 1 parser failed somehow near here: (EOL)

Joulinar · November 9, 2021, 5:37pm

there are more configuration files available inside /etc/lighttpd/conf-enabled/
I guess something got activated that is not needed. Do you use SSL? If not you could remove symbolic link of 10-ssl.conf. This should fix some of the issues.

mcnahum · November 9, 2021, 6:30pm

Yes I’m using certificate

By following this: https://discourse.pi-hole.net/t/webinterface-not-accessable-lighttpd-not-started/40593/8 I know have:

root@DietPi:/mnt/RPiBackup#  /usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf 
Duplicate config variable in conditional 3 global / $SERVER["socket"] == "[::]:443": ssl.engine
2021-11-09 18:47:27: configfile.c.1970) source: /etc/lighttpd/conf-enabled/50-dietpi-https.conf line: 32 pos: 13 parser failed somehow near here: (EOL)
2021-11-09 18:47:27: configfile.c.1970) source: /etc/lighttpd/lighttpd.conf line: 47 pos: 15 parser failed somehow near here: (EOL)

root@DietPi:/mnt/RPiBackup# ls -l /etc/lighttpd/conf-enabled/
total 0
lrwxrwxrwx 1 root root 33 19 oct  2020 10-fastcgi.conf -> ../conf-available/10-fastcgi.conf
lrwxrwxrwx 1 root root 29  9 nov 12:19 10-ssl.conf -> ../conf-available/10-ssl.conf
lrwxrwxrwx 1 root root 37 19 oct  2020 15-fastcgi-php.conf -> ../conf-available/15-fastcgi-php.conf
lrwxrwxrwx 1 root root 38 16 sep 12:57 50-dietpi-https.conf -> ../conf-available/50-dietpi-https.conf
lrwxrwxrwx 1 root root 47 16 sep 12:57 98-dietpi-https_redirect.conf -> ../conf-available/98-dietpi-https_redirect.conf
lrwxrwxrwx 1 root root 38 19 oct  2020 99-unconfigured.conf -> ../conf-available/99-unconfigured.conf
root@DietPi:/mnt/RPiBackup# cp /etc/lighttpd/conf-enabled/50-dietpi-https.conf ./tst/50-dietpi-https.conf

MichaIng · November 10, 2021, 8:23pm

Please show the content of /etc/lighttpd/lighttpd.conf and /etc/lighttpd/conf-available/50-dietpi-https.conf.

Did you install Pi-hole with the official installer or via dietpi-software?

Joulinar · November 10, 2021, 9:25pm

PiHole installer as stated on first post MichaIng

mcnahum · November 11, 2021, 10:19am

find the 2 files below, I just obfuscated the certificates name

/etc/lighttpd/lighttpd.conf

server.modules = (
	"mod_indexfile",
	"mod_access",
	"mod_alias",
 	"mod_redirect",
)

server.document-root = "/var/www"
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
server.errorlog             = "/var/log/lighttpd/error.log"
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "www-data"
server.groupname            = "www-data"
server.port                 = 80

# strict parsing and normalization of URL for consistency and security
# https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_http-parseoptsDetails
# (might need to explicitly set "url-path-2f-decode" = "disable"
#  if a specific application is encoding URLs inside url-path)
server.http-parseopts = (
  "header-strict"           => "enable",# default
  "host-strict"             => "enable",# default
  "host-normalize"          => "enable",# default
  "url-normalize-unreserved"=> "enable",# recommended highly
  "url-normalize-required"  => "enable",# recommended
  "url-ctrls-reject"        => "enable",# recommended
  "url-path-2f-decode"      => "enable",# recommended highly (unless breaks app)
 #"url-path-2f-reject"      => "enable",
  "url-path-dotseg-remove"  => "enable",# recommended highly (unless breaks app)
 #"url-path-dotseg-reject"  => "enable",
 #"url-query-20-plus"       => "enable",# consistency in query string
)

index-file.names            = ( "index.php", "index.html" )
url.access-deny             = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

compress.cache-dir          = "/var/cache/lighttpd/compress/"
compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )

# default listening port for IPv6 falls back to the IPv4 port
include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
include_shell "/usr/share/lighttpd/create-mime.conf.pl"
include "/etc/lighttpd/conf-enabled/*.conf"

#server.compat-module-load   = "disable"
server.modules += (
	"mod_compress",
	"mod_dirlisting",
	"mod_staticfile",
)

/etc/lighttpd/conf-available/50-dietpi-https.conf

# Based on: https://ssl-config.mozilla.org/#server=lighttpd
server.modules += ( "mod_openssl" )
# IPv4
$SERVER["socket"] == ":443" {
	protocol = "https://"
	ssl.engine = "enable"

	# pemfile is cert+privkey, ca-file is the intermediate chain in one file
	ssl.pemfile = "/etc/letsencrypt/live/pi-hole.mydomain.net/combined.pem"
	ssl.ca-file = "/etc/letsencrypt/live/pi-hole.e-mydomain.net/fullchain.pem"

	# For DH/DHE ciphers, dhparam should be >= 2048-bit
	#ssl.dh-file = "/path/to/dhparam.pem"
	# ECDH/ECDHE ciphers curve strength, see "openssl ecparam -list_curves"
	ssl.ec-curve = "secp384r1"

	# Environment flag for HTTPS enabled
	setenv.add-environment = ( "HTTPS" => "on" )

	# Intermediate configuration, tweak to your needs
	ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.2", "Options" => "-SessionTicket")
	ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
	ssl.honor-cipher-order = "disable"
	ssl.disable-client-renegotiation = "enable"
}
# IPv6
$SERVER["socket"] == "[::]:443" {
	protocol = "https://"
	ssl.engine = "enable"

	# pemfile is cert+privkey, ca-file is the intermediate chain in one file
	ssl.pemfile = "/etc/letsencrypt/live/pi-hole.mydomain.net/combined.pem"
	ssl.ca-file = "/etc/letsencrypt/live/pi-hole.mydomain.net/fullchain.pem"

	# For DH/DHE ciphers, dhparam should be >= 2048-bit
	#ssl.dh-file = "/path/to/dhparam.pem"
	# ECDH/ECDHE ciphers curve strength, see "openssl ecparam -list_curves"
	ssl.ec-curve = "secp384r1"

	# Environment flag for HTTPS enabled
	setenv.add-environment = ( "HTTPS" => "on" )

	# Intermediate configuration, tweak to your needs
	ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.2", "Options" => "-SessionTicket")
	ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
	ssl.honor-cipher-order = "disable"
	ssl.disable-client-renegotiation = "enable"
}

Joulinar · November 11, 2021, 1:08pm

pls can you share as well following

/etc/lighttpd/conf-enabled/10-ssl.conf

mcnahum · November 11, 2021, 1:35pm

of course:

10-ssl.conf

server.modules += ( "mod_openssl" )

# ssl.* in global scope gets inherited by
#   $SERVER["socket"] == "..." { ssl.engine = "enable" }
ssl.pemfile = "/etc/lighttpd/server.pem"
ssl.cipher-list = "HIGH"

$SERVER["socket"] == "0.0.0.0:443" {
	ssl.engine  = "enable"
}
include_shell "/usr/share/lighttpd/use-ipv6.pl 443"

Joulinar · November 11, 2021, 1:39pm

there you have the duplicate configuration for port 443 as it is defined in

/etc/lighttpd/conf-enabled/10-ssl.conf
/etc/lighttpd/conf-available/50-dietpi-https.conf

10-ssl.conf is a dummy configuration and it seems it got activate during update. I would recommend to remove

rm /etc/lighttpd/conf-enabled/10-ssl.conf

and check config again

/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf

mcnahum · November 11, 2021, 1:45pm

I think that we are progressing there…

root@DietPi:/mnt/RPiBackup/tst# rm /etc/lighttpd/conf-enabled/10-ssl.conf
root@DietPi:/mnt/RPiBackup/tst#  /usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf 
2021-11-11 14:43:58: configfile.c.461) Warning: "mod_compress" is DEPRECATED and has been replaced with "mod_deflate".  A future release of lighttpd 1.4.x will not contain mod_compress and lighttpd may fail to start up
2021-11-11 14:43:58: plugin.c.195) dlopen() failed for: /usr/lib/lighttpd/mod_deflate.so /usr/lib/lighttpd/mod_deflate.so: cannot open shared object file: No such file or directory
2021-11-11 14:43:58: server.c.1238) loading plugins finally failed

Joulinar · November 11, 2021, 1:49pm

did you reinstall lighttpd after Bullseye upgrade? Usually our install will correct this on reinstall automatically.

mcnahum · November 11, 2021, 1:58pm

no I didn’t, I did multiple pihole -r / pihole -a … but I didn’t touched lighttps as it’s coming with dietpi …

any idea how to fix it?

Joulinar · November 11, 2021, 2:03pm

try following

sed -Ei '/^compress\..*=[[:blank:]]*["(].*[")]$/d' /etc/lighttpd/lighttpd.conf
sed -i '/^[[:blank:]]*"mod_compress",$/d' /etc/lighttpd/lighttpd.conf

and check config again

/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf

mcnahum · November 11, 2021, 2:17pm

we are almost there:

root@DietPi:/mnt/RPiBackup/tst# /usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf
2021-11-11 15:14:10: configfile.c.1142) WARNING: unknown config-key: setenv.add-environment (ignored)
2021-11-11 15:14:10: configfile.c.1142) WARNING: unknown config-key: setenv.add-environment (ignored)

So I started lighttps and … yes it’s active

The only issue now is going to https://pi-hole.mysite.net/admin/ give me a 404 Not Found instead of the Pihole web interface

MichaIng · November 11, 2021, 2:18pm

That lighttpd.conf isn’t the one from the Pi-hole installer but ours.

Actually we do a wrong migration in dietpi-software as we enable this 10-ssl.conf while dietpi-letsencrypt doesn’t do it but includes these directives with the final/correct values. Here is the fix: https://github.com/MichaIng/DietPi/commit/f5b2074a0f4f89adbea1036142a1d0822e44080c

Other migration steps are fine and still missing in your case:

apt install lighttpd-mod-deflate
sed -Ei '/^compress\..*=[[:blank:]]*["(].*[")]$/d' /etc/lighttpd/lighttpd.conf
sed -i '/^[[:blank:]]*"mod_compress",$/d' /etc/lighttpd/lighttpd.conf
lighty-enable-mod deflate

MichaIng · November 11, 2021, 2:20pm

Ah setenv.add-environment is deprecated. I didn’t know that. Something to fix in dietpi-letsencrypt to assure that the HTTPS variable is still set. However it should not break anything. What the hack, no hint about this in the official docs : https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModSetEnv

Joulinar · November 11, 2021, 2:21pm

MichaIng
I have seen the code line as well and was already going to ask about. Good you changed it already.

MichaIng · November 11, 2021, 2:28pm

About the setenv warning, logs misinterpreted by me. The module is not loaded, fix it via:

G_CONFIG_INJECT '"mod_setenv"' '	"mod_setenv",' /etc/lighttpd/lighttpd.conf '"mod_.+",'
systemctl restart lighttpd

This is already done in dietpi-letsencrypt, so probably copy&pasting configs in attempt to solve things removed it, e.g. if the user from the Pi-hole forum didn’t have HTTPS enabled .