Killed my Lighttpd

Troubleshooting
1

Hello!

Seeing how my Pihole showed LOTS of requests to “r3.o.lencr.org”, I investigated and found out that something’s wrong with the SSL certification of Let’s Encrypt. Thinking it was an easy fix, I reran dietpi-letsencrypt, but it didn’t help. I then manually deleted the SSL certificate via

certbot delete

Now, lighthttpd doesn’t start, because it looks for the certificate. Without the started lighthttpd, I can’t get a new certificate. Before I screw up more, I thought I’d ask for help.

journalctl -u lighttpd.service




Feb 28 13:04:46 DietPi systemd[1]: Starting Lighttpd Daemon...
Feb 28 13:04:46 DietPi lighttpd[4921]: 2021-02-28 13:04:46: (mod_openssl.c.885) SSL: error:02001002:system library:fopen:No such file or directory /etc/letsencrypt/live/MY_DOMAIN/fullchain.pem
Feb 28 13:04:46 DietPi lighttpd[4921]: 2021-02-28 13:04:46: (mod_openssl.c.1089) SSL: error:20074002:BIO routines:file_ctrl:system lib /etc/letsencrypt/live/MY_DOMAIN/fullchain.pem
Feb 28 13:04:46 DietPi lighttpd[4921]: 2021-02-28 13:04:46: (server.c.1183) Initialization of plugins failed. Going down.
Feb 28 13:04:46 DietPi systemd[1]: lighttpd.service: Control process exited, code=exited, status=255/EXCEPTION
Feb 28 13:04:46 DietPi systemd[1]: lighttpd.service: Failed with result 'exit-code'.
Feb 28 13:04:46 DietPi systemd[1]: Failed to start Lighttpd Daemon.
Feb 28 13:04:46 DietPi systemd[1]: lighttpd.service: Service RestartSec=100ms expired, scheduling restart.
Feb 28 13:04:46 DietPi systemd[1]: lighttpd.service: Scheduled restart job, restart counter is at 1.
Feb 28 13:04:46 DietPi systemd[1]: Stopped Lighttpd Daemon.
Feb 28 13:04:46 DietPi systemd[1]: Starting Lighttpd Daemon...
2

Hi,

you would need to remove Lighttpd SSL configuration as well if you remove the certificates. Otherwise Lighttpd will not start anymore. Let’s have a look

ls -la /etc/lighttpd/conf-enabled/
3 
root@DietPi:~# ls -la /etc/lighttpd/conf-enabled/
total 8
drwxr-xr-x 2 root root 4096 Dec  6 00:33 .
drwxr-xr-x 4 root root 4096 Feb 28 13:28 ..
lrwxrwxrwx 1 root root   33 Oct 29 17:15 10-fastcgi.conf -> ../conf-available/10-fastcgi.conf
lrwxrwxrwx 1 root root   33 Oct 29 17:15 10-rewrite.conf -> ../conf-available/10-rewrite.conf
lrwxrwxrwx 1 root root   37 Oct 29 17:15 15-fastcgi-php.conf -> ../conf-available/15-fastcgi-php.conf
lrwxrwxrwx 1 root root   38 Oct 29 18:50 50-dietpi-https.conf -> ../conf-available/50-dietpi-https.conf
lrwxrwxrwx 1 root root   37 Oct 29 18:54 98-dietpi-hsts.conf -> ../conf-available/98-dietpi-hsts.conf
lrwxrwxrwx 1 root root   47 Oct 29 18:50 98-dietpi-https_redirect.conf -> ../conf-available/98-dietpi-https_redirect.conf
lrwxrwxrwx 1 root root   45 Oct 29 17:15 99-dietpi-dav_redirect.conf -> ../conf-available/99-dietpi-dav_redirect.conf
lrwxrwxrwx 1 root root   41 Oct 29 17:15 99-dietpi-owncloud.conf -> ../conf-available/99-dietpi-owncloud.conf
lrwxrwxrwx 1 root root   58 Oct 29 17:16 99-dietpi-pihole-block_public_admin.conf -> ../conf-available/99-dietpi-pihole-block_public_admin.conf
lrwxrwxrwx 1 root root   39 Oct 29 17:16 99-dietpi-pihole.conf -> ../conf-available/99-dietpi-pihole.conf
lrwxrwxrwx 1 root root   38 Oct 29 17:06 99-unconfigured.conf -> ../conf-available/99-unconfigured.conf
4

pls remove following two sym links

50-dietpi-https.conf
98-dietpi-https_redirect.conf

once done, verify the configuration

/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf

if all is fine, restart Lighttpd

5

This worked, the process is running again and I can access the Pihole website via http, https is not working (and since I forward http → https, I can’t access my owncloud).

Can I just run dietpi-letsencrypt again?

6

yes should be possible

7 
[  OK  ] DietPi-LetsEncrypt | Lighttpd webserver detected
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
live directory exists for MY_DOMAIN
[FAILED] Certbot failed with error code (1), please check its terminal output. Aborting...
[ SUB1 ] DietPi-Services > restart
[  OK  ] DietPi-Services | occ maintenance:mode --on
[  OK  ] DietPi-Services | restart : redis-server
[   .. ] DietPi-Services | mariadb

Checking what happens…

cat /var/log/letsencrypt/letsencrypt.log
[...]
2021-02-28 18:00:21,505:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1250, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 121, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 428, in obtain_and_enroll_certificate
    self.config)
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 1008, in new_lineage
    "live directory exists for " + lineagename)
certbot.errors.CertStorageError: live directory exists for MY_DOMAIN




root@DietPi:~# certbot delete
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Which certificate(s) would you like to delete?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: MY_DOMAIN
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
An unexpected error occurred:
TypeError: remove: path should be string, bytes or os.PathLike, not NoneType
Please see the logfiles in /var/log/letsencrypt for more details.

It seems an empty certificate folder gets created somehow, so it blocks the creation of a new one. If I run dietpi-letsencrypt again, it will create MY_DOMAIN-0001, which seems to be a working key. Then I have two directories listed under Certbot.

Could I just rename -0001 to my normal one, and expect it to work?

8

you would need to check which certificate is configured inside 50-dietpi-https.conf

9

Seems like I hit a rate limit on LetsEncrypt servers, so I can’t get new certificates now. I’ll try again later.

10

yes your limited to a fixed amount of request on a specific time frame. Some sort of protection from LetsEncrypt side

11

Yeah, it looks like 5/week. Not unreasonable, but for a noob like me it’s a probem now :frowning: I can’t use https for a week now. Well, I just hope I can fix it next sunday! :slight_smile:

12

are you sure it’s 5/week only? :thinking:

If you have a backup, probably you are able to copy certificate files back

13

Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week.

Sadly, I don’t have backups of the certificates. There are a few files in /etc/letsencrypt/keys (00xx_key-certbot.pem), but I’m not sure if I can use them.

14

just give it try tomorrow again :wink:

15

It really took a week! Had to take down external access to my cloud in the meantime, but now I could request one more certificate (only to be blocked again on my next request… I hope I can try again later today!)

I got an error:

[  OK  ] DietPi-LetsEncrypt | Lighttpd webserver detected
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
live directory exists for MY_DOMAIN
[FAILED] Certbot failed with error code (1), please check its terminal output. Aborting...
[ SUB1 ] DietPi-Services > restart
[  OK  ] DietPi-Services | occ maintenance:mode --on
[  OK  ] DietPi-Services | restart : redis-server
[  OK  ] DietPi-Services | restart : mariadb
[  OK  ] DietPi-Services | restart : php7.3-fpm
[..    ] DietPi-Services | lighttpd

Looking at the installed certificates:

root@DietPi:~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/MY_DOMAIN.conf produced an unexpected error: renewal config file {} is missing a required file reference. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The following renewal configurations were invalid:
  /etc/letsencrypt/renewal/MY_DOMAIN.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
16

I guess you would need to clean all the old config files to get a fresh SSL config

17

I guess you would need to clean all the old config files to get a fresh SSL config

Can you tell me where they’re stored? I’d like to use Dietpi-Letsencrypt again, but since it always requests a certificate, even if it fails later, I’m probably gonna run out of trys before I get it working if I do it alone again.

18

usually files should be located on /etc/letsencrypt/

19 
root@DietPi:/etc/letsencrypt# ls
accounts  archive  cli.ini  csr  keys  live  renewal  renewal-hooks

Could I just nuke it all, since I get a new certificate anyway?

20

have a look into the folder and remove what is pointing to the old domain