Vaultwarden using certs generated from external ACME server

Troubleshooting
,
1

I’m using a second server as an ACME agent to generate certificates for my Vaultwarden instance. I’m using an automated scp command to send the certificate files to the Vaultwarden server.
I have previously just used the ACME agent on the same box that Vaultwarden is installed which was working just fine. I was trying to simplify my life and have 1 instance of ACME for all my devices.

The ACME server produces 3 .pem files (cert.pem, chain.pem, fullchain.pem). When I copy those 3 files to my Vaultwarden server, it kills the entire Vaultwarden service and I have to restore the entire diet-pi instance from backup. I’ve tried to delete the new files and just put the original .pem files (that were working) back and it doesn’t matter. Once the service is failed, it will never start again unless I reload from backup which makes troubleshooting really difficult.

I’m honestly a little stuck.

  1. I’m not sure why the certificates are making the Vaultwarden service fail endlessly.
  2. I have no earthly idea as to why restoring the original files back to their original locations doesn’t fix the problem and I have to resort to a backup.

I’m looking for any guidance that can be provided.

Creating a bug report/issue

Required Information

  • DietPi version | 8.18.2
  • Distro version | bullseye 0
  • Kernel version | Linux 6.1.21-v8+ aarch64 GNU/Linux
  • Architecture | arm64
  • SBC model | RPi3
  • Power supply used | Generic 5v 2.1a
  • SD card used | SanDisk

Additional Information (if applicable)

  • Software title | Vaultwarden
  • Was the software title installed freshly or updated/migrated? | software hasn’t been changed other than the certificates that I’m working with.
  • Can this issue be replicated on a fresh installation of DietPi? | haven’t tried that.

Steps to reproduce

  1. Make copies of current certificate files (cert.pem, fullchain.pem, privkey.pem) in /mnt/dietpi_userdata/vaultwarden
  2. Create new certificate files from external ACME server
  3. Copy newly created cert.pem, chain.pem, fullchain.pem files from the external ACME server to the /mnt/dietpi_userdata/vaultwarden directory on the Vaultwarden server.
  4. Reboot Vaultwarden server.

Expected behaviour

  • Vaultwarden uses the new certificate files that were copied over from the external ACME server

Actual behaviour

  • Vaultwarden service fails to start
  • systemctl status vaultwarden.service
    ● vaultwarden.service - vaultwarden (DietPi)
    Loaded: loaded (/lib/systemd/system/vaultwarden.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Fri 2023-06-30 20:07:46 CDT; 5s ago
    Docs: GitHub - dani-garcia/vaultwarden: Unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs
    Process: 1581 ExecStart=/opt/vaultwarden/vaultwarden (code=exited, status=1/FAILURE)
    Main PID: 1581 (code=exited, status=1/FAILURE)
    CPU: 300ms
  • vaultwarden.service: Scheduled restart job, restart counter is at 5.
    Stopped vaultwarden (DietPi).
    vaultwarden.service: Start request repeated too quickly.
    vaultwarden.service: Failed with result ‘exit-code’.
    Failed to start vaultwarden (DietPi).

Extra details

  • Attempted to send ‘reset-failed vaultwarden.service’ to restore the failed service, but the process will not start and ultimately fails again.
2

Please fill out the trouble shoot template.
Also, what webserver do you use? The vaultwarden internal one or an external one?
And when you copied the certs and restarting vaultwarden, can you have a look into
journalctl -u vaultwarden.service.

3

I am using the vaultwarden internal server.
Results from ‘journalctl -u vaultwarden.service’

-- Journal begins at Sat 2023-07-01 10:26:08 CDT, ends at Sat 2023-07-01 10:26:53 CDT. --
Jul 01 10:26:11 BitWarden systemd[1]: Started vaultwarden (DietPi).
Jul 01 10:26:13 BitWarden vaultwarden[364]: /--------------------------------------------------------------------\
Jul 01 10:26:13 BitWarden vaultwarden[364]: |                        Starting Vaultwarden                        |
Jul 01 10:26:13 BitWarden vaultwarden[364]: |--------------------------------------------------------------------|
Jul 01 10:26:13 BitWarden vaultwarden[364]: | This is an *unofficial* Bitwarden implementation, DO NOT use the   |
Jul 01 10:26:13 BitWarden vaultwarden[364]: | official channels to report bugs/features, regardless of client.   |
Jul 01 10:26:13 BitWarden vaultwarden[364]: | Send usage/configuration questions or feature requests to:         |
Jul 01 10:26:13 BitWarden vaultwarden[364]: |   https://vaultwarden.discourse.group/                             |
Jul 01 10:26:13 BitWarden vaultwarden[364]: | Report suspected bugs/issues in the software itself at:            |
Jul 01 10:26:13 BitWarden vaultwarden[364]: |   https://github.com/dani-garcia/vaultwarden/issues/new            |
Jul 01 10:26:13 BitWarden vaultwarden[364]: \--------------------------------------------------------------------/
Jul 01 10:26:13 BitWarden vaultwarden[364]: [INFO] No .env file found.
Jul 01 10:26:14 BitWarden vaultwarden[364]: [2023-07-01 10:26:14.331][_][WARN] Detected TLS-enabled liftoff without enabling HSTS.
Jul 01 10:26:14 BitWarden vaultwarden[364]: [2023-07-01 10:26:14.332][_][WARN] Shield has enabled a default HSTS policy.Jul 01 10:26:14 BitWarden vaultwarden[364]: [2023-07-01 10:26:14.332][start][INFO] Rocket has launched from https://0.0.0.0:8001
Jul 01 10:26:52 BitWarden vaultwarden[364]: [2023-07-01 10:26:52.880][rustls::conn][ERROR] TLS alert received: AlertMessagePayload {
Jul 01 10:26:52 BitWarden vaultwarden[364]:     level: Fatal,
Jul 01 10:26:52 BitWarden vaultwarden[364]:     description: DecryptError,
Jul 01 10:26:52 BitWarden vaultwarden[364]: }
Jul 01 10:26:52 BitWarden vaultwarden[364]: [2023-07-01 10:26:52.881][rocket_http::tls::listener][WARN] tls handshake with 172.16.0.20:57147 failed: received fatal alert: DecryptError
Jul 01 10:26:53 BitWarden vaultwarden[364]: [2023-07-01 10:26:53.000][rustls::conn][ERROR] TLS alert received: AlertMessagePayload {
Jul 01 10:26:53 BitWarden vaultwarden[364]:     level: Fatal,
Jul 01 10:26:53 BitWarden vaultwarden[364]:     description: DecryptError,
Jul 01 10:26:53 BitWarden vaultwarden[364]: }
Jul 01 10:26:53 BitWarden vaultwarden[364]: [2023-07-01 10:26:53.001][rocket_http::tls::listener][WARN] tls handshake with 172.16.0.20:57148 failed: received fatal alert: DecryptError
4

Do you know which device 172.16.0.20 is? As well, it might be a good idea to contact Vaultwarden developer in parallel. Maybe he has an idea what the error message is about exactly. For me it seems to be an issue with the certificate.

5

172.16.0.20 is my workstation.
I can completely understand if there was an issue with the certificate. I would expect to just get a certificate error on the site if the cert I’m using is bad. However what I don’t understand is that changing the cert makes the entire service stop working. On top of that, restoring the certificate that was there before doesn’t fix the issue either. I have to restore a backup to get the system stable again.

6

maybe an issue with file system permission or the certificate format. Best to ask Vaultwarden guys on this Issues · dani-garcia/vaultwarden · GitHub

1 Like