Vaultwarden update command fail & then vaultwarden not start even after reboot

 - Command: cargo build --features sqlite --release
│  - Exit code: 101
│  - DietPi version: v8.6.1 (MichaIng/master) | HW_MODEL: 2 | HW_ARCH: 2 |
│ DISTRO: 6
│  - Image creator: DietPi Core Team
│  - Pre-image: Raspbian Lite
│  - Error log:
│ info: syncing channel updates for 'stable-armv7-unknown-linux-gnueabihf'
│ info: latest update on 2022-06-30, rust version 1.62.0 (a8314ef7d
│ 2022-06-27)
│ info: downloading component 'cargo'
│ info: downloading component 'rust-std'
│ info: downloading component 'rustc'
│ info: installing component 'cargo'
│ info: installing component 'rust-std'

- - -
    Compiling reqwest v0.11.10
│    Compiling yubico v0.11.0
│ LLVM ERROR: out of memory
│ Allocation failed
│ error: could not compile `vaultwarden`
│
│ Caused by:
│   process didn't exit successfully: `rustc --crate-name vaultwarden
│ --edition=2021 src/main.rs --error-format=json
│ --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin
│ --emit=dep-info,link -C opt-level=3 -C lto=thin --cfg
│ 'feature="libsqlite3-sys"' --cfg 'feature="sqlite"' -C
│ metadata=319d1306a6bd97cd -C extra-filename=-319d1306a6bd97cd --out-dir

You are running out of memory still. Because the final vaultwarden build is taking way to much resources for a 32bit operating system it seems. Not relay something we could work around.

Still something that needs to be reported to vaultwarden directly.


I did another test on my RPi3B+ 32bit which failed.

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
error: could not compile `vaultwarden`

Tried to capture memory values for the process, but it looks it did not exceed the 3GB limit :thinking:

root@DietPi3:/tmp# ./memu.sh pid=11153 watch=5
--------------------   ------------   ------------   ------------   ------------   ------------   ------------   ------------   ------------
Date                        Process   Username        Swapped(Mb)     Unique(Mb)     Shared(Mb)        Rss(Mb)        Pss(Mb)      Command
--------------------   ------------   ------------   ------------   ------------   ------------   ------------   ------------   ------------
04 Jul 2022 12:17:21          11153   root                  930.8          831.2            0.2          830.8          831.2   /tmp/DietPi-Software/.rustup/t
04 Jul 2022 12:17:28          11153   root                  927.3          805.2            1.0          807.4          807.2   /tmp/DietPi-Software/.rustup/t
04 Jul 2022 12:17:34          11153   root                  926.2          824.9            1.0          826.9          826.8   /tmp/DietPi-Software/.rustup/t

     The procces with PID 11153 does not exist. Terminated.

root@DietPi3:/tmp#

Indeed I faced the same issue on RPi 2. The fix in v8.6 was mainly for single core x86_64 and aarch64:

  • Single core x86_64 were having the swap size increased to assure 3 GiB memory only, but 4 GiB are required according to my tests.
  • On aarch64 even 6 GiB were required, even just for a second or so: RAM usage stays at below 4 GiB during crate/dependency compiling, even with 6 cores/jobs. When vaultwarden itself is compiled, usage raises up to 5 GiB, with the single job consuming up to 3 GiB alone. At the very end, when the linker kicks in, my Odroid N2+ (respectively the htop output on SSH) was freezing for some seconds, after which I saw for a second or so a memory usage close to 6 GiB. This doesn’t happen to this extend on x86_64 for some reason, where it finished well with 4 GiB overall memory.
  • On armv7l the memory usage during crate compiling is significantly lower compared to aarch64, but it indeed fails as far as I could see and measure, at the very end, probably exactly when the linker kicks. I could at least imagine that it want to use > 3 GiB RAM, which isn’t possible for a single process on 32-bit. I couldn’t see that quickly on Odroid N2+ which exact process took how much memory, it was done too fast after the hanging htop started moving again, so it’s still a theory.

I was in the meantime thinking to provide pre-compiled vaultwarden binaries, respectively packages. Compiling can be done on GitHub, like we do for Armbian. It’s simple now that we have DietPi container images for all architectures and all Debian versions. Not sure whether in emulated armv7l the build succeeds, but worth to give a shot. And it will save our users a lot of compiling time and us the hasse to deal with compiling issues and continuously increasing memory demand of it.

that would be a pretty good idea. It would save a hell of time for our user to compile stuff. But question is, if we are able to success using GitHub actions. :thinking:

1 Like

Just saw an update for Vaultwarden to 1.25.1, but probably doesn’t have any fixes for this issue.

nope, still running into memory issues

memory allocation of 29622276 bytes failed
error: could not compile `vaultwarden`

it seems somehow related to the SWAP usage. If I boot the very same system on a RPi4B 8GB (swapping SD card), it’s working without issues

   Compiling yubico v0.11.0
    Finished release [optimized] target(s) in 37m 46s
[  OK  ] DietPi-Software | cargo build --features sqlite --release

@MichaIng
maybe we should come back to the idea of creating pre-compiled packages using GitHub actions.

1 Like

Agreed. I hope to find time this evening. Yesterday trains had long delays, arrived very late at home after work :disappointed:.

We will provide a ready to use Debian package on next release DietPi-Software | vaultwarden: Add GitHub workflow and build scripts by MichaIng · Pull Request #5638 · MichaIng/DietPi · GitHub

It will save all your time and there is no need to compile Vaultwarden again :slight_smile:

3 Likes

This is awesome. Thanks to the DietPi team for all the hard work!

Packages are currently built, after some trial and error and fixing, now I hope it goes through: vaultwarden · MichaIng/DietPi@f3b076c · GitHub

Testing can be done with the DietPi v8.7 beta released after builds succeeded.

2 Likes

well done @MichaIng , well done. :+1:

1 Like

Sadly it seems like ARMv6 isn’t supported at all. I’ll test manual builds tomorrow, but not much I can do if I run into the same issue with the swap file. However, on ARMv7 it works well.

Here is details and how to apply to the beta: Open Beta v8.7 | Please help testing and hardening our upcoming release · Issue #5643 · MichaIng/DietPi · GitHub

On RPi3B installation is done in around 1 minute, which is a massive improvement compare to the 60+ minutes as it was before. :smiley:

1 Like

Interesting, the build (over night) on RPi Zero W succeeded. I used 8 GiB swap on USB stick. And the process runs. So it is the GRUB emulation, not ARMv6 in general, which has the issues.

Now I see the issue in build logs:

info: default host triple is armv7-unknown-linux-gnueabihf

https://doc.rust-lang.org/cargo/commands/cargo-build.html#compilation-options

Hopefully this helps: v8.7 · MichaIng/DietPi@a9b0539 · GitHub
New ARMv6 builds in progress.

nope still seems to have some issues on ARMv6

root@DietPi1:/tmp# /opt/vaultwarden/vaultwarden
Illegal instruction
root@DietPi1:/tmp#

The build has following issues.

[ 1240.280556] rc.local[586]: [ INFO ] ./rustup-init.sh -y --profile minimal --default-toolchain none, please wait...
[ 1240.523527] rc.local[5024]: head: error reading '/proc/self/exe': Bad file descriptor
[ 1240.524430] rc.local[5024]: head: failed to close '/proc/self/exe': Bad file descriptor
[ 1240.528755] rc.local[5024]: rustup: unknown platform bitness
[ 1240.597836] rc.local[5024]: rustup: command failed: grep ^Features /proc/cpuinfo
[ 1240.631523] rc.local[5024]: info: downloading installer
[ 1242.821004] rc.local[5024]: error: $HOME differs from euid-obtained home directory: you may be using sudo
[ 1242.821749] rc.local[5024]: error: $HOME directory: /tmp/vaultwarden
[ 1242.822237] rc.local[5024]: error: euid-obtained home directory: /root
[ 1242.849979] rc.local[5024]: info: profile set to 'minimal'
[ 1242.850520] rc.local[5024]: info: default host triple is armv7-unknown-linux-gnueabihf

Yes, the build target/tools need to be additionally defined when installing Rust, and the binary is stored in a sub dir. Will fix this and rerun builds after work.

1 Like

Finally, it works on on ARMv6 :smiley:.

confirmed. :+1: is working on my RPi 1 as well

1 Like

There was a vaultwarden update a few days ago. How will it work with future vaultwarden updates? Will you do a build and update via a dietpi patch or something?

1 Like