apt update readings lists very slow. Topic is solved

Have some feedback, questions, suggestions, or just fancy a chat? Pop it in here.
Post Reply
bassopt
Posts: 13
Joined: Sat Mar 03, 2018 11:25 am

apt update readings lists very slow.

Post by bassopt »

Hi!

For some reason after updating to the lastest version of DietPi every time i try to do a sudo apt update when it gets to reading package lists.... its really really slow.

I'm not sure if this is a dietpi problem or the fact that debian updated to 10.6. since both updates occurred at the time so i'm not sure what it can be...although i have a vpn running debian 10.6 without dietpi and it doesn't happend.

I was able to reproduce this on my Raspberry pi4 and a x86 dual core i3. On the Rpi4 it become so slow it's almost painful.

Thanks.
J.
User avatar
Joulinar
Posts: 2300
Joined: Sat Nov 16, 2019 12:49 am

Re: apt update readings lists very slow.

Post by Joulinar »

Hi,

many thanks for your message. Yes indeed, apt could takes longer on DietPi for some reasons. Especially on slower devices and/or slower SD cards.
  1. We disable APT cache by default, to avoid writing 60 - 100 MiB to SD card on each apt update.
  2. We leave downloaded APT list files xz-compressed to further reduce SD card writes by about the same or more, depending on how many lists need to be updated.
  3. This means APT needs to extract the lists and parse them in text (instead of cached binary) form, which takes a while.
  4. Since SD card deaths are still number one reason for system failures and data loss, we prioritise reduced SD card writes over performance here ;) .
How long does it takes on your RPi4? For me, apt update finished in around 50s.

Code: Select all

root@DietPi4:~# date
Sun Oct  4 20:19:07 CEST 2020
root@DietPi4:~# apt update
Get:1 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Get:2 http://raspbian.raspberrypi.org/raspbian bullseye InRelease [15.0 kB]
Get:3 https://archive.raspberrypi.org/debian buster InRelease [32.6 kB]
Get:4 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages [13.0 MB]
Get:5 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages [12.7 MB]
Fetched 25.8 MB in 22s (1169 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
root@DietPi4:~# date
Sun Oct  4 20:19:59 CEST 2020
root@DietPi4:~#
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
bassopt
Posts: 13
Joined: Sat Mar 03, 2018 11:25 am

Re: apt update readings lists very slow.

Post by bassopt »

Hi.

Was this a thing on recent versions or always this way? because it started only with latest update. If so is there a way to revert it? specially on my i3 machine where i have a 500gb ssd in ... it used to take like 4 or 5 seconds to apt update now its taking almost 30.

I don't have the Rpi4 with dietpi right now. i had to use it for another project.
But this is what is happening on a Asus ThinkerBoard i have running for months. It was never blazing fast but was WAY faster than it is now

Code: Select all

Get:1 http://deb.debian.org/debian unstable InRelease [146 kB]
Hit:2 https://deb.debian.org/debian buster InRelease
Get:3 https://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Hit:4 https://deb.debian.org/debian-security buster/updates InRelease
Get:5 https://deb.debian.org/debian buster-backports InRelease [46.7 kB]
Get:7 http://deb.debian.org/debian unstable/main armhf Packages.diff/Index [27.9 kB]
Get:8 http://deb.debian.org/debian unstable/main armhf Packages 2020-10-03-1400.50.pdiff [6,089 B]
Get:9 http://deb.debian.org/debian unstable/main armhf Packages 2020-10-03-2000.50.pdiff [8,511 B]
Get:10 http://deb.debian.org/debian unstable/main armhf Packages 2020-10-04-0200.53.pdiff [4,352 B]
Get:11 http://deb.debian.org/debian unstable/main armhf Packages 2020-10-04-0800.48.pdiff [1,709 B]
Get:12 http://deb.debian.org/debian unstable/main armhf Packages 2020-10-04-1401.12.pdiff [13.3 kB]
Get:12 http://deb.debian.org/debian unstable/main armhf Packages 2020-10-04-1401.12.pdiff [13.3 kB]
Get:6 https://us.mirrors.fossho.st/armbian/apt buster InRelease [18.3 kB]
Get:13 https://deb.debian.org/debian buster-backports/main armhf Packages.diff/Index [27.8 kB]
Get:14 https://deb.debian.org/debian buster-backports/main armhf Packages 2020-10-03-1400.50.pdiff [188 B]
Get:14 https://deb.debian.org/debian buster-backports/main armhf Packages 2020-10-03-1400.50.pdiff [188 B]
Get:15 https://us.mirrors.fossho.st/armbian/apt buster/main armhf Packages [230 kB]
Fetched 583 kB in 1min 32s (6,322 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.

real	1m56.137s
user	2m1.009s
sys	0m2.216s
User avatar
Joulinar
Posts: 2300
Joined: Sat Nov 16, 2019 12:49 am

Re: apt update readings lists very slow.

Post by Joulinar »

If I'm not mistaken, it was introduced in one of the last releases. If needed you could deactivate the compressed feature as follow

Code: Select all

echo 'Acquire::GzipIndexes "false";' > /etc/apt/apt.conf.d/98dietpi-uncompressed
/boot/dietpi/func/dietpi-set_software apt-cache clean
apt update
On my RPI3B+, apt update execution time reduced from 20s down to 6s

Before

Code: Select all

root@DietPi3:~# time apt update
Hit:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
Hit:2 https://archive.raspberrypi.org/debian buster InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.

real    0m19.475s
user    0m19.106s
sys     0m0.449s
root@DietPi3:~#
After

Code: Select all

root@DietPi3:/etc/apt/apt.conf.d# time apt update
Hit:1 https://archive.raspberrypi.org/debian buster InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.

real    0m6.336s
user    0m5.005s
sys     0m0.496s
root@DietPi3:/etc/apt/apt.conf.d#
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
bassopt
Posts: 13
Joined: Sat Mar 03, 2018 11:25 am

Re: apt update readings lists very slow.

Post by bassopt »

Thanks!!! Solved :)
Post Reply