Hah, after searching for 30 minutes and not finding anything I decided to post a question.
Of course, 5 minutes later I found a solution with a slightly different search term 
Turning off the network time sync in /boot/dietpi.txt seems to have done the trick
# Network time sync: 0=disabled | 1=boot only | 2=boot + daily | 3=boot + hourly | 4=Daemon + Drift
CONFIG_NTP_MODE=0
dietpi@DietPi:~$ sudo systemd-analyze blame
9.473s ifup@wlan0.service
6.472s systemd-random-seed.service
5.514s dev-mmcblk0p2.device
3.177s keyboard-setup.service
2.818s systemd-udev-trigger.service
2.091s systemd-journald.service
1.508s dietpi-boot.service
1.348s systemd-fsck-root.service
1.348s dietpi-preboot.service
1.307s networking.service
1.263s systemd-fsck@dev-disk-by\x2dpartuuid-0e2887fb\x2d01.service
1.086s systemd-rfkill.service