Serial console for first boot

Creating a bug report/issue

x I have searched the existing open and closed issues

Required Information

  • DietPi version | 9.18.1
  • Distro version | bookworm
  • Kernel version | uname --all
  • Architecture | arm64
  • SBC model | RPi4b
  • Power supply used | PoE with PoE Hat
  • SD card used | none - boot via TFTP and iSCSI

Additional Information (if applicable)

  • Can this issue be replicated on a fresh installation of DietPi?

Set up serial console in dietpi.txt before first boot, boot the Raspberry

Expected behaviour

boot log should be shown on the UART / on a system connected via UART debug cable

Actual behaviour

nothing is shown on the serial console

Extra details

I can easily turn on serial console on the running system and this works just fine. But it’s the first (and second) boot that I want to be able to track via serial console and I cannot figure out how to turn that on in the DietPi image.

Please share the settings you have done before first boot and the pins used. This way I can try to replicate on a test system.

The UART works. Connected to pins 6/8/10 - it works if manually enabled with the dietpi-config tool after boot (and of course during boot when booting Raspbian).

The settings I was trying to find documentation for and kinda failed - the problem of (a) maybe using the wrong search terms and (b) too much AI generated nonsense out there that clearly has no idea what I’m actually asking about.

My reading of the docs implied that I should set

CONFIG_SERIAL_CONSOLE_ENABLE=1

in dietpi.txt and that should do the trick.
There is conflicting information about whether anything is needed in cmdline.txt, so I have

console=serial0,115200 console=tty1

at the beginning of the command line, just in case.

And maybe one needs

uart_enable=1

in config.txt – again, nothing definitive I was able to find anywhere. Just contradictory statements.

And sadly, most of the posts you can find about this are from 2016/2018.

Our raspberrypi-sys-mods package requires an update for the new kernel stack. The /dev/serial0 symlink is not created correctly. You could manually replace it with console=ttyS0 to get this working. On RPi models without onboard WiFi/BT it would be ttyAMA0.

let me try that… That hadn’t be suggested in any of the docs I was able to find…

Edited to add: yup, that seems to work

Thanks

Oh, and just as I claimed that it worked I realize that DietPi then appears to turn the serial console off about 20 seconds into a boot. Not quite sure, yet, why it does that. Need to compare the log on serial with the logs in the filesystem on the running system…

[   16.659809] EXT4-fs (sda2): re-mounted 11ce6a8b-4f36-4992-9e52-bf8891430db1.
[   16.681437] systemd[1]: Finished fake-hwclock.service - Restore / save the current clock.
[   16.693047] systemd[1]: Started systemd-journald.service - Journal Service.
[   16.814369] systemd-journald[253]: Received client request to flush runtime journal.

and then nothing on serial, and sadly that original dmesg log doesn’t appear to get saved in /var/tmp/dietpi/logs…
Looking at the most recent boot for roughly the same spot…

[   21.037712] systemd[1]: Starting systemd-journald.service - Journal Service...
[   21.060178] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...
[   21.080293] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
[   21.102733] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[   21.130403] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[   21.144417] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[   21.157689] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[   21.160280] EXT4-fs (sda2): re-mounted 11ce6a8b-4f36-4992-9e52-bf8891430db1.
[   21.191599] systemd[1]: Finished fake-hwclock.service - Restore / save the current clock.
[   21.206550] systemd[1]: modprobe@configfs.service: Deactivated successfully.
[   21.216961] systemd[1]: Finished modprobe@configfs.service - Load Kernel Module configfs.
[   21.236363] systemd[1]: Finished keyboard-setup.service - Set the console keyboard layout.
[   21.251067] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[   21.261347] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[   21.279751] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[   21.290329] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[   21.307674] systemd[1]: modprobe@fuse.service: Deactivated successfully.
[   21.318018] systemd[1]: Finished modprobe@fuse.service - Load Kernel Module fuse.
[   21.334596] systemd[1]: Started systemd-journald.service - Journal Service.
[   21.468396] systemd-journald[258]: Received client request to flush runtime journal.

I don’t see anything obvious here that would disable the serial console…

Interestingly enough, it actually comes back:

[  242.658967] INFO: task vchiq-keep/0:79 blocked for more than 120 seconds.
[  242.666770]       Not tainted 6.12.34+rpt-rpi-v8 #1 Debian 1:6.12.34-1+rpt1~bookworm
[  242.675515] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  264.674785]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294955916, last ping 4294957168, now 4294958464
[  284.885557] systemd-shutdown[1]: Syncing filesystems and block devices - timed out, issuing SIGKILL to PID 6940.
[  294.908250] systemd-shutdown[1]: Waiting for process: 6940 ((sd-sync)), 3829 (systemd-journal), 5733 (systemd-udevd)
[  363.490991] INFO: task vchiq-keep/0:79 blocked for more than 241 seconds.
[  363.499848]       Not tainted 6.12.34+rpt-rpi-v8 #1 Debian 1:6.12.34-1+rpt1~bookworm
[  363.509161] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  384.926719] systemd-shutdown[1]: Waiting for process: 6940 ((sd-sync)), 3829 (systemd-journal), 5733 (systemd-udevd)

so this is a failed shutdown (see my other posts about this… iSCSI gets turned off and we still try to write things to file system that’s mounted over iSCSI). So there’s something that systemd started which turned off serial console, and then at the end when it gets shut down, serial console starts working again…

Any ideas, @MichaIng ?

It does not. But the primary console used by the init system is tty1, respectively the last console entry in cmdline. Others are used by the kernel only, so after boot output from kernel finished, there is expectetly some seconds silence. Currently not sure whether the images explicitly contain a mask on serial-getty@ttyS0.service, expecting serial0 to exist and to avoid doubled login prompt in that case. Otherweise, once the init system finished, you should get a login prompt on the serial console if an actual UART adapter is connected.

During first boot, depending on RPi model, the serial0 entries are replaced with ttyS0 or ttyAMA0 automatically.

ahh, interesting. thanks for that detail. I think I can work with that to get more data :slight_smile:

I’ll fix the sys mods package this weekend. Long overdue. Does not help for existing images, but will regenerate our RPi ones then.

Since we build on top of our own, slightly ‘hacked’ images, please let me know once you pushed those changes and I can rebuild ours as well.

BTW: removing the regular console from the cmdline.txt and only having the serial console in there helped to a point - at which point DietPi sent all text back to the regular console (so it started to show up on my monitor, not on my serial console :joy:

For automated testing this isn’t ideal, but I realize (as I do so often) that my use case really is a corner case given what most of your users do…

Huh, that is unexpected indeed. I am doing this myself regularly when testing things on SBCs and do not want to attach a HDMI for some reason. Does this happen as well if no HDMI screen is attached, and do you actually get a login prompt on the serial console at some point? Maybe systemd does some magic there if HDMI is attached and getty@tty1 starts and/or the serial-getty@.service instance is masked. That service at least changes the group of the related dev node from dialout to tty. Maybe systemd behaves differently when this did not happen or so :thinking:.

In any case, better to test with fixed sys mods package when the serial0 node exists and serial-getty@serial0.service can start. I’ll ping you once done.

oddly, I don’t want a login on the serial console for this test scenario (and I don’t get it, either, since I have systemd.getty_auto=0 on my cmdline

HDMI is always connected in this scenario (because this RPi is one of a fleet of test boards that all sit behind an IP KVM).

Oh, I didn’t know about this cmdline option. That would make the masks for disabled serial gettys obsolete :thinking:. On the other hand these auto gettys can be handy: the cmdline entry can be usually changed via trailing FAT partition on DietPi images before first boot, or if config is on boot partition anyway like on RPi. But to enable a systemd unit one requires access to the ext4 partition, i.e. another Linux system. But still good to know it can be disabled :slightly_smiling_face:.

I find it mindblowing how many random things there are to ‘know’ about how the various parts of the system interact. And how much outdated and misleading information is out there to be found. That command line option was one of those random finds that I tried and it worked…

1 Like

Absolutely, even when you think you know or should know everything relevant already after years/decades of Linux experience, there are still regularly surprises. And of course things do change as well, especially also in systemd things are added or removed (more added), internal dependencies and behavior, not always fully intended.

… I very recently learned some surprising bash or shell behavior … but I now do not remember anymore what it was and do not find the issue anymore :sweat_smile:. Anyway, reminded me to never be 100% sure whether I know everything about bash, despite working and scripting so intensively with it since ~7 years.

I now get all the way to this

[  OK  ] Finished systemd-update-ut… - Record Runlevel Change in UTMP.

and then it switches back to the monitor and stops logging on the serial console

New raspberrypi-sys-mods package is available via APT repo. It fixes the /dev/serial0 symlink and hence login prompt on older RPi models.

Interestingly, on RPi 5 this was working with the old package, but also with the newest version of the udev rules, /dev/serial0 points to /dev/ttyAMA0, the UART on pins 6-8-10, if enabled via enable_uart=1. I was expecting that symlink to always point to /dev/ttyAMA10, the dedicated UART port, but this is only the case if enable_uart=1 is not set. I was wondering before, whether users mostly use the common UART pins 6-8-10 or the dedicated UART port of the RPi 5. The latter requires a specific UART adapter/cable, which I do not have, hence need to use ttyAMA0 on pins 6-8-10 with generic pin cables. To be most compatible on first boot, we could enable both :thinking:.

1 Like