Special characters are being replaced by ♦ in Linux-Console

Hi guys,

I am developing a Terminal UI software in rust, in which I want to use ASCII arts. I want to run this software on a Rpi Zero W with DietPI in the long term and I want to display the Terminal UI over the Rpi Zero’s HDMI port and NOT via SSH on another machine. When I test the program on the DietPI via SSH everything works fine.

However, when I attach a monitor and a keyboard to the Rpi and I am running the same program on the terminal application of DietPi then all the special characters I am trying to render are being displayed as little diamonds (Unicode Character “:diamonds:” (U+2666)) but I am trying to render ⣾ (U+28FE : BRAILLE PATTERN DOTS-2345678) for example

To see whether the issue is caused by my program I checked the source code of it in nano text editor since the source code of my program itself contains the ASCII arts as string literals. My special characters are being displayed as diamonds using nano too.

If I run nano over SSH (with gnome-terminal) on the Rpi the special characters are being rendered just fine.

Another issue is that the background color of the progress-bar is darker behind the numbers, as you can see on the bottom of the second image, but that issue is for another day… :smiley:

I have tried to change the fonts of the terminal using setfont but it did not help. I have tried to look up what terminal emulator DietPi is using by default or can it be replaced with another one but I found no useful information what so ever.

Can you think of any ideas or advice? What could cause this diamond conversion of the special characters? What terminal emulator is running on DietPi? Can it be replaced? Thank you very much for your help in advance.

More images: dietpi forum – Google Drive

I am using DietPi v9.1.1

I did a bit of googling and basically, the tty console provided by the Linux kernel (aka the one you see when directly running dietpi’s console without ssh) doesn’t natively support unicode chars. When you use SSH, the output of the TTY is sent to the client and decoded by whatever console attached to it – which may or may not support unicode.

There’s a command someone mentioned called “unicode_start” that supposedly enables unicode mode for the linux TTY, so maybe try running that before your program :thinking:

Hi,

Thank you for your reply. :slight_smile: I have tried unicode_start command but it did not work either. :confused:

I have found an easy way to reproduce the issue. If I run the following command on Linux-Console I should get a braille pattern but I get a diamond instead: echo -e '\u2813'

I am getting to the point where I am giving up on Linux-Console. I have read somewhere that many pull-request have been created that would give real support for Unicode character-set in Linux-Console, but the Linux Foundation rejected them all because they want it to be as simple as possible. According to them it is only for emergency scenarios where there is nothing else left for I/O.

As I mentioned earlier I have other issues with it besides this Unicode rendering misery. Is there a way to install another (more advanced) terminal emulator on DietPi without installing a whole window manager system? I want my program to run on the Rpi Zero W and I have limited RAM capacity and processing power on it… Are there terminal emulators that can run on DietPi the same way as Linux-Console does?

Thank you. :slight_smile:

“Terminal emulator” is called such, as it indeed emulates a terminal software-wise from a desktop/X/GUI session. The problem is indeed that an actual terminal is attached to the /dev/tty* consoles, not an emulator which could add further features software-wise.

With dpkg-reconfigure console-setup, you can already configure the console to use UTF-8 encoding (which is also the default). AFAIK the problem is that it supports only 256 or 512 symbols at a time, and hence a console font would be needed which contains those you want to use. /etc/default/console-setup contains a small example about how to add custom fonts.

unicode_start seems to have the same limitation: It enables UTF-8 mode, but the console font needs to contain the characters as well. And for the keyboard input part, one needs to attach the wanted UTF-8 characters to the wanted keyboard keys first. So it is not like just running unicode_start enables support for all unicode characters OOTB.

This is btw true for all Linux distributions. Hence when searching only for solutions/workarounds, it does not matter much about which Linux distro it is. Packages may have different names, but the basics are the same.