[SOLVED] DietPi-Cloudshell crashes

mike175de · 28 December 2016 19:17

Hi there,

I use the DietPi-Cloudshell on my new installed XU4. After about 20 hours the Cloudshell crashes with this message in the display:

DietPi-Cloudshell terminated have a nice day!

The cloudshell.log is empty. Most of the time I can restart the Cloudshell with dietpi-cloudshell. But sometimes I can’t access the XU4 with SSH or by plug in a keyboard and trying to start a new console. Then only a hard reset brings the device back to life.

Maybe someone has a hint why the Cloudshell crashes and someone experienced the same problem?

If any logs are necessary please let me know.

Thx for your help and time.

Mchael

Fourdee · 29 December 2016 11:57

Hi,

But sometimes I can’t access the XU4 with SSH or by plug in a keyboard and trying to start a new console. Then only a hard reset brings the device back to life.

Very strange.
I need to redo our XU4 image as I believe it may contain a filesystem corruption. Once thats done, would you be willing to reinstall and re-test?

In the mean time, lets see if the cloudshell service has any information on why it terminated. Please paste results of:

systemctl status dietpi-cloudshell -l

mike175de · 29 December 2016 12:51

Hi Fourdee,

thx for you reply.

# systemctl status dietpi-cloudshell -l
● dietpi-cloudshell.service - dietpi-cloudshell on main screen
   Loaded: loaded (/etc/systemd/system/dietpi-cloudshell.service; disabled)
   Active: active (running) since Mi 2016-12-28 20:04:03 CET; 17h ago
  Process: 1944 ExecStop=/bin/bash -c /DietPi/dietpi/func/dietpi-notify 0 DietPi-Cloudshell terminated, have a nice day! (code=exited, status=0/SUCCESS)
  Process: 1941 ExecStop=/usr/bin/setterm --reset (code=exited, status=0/SUCCESS)
  Process: 3980 ExecStartPre=/usr/bin/setterm --term linux --blank 0 --powersave off --cursor off (code=exited, status=0/SUCCESS)
 Main PID: 3983 (dietpi-cloudshe)
   CGroup: /system.slice/dietpi-cloudshell.service
           ├─ 3983 /bin/bash /DietPi/dietpi/dietpi-cloudshell 1
           └─29608 sleep 5

Dez 28 20:04:03 dietpiserver systemd[1]: Started dietpi-cloudshell on main screen.

I pause the Cloudshell between 22h and 07h.

Michael

mike175de · 29 December 2016 16:07

Hi Fourdee,

1 hour ago the system crashed again and wasn’t responding anymore. only a hard reset helped to bring back the system.

I send you a bug report (after booting the system again): 197992ab-e8c3-4712-a55a-4efefcd70b93-0

UPDATE/EDIT:
I generated another bugreport. This time the system was accessable via ssh so that I could generate a bugreport right after cloudshell crashed:
197992ab-e8c3-4712-a55a-4efefcd70b93-1
Maybe that helps?

Thx a lot for your time an help again ;o)

Michael

mike175de · 2 January 2017 18:53

Hi Fourdee,

after I disabled the cloudshell and rebooted my system, no more crashes occured. The system is stable since that.

Maybe that is useful for you and others with similar problems.

Greets, Michael

Fourdee · 2 January 2017 19:19

Hi Micheal,

Thanks for sending the report.

I pause the Cloudshell between 22h and 07h.

Is this by using the screen off feature in dietpi-cloudshell, or another method?

after I disabled the cloudshell and rebooted my system, no more crashes occured. The system is stable since that.

Weird. Cloudshell logs look fine.

Your bug report didn’t appear to pull in all the requested files, was there any errors on screen during the bugreport send?
Can you please paste results of /etc/fstab

1 hour ago the system crashed again and wasn’t responding anymore. only a hard reset helped to bring back the system.

Can you remember if anything (eg: errors) was displayed on the Cloudshell LCD, or HDMI output before you did the hard reset?

mike175de · 2 January 2017 22:54

Hi Fourdee,

Thx for your reply.

Is this by using the screen off feature in dietpi-cloudshell, or another method?

I used the feature of the dietpi-cloudshell tool.

Can you please paste results of /etc/fstab

Sure:

#Internal Drives---------------------------------------------------
proc            /proc           proc    defaults											0 0
/dev/mmcblk0p1  /boot           auto    defaults,noatime,discard							0 2
##rootfs on sdcard
##/dev/mmcblk0p2  /               auto    defaults,noatime,discard							0 1
## rootfs on USB device
/dev/disk/by-uuid/053da7d3-5570-4b4a-b1d8-68d20c962942    /        ext4    defaults,noatime,nodiratime                  0 1
tmpfs 			/tmp  			tmpfs 	defaults,noatime,nodev,nosuid,mode=1777				0 0
tmpfs 			/var/log 		tmpfs 	defaults,size=20m,noatime,nodev,nosuid,mode=1777	0 0
tmpfs 			/DietPi 		tmpfs 	defaults,size=10m,noatime,nodev,nosuid,mode=1777	0 0

#External Drives---------------------------------------------------
#NB: Please use dietpi-drive_manager to setup and control your external drives.
##/dev/sda1       /mnt/usb_1      auto     defaults,noatime,nofail,x-systemd.automount  0 0
#/dev/sdb1       /mnt/usb_2      auto     defaults,noatime,nofail,x-systemd.automount  0 0
#/dev/sdc1       /mnt/usb_3      auto     defaults,noatime,nofail,x-systemd.automount  0 0
#/dev/sdd1       /mnt/usb_4      auto     defaults,noatime,nofail,x-systemd.automount  0 0
#/dev/sde1       /mnt/usb_5      auto     defaults,noatime,nofail,x-systemd.automount  0 0

#USB-Drives
UUID=ddbce875-95ba-4147-ae35-80b4147556db	/western	ext4	defaults	0	0
UUID=77437d55-0190-4710-b254-a588406215b6	/toshiba	ext4	defaults	0	0
UUID=763755c3-04ca-42e9-bd3b-d262cde8d055	/logilink	ext4	defaults	0	0

#Samba Client------------------------------------------------------
#/mnt/samba . Please use dietpi-config and the Networking Options: NAS menu to setup this mount

#FTP Client Mount--------------------------------------------------
#/mnt/ftp_client . Please use dietpi-config and the Networking Options: NAS menu to setup this mount

#NFS Client Mount--------------------------------------------------
#/mnt/nfs_client . Please use dietpi-config and the Networking Options: NAS menu to setup this mount
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

I use the SDcard only to boot, everything else comes from my connected USB-Drive within the XU4-Cloudshell.
The three USB-Drives are my mass storage drives that I connect via Samba to my network.

Can you remember if anything (eg: errors) was displayed on the Cloudshell LCD, or HDMI output before you did the hard reset?

There was only the message:

DietPi-Cloudshell terminated have a nice day!

Nothing more. Sorry that I can’t provide more details on that.

Again a big Thank you! for your time and hard work with DietPi!

Greets, Michael

Fourdee · 3 January 2017 19:35

Hi Michael,

Thanks for the results.

How did you apply the rootfs on USB drive, was this with dietpi-drive_manager, or manually?

3 USB drives:

May be a power issue.

Are these powered by USB, or 3.5inch with dedicated PSU’s?
Which PSU are you running for the board (eg: 5v/6a)?
Are the USB <> Sata converters built into the USB drive case (eg: WD elements), or third party (eg: you purchased a 2.5 inch drive, then a USB <> Sata converter with case)?

It may also be worth disabling HDD spindown on /dev/sda1 and re-testing. Possibly the drive/USB converter doesn’t “like” it.
Disable the following lines in /etc/hdparm.conf, then reboot:

#DietPi external USB drive. Power management settings.
/dev/sda {
        #10 mins
        spindown_time = 120

        #
        apm = 254
}

Thanks Michael.

mike175de · 4 January 2017 08:39

Hi Fourdee,

How did you apply the rootfs on USB drive, was this with dietpi-drive_manager, or manually?

I used the the Community Tutorial “Move the filesystem to a USB Drive” by K-Plan [Tutorial - Outdated] Odroid: - Move the filesystem to a USB Drive ==> so I did it manually.

May be a power issue.

Are these powered by USB, or 3.5inch with dedicated PSU’s?

Which PSU are you running for the board (eg: 5v/6a)?

Are the USB <> Sata converters built into the USB drive case (eg: WD elements), or third party (eg: you purchased a 2.5 inch drive, then a USB <> Sata converter with case)?

The 3 USB-Drives are connected to an active hub, which I already used before I did the reinstall of DietPi - worked without problems. The power should be more than sufficent. 2 drives are “whole in one-solutions” (eg :WD elements), one is a third party (as you mentioned) solution.

It may also be worth disabling HDD spindown on /dev/sda1 and re-testing. Possibly the drive/USB converter doesn’t “like” it.
Disable the following lines in /etc/hdparm.conf, then reboot:

I disabled hdparm as you suggested.

BTW: You mentioned that the filesystem may be corrupted:

I need to redo our XU4 image as I believe it may contain a filesystem corruption. Once thats done, would you be willing to reinstall and re-test?

Maybe that is the problem because I still have crashes/freezes after 20-30h of operations although cloudshell is disabled

Greets Michael

mike175de · 4 January 2017 12:03

Hi Fourdee,

so I am absolute sure that it isn’t cloudshell related. DietPi crashed again and only a hard reset brought the device back to life.

I uploaded some log files (syslog, kernel.log, debug always the last 500 lines before the crash) to pastebin:
http://pastebin.com/6h4MGiMC
Maybe that helps to find the problem?

Thx again, Michael

Fourdee · 13 January 2017 13:23

Hi Michael,

Yep definatly sounds like a system issue. From what I can see in the logs, dietpi-cloudshell is running correctly with no errors.

This looks worrying, each power on, external drives are being recovered. This might simply be a check due to not being powered down correctly, but still, its not good to see:

Jan  4 11:17:05 dietpiserver kernel: [   23.235112] [c0] EXT4-fs (sdd1): recovery complete
Jan  4 11:17:05 dietpiserver kernel: [   23.237332] [c1] EXT4-fs (sdd1): mounted filesystem with ordered data mode. Opts: (null)
Jan  4 11:17:05 dietpiserver kernel: [   23.767618] [c2] EXT4-fs (sdb3): recovery complete
Jan  4 11:17:05 dietpiserver kernel: [   23.819377] [c2] EXT4-fs (sdb3): mounted filesystem with ordered data mode. Opts: (null)
Jan  4 11:17:05 dietpiserver kernel: [   23.916532] [c1] EXT4-fs (sdc1): recovery complete
Jan  4 11:17:05 dietpiserver kernel: [   23.932516] [c1] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: (null)

The 3 USB-Drives are connected to an active hub, which I already used before I did the reinstall of DietPi - worked without problems. The power should be more than sufficent. 2 drives are “whole in one-solutions” (eg :WD elements), one is a third party (as you mentioned) solution.

Things that pop to mind, for testing.
As the problem only occurs 20-30h’s in, this might take some time to find the cause:

Remove the third party USB <> Sata controller. “Cheap” controllers can be problematic. This would also remove 500-750mA power usage on PSU.
Remove the USB hub. May be problematic. If powered by its own PSU, it will need to be at least 5v/2.5A and a stable 5v.
Around 20 hours in, have a quick look at htop and see if RAM + Swap is full.
If you are using a 5v/4a with 3x USB drives + Cloudshell, its not enough power (unless the hub is powered by its own 5v/2.5A PSU). The XU4 will peak to around 3A+ during high load. With 3 USB drives on top of that at 500-750mA each, you’d probably need at least a 5v/8a to truly cover the power requirements.

mike175de · 13 January 2017 18:36

Hi Fourdee,

thx for your reply.

My system is fine and running for almost 10 days now.

What I did was to remove hdparm and replace it with hd-idle. Since then everything works. Maybe this is helpful for somebody else who is using more than 2 USB-Drives powered with an active hub.

I mark the post as solved. Thx again for your help and the hint that pointed me to hdparm.

Greets Michael