Improved automation during installation

Suggestions for features and software you would like to see in DietPi, goes here.
Post Reply
yuj
Posts: 3
Joined: Sat May 23, 2020 5:49 pm

Improved automation during installation

Post by yuj »

Hello there,
i'm new by here, and first of all, thank you for the great piece of software !

I came to DietPi while starting a home project using a small cluster of 4 XU4 boards (Odroid MC1) which i want to fully automate (installation and then containerization using Docker or similar software). DietPi was the best choice for that purpose, to my mind.

Anyway, there are 2 improvements that i'd like to do :
1) the root fs (userdata) needs to be deported to a USB device during installation.
2) the boot fs must be set readonly on the µSD once the installation is done.

=> I don't want to write ANY single bit of data to the µSD card after the final automated installation process. The USB device will be used for this purpose.

I spent a few hours looking at the DietPi scripts, and i believe this can be done "easily".
I did it manually just to be sure, using the GUI (dietpi-drive_manager) and CLI (nano /etc/fstab, option ro on /boot)
Reboot and here we go... it works \o/

So here is the workflow i'm thinking about :
  1. Flash the SD card with a fresh DietPi image,
  2. Replace some scripts with customized ones (TODO),
  3. put the SD card and a USB storage in the XU4,
  4. power on,
    The following will be automated :
  5. the boot process starts and detects a fresh install, with a custom option "Use external USB storage and secure µSD card",
  6. /boot and /rootfs are mounted from the µSD, and a /dev/sda is detected,
  7. the custom script launches dietpi-drive_manager with no user input to :
    1. format the USB storage with EXT4, and maybe fix its UID,
    2. rsync the root to USB storage,
    3. modify the mount point of /
    4. reboot
  8. the boot process starts and detects a not-so-fresh install, with a custom option "Use external USB storage and secure µSD card",
  9. ...do the usual update and other automations...
  10. before last reboot, the custom script sets the boot fs to read-only.
Before actually modifying the scripts for my needs, do you have some hint, advice or feedback for me to do this ?
Do you foresee any problem ?

Thanks in advance for your feedback !
yuj
Posts: 3
Joined: Sat May 23, 2020 5:49 pm

Re: Improved automation during installation

Post by yuj »

A little feedback on this topic :

- Step 5 "detects a fresh install, with a custom option "Use external USB storage and secure µSD card"" is trivial : just by adding a Automation_Custom_PreScript.sh to /boot, DietPi will detect and execute it. Great !

- Step 7 "the custom script launches dietpi-drive_manager with no user input" is OK by modifying dietpi-drive_manager to support a "headless" mode, at least for my special needs. That's a bit of work, though.

- Step 7.b is KO : at that point, rsync is not available and the network is not ready for use. I need to use dd instead :-/
That means dietpi-drive_manager won't help me much, since its Move_RootFS helper function won't apply.

So i decided to do all customization in Automation_Custom_PreScript.sh
yuj
Posts: 3
Joined: Sat May 23, 2020 5:49 pm

Re: Improved automation during installation

Post by yuj »

Hi,

well no success at the moment :?

The script below seems to stop during the "Copy RootFS, including its PTUUID and PARTUUID" step : dd if=$currentRootFS of=/dev/sda1
I test it on a Rpi 3 B+ and i guess it crashes while copying to the USB key, with the key becoming a bit hot....

Here is the script, any feedback/advice is welcome.

Code: Select all

#!/bin/bash
#
# DietPi Automation script : pre-installation steps.
#
# This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
# You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
#
# Name  : /boot/Automation_Custom_PreScript.sh
#
# Goal  : This script is meant to be used during the first boot of a fresh DietPi installation.
#	  It will be automatically executed before network is up and before DietPi starts its own installation. See the  DietPi documentation in dietpi.txt
#         It will disappear after use.
#
# Usage : - Flash a DietPi image to a SDcard, 
#	  - Copy this file in the /boot partition of the SD card,
#	  - Insert the SD Card in your SBC **it should be a Rpi or Odroid XU4, HC-1, HC-2 or MC1**,
#	  - Insert an **EMPTY** USB key or drive in your SBC,
#	  - Power on the SBC.
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# /!\ WARNING: this script will DESTROY your DATA. USE IT WITH CARE AND KNOWLEDGE, OR DO NOT USE IT AT ALL. You've been warned ! /!\
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# This script is designed to automatically look for the first storage medium available (supposely a USB drive or key), and if one is found then :
# - format this storage using EXT4 with a new UUID,
# - copy the whole rootFS to the USB storage,
# - adapt /etc/fstab and related files,
# - auto-clean up and reboot to continue normal DietPi installation.
#
echo "Automation_Custom_PreScript : starting" > /boot/so-automation.log
Do_It() {
local i=1 currentRootFS=$(findmnt -rn -o SOURCE /)

while :
do
  # Check if /dev/sda exists
  if [[ $(lsblk -Snr -o NAME,TYPE | grep sda) != "sda disk" ]]; then
    echo "Cannot find /dev/sda USB disk. Skipping automation" >> /boot/so-automation.log
    break
  fi

  # Check if any /dev/sdaX exist and is mounted
  echo "Unmounting /dev/sdaX partitions" >> /boot/so-automation.log
  while [[ -n $(findmnt -S /dev/sda$i) ]]; do
    echo "Found /dev/sda$i" >> /boot/so-automation.log
    umount /dev/sda$i
    dd if=/dev/zero of=/dev/sda$i bs=4K count=10 # Partition wipe must be done 1st, else blkid still reports UUIDs etc
    ((i++))
  done

  # Clear partition table from device
  echo "Clearing partition table on /dev/sda" >> /boot/so-automation.log
  dd if=/dev/zero of=/dev/sda bs=4K count=1337 # Block device wipe

  # Create partition table type
  echo "Creating partition table, with target type: gpt" >> /boot/so-automation.log
  parted -s /dev/sda mklabel gpt
  parted -s /dev/sda mkpart primary 0% 100%
  partprobe /dev/sda

  # Format as EXT4
  mkfs.ext4 -q -F -m 0 /dev/sda1 >> /boot/so-automation.log

  # Copy RootFS, including its PTUUID and PARTUUID
  echo "Copying $currentRootFS to /dev/sda1" >> /boot/so-automation.log
  dd if=$currentRootFS of=/dev/sda1

  # Update fstab to use new root /dev/sda1
  echo "Changing rootfs source to /dev/sda1"
  cp -a /etc/fstab /etc/fstab.old # just in case of failure, for manual restore
  # - Replace old rootfs by new one
  sed -i "\@UUID=[^[:blank:]]*[[:blank:]][[:blank:]]*/[[:blank:]]@c /dev/sda1 / ext4 noatime,lazytime,rw 0 1" /etc/fstab

  # Finally update manufacturers' related files
  # RPi | /boot/cmdline.txt
  if [[ -e /boot/cmdline.txt ]]; then
    echo "Updating /boot/cmdline.txt" >> /boot/so-automation.log
    local newPARTUUID=$(blkid -s PARTUUID -o value "/dev/sda1")
    # Find current root= and replace
    local rootfs_current=$(mawk '{for(i=1;i<=NF;i++) {print $i} }' /boot/cmdline.txt | grep -m1 '^root=')
    sed -i "s#$rootfs_current#root=PARTUUID=$newPARTUUID#g" /boot/cmdline.txt

    # Set FS type
    local rootfstype_current=$(mawk '{for(i=1;i<=NF;i++) {print $i} }' /boot/cmdline.txt | grep -m1 '^rootfstype=')
    sed -i "s#$rootfstype_current#rootfstype=ext4#g" /boot/cmdline.txt

    # Add root delay
    grep -qE '(^|[[:blank:]])rootdelay=' /boot/cmdline.txt || sed -i "s#rootfstype=ext4#rootfstype=ext4 rootdelay=10#" /boot/cmdline.txt
  fi

  # C2/XU4 | /boot/boot.ini
  if [[ -e /boot/boot.ini ]]; then
    echo "Updating /boot/boot.ini" >> /boot/so-automation.log
    # Find current root= to replace
    local rootfs_current=$(mawk '{for(i=1;i<=NF;i++) {print $i} }' /boot/boot.ini | grep -m1 '^root=' | sed 's/\"//')
    sed -i "s#$rootfs_current#root=/dev/sda1#g" /boot/boot.ini
  fi
  
  break
done

echo "Automation_Custom_PreScript : end" >> /boot/so-automation.log
rm /boot/Automation_Custom_PreScript.sh
sync
}
Do_It
reboot
Post Reply