over-engineered dietpi-writeimage tool... Topic is solved

Have some feedback, questions, suggestions, or just fancy a chat? Pop it in here.
Post Reply
rhkean@gmail.com

over-engineered dietpi-writeimage tool...

Post by rhkean@gmail.com »

**********************************
UPDATED -- Pi-Day Release!!
**********************************

I wrote a little bash script to automate writing the dietpi images to a uSD card...
it will download the 7z file, extract it, write it to the uSD and run an image customizer if found...

here the -h output

Code: Select all

root@keanbean:~# dietpi-writeimage -h

dietpi-writeimage:  Write DietPi Installation image to uSD card.

Usage:
    dietpi-writeimage
    dietpi-writeimage [FLAG]... [OPTION]... [TARGET]

  if no parameters given, dietpi-writeimage will prompt for the block DEVICE
      to write to and the TARGET single board computer.

  if -Y (Assume Defaults) is used: (override any with FLAGs and OPTIONs
    * TARGET defaults to /boot/dietpi/.hw_info or prompts if not found (-T)
    * downloads latest DietPi image from dietpi.com for selected SBC (-I)
    * if found, copies /boot/Automation_Custom_Script (-A)
    if /boot/custom folder is found, it is copied to new image
    if /boot/custom/Customize_Image.sh is found, it is run to customize image

FLAGs:
    -t                 Use testing images
    -v                 Verbose: increase output (may be used multiple times)
    -q                 Quiet:  decrease output (may be used multiple times)
    -h                 Show this usage statement
    -Y                 Assume Defaults for all prompts
    -N                 No-Action, dry-run: do not permanently alter the system

    -i                 write image file to DEVICE
    -d                 same as '-i' + copy dietpi.txt to DEVICE
    -a                 same as '-d' + copy Automation_Custom_Script.sh to DEVICE
    -c                 same as '-a' + copy custom folder to DEVICE

OPTIONs:
    -b DEVICE          Target block device (default /dev/sda)
    -W WORK_DIR        Override temporary working directory.  (default /tmp)

    -T TARGET          Target Single Board Computer (SBC):
                           RPi - Raspberry Pi, Pi2, or Pi3
                           XU4 - Odroid-XU4
                            C1 - Odroid-C1
                           OPi - Orange Pi PC

    -I IMG_FILE        Disable image download and write specific file (sets -i)
    -D DIETPI_TXT      Use this dietpi.txt (sets -d)
    -A AUTO_CUSTOM     Use this Automation_Custom_Script.sh (sets -a)
    -C CUSTOM_DIR      Copy this custom directory to boot partition (sets -c)
Attachments
dietpi-writeimage(v0.9).zip
Pi-Day Release!
(7.76 KiB) Downloaded 162 times
Last edited by rhkean@gmail.com on Mon Mar 14, 2016 1:51 pm, edited 1 time in total.
User avatar
Fourdee
Site Admin
Posts: 2787
Joined: Tue Feb 06, 2007 1:36 pm

Re: over-engineered dietpi-writeimage tool...

Post by Fourdee »

rhkean@gmail.com wrote:
dietpi-writeimage.zip
I wrote a little bash script to automate writing the dietpi images to a uSD card...
it will download the 7z file, extract it, write it to the uSD and run an image customizer if found...
Very nice! I like the the idea of this, could come in handy.

I tried it today, had a few issues:

Code: Select all

-bash: 7z: command not found

Code: Select all

apt-get install -y p7zip

Code: Select all

-bash: 7z: command not found
7zr is the command that worked for me. I replaced references of "7z" to "7zr" in the code.

8GB sd card:

Code: Select all

Download and/extract img file from dietpi.com website
  downloading /var/tmp/DietPi_v109_RPi-(Jessie).img
    deleting existing 7z file from /var/tmp
    downloading http://dietpi.com/downloads/images/DietPi_RPi-(Jessie).7z
    testing /var/tmp/DietPi_RPi-(Jessie).7z
  extracting DietPi_RPi-(Jessie).7z
    extracting /var/tmp/DietPi_RPi-(Jessie).7z
Write img file to uSD card and use dd + md5sum to verify
  writing DietPi_v109_RPi-(Jessie).img to /dev/sda
    Writing /var/tmp/DietPi_v109_RPi-(Jessie).img to /dev/sda
    unmount /dev/sda if it is mounted.
      sync'ing... ... ...
    reading image from /var/tmp/DietPi_v109_RPi-(Jessie).img (639631360 bytes)
    writing 1249 blocks of 512000 bytes/block to /dev/sda
    please wait...
dd: error writing ‘/dev/sda’: No space left on device
      sync'ing... ... ...

!!! Failed to write image to uSD card. !!!
I think a simple option/prompt style frontend (not whiptail) would work really well. Just a few options to allow the user to select the device image they require, then the block device?
This would also prevent inexperienced users (or people who didnt check the CLI list) from running the command and overwriting /dev/sda by accident.

We could pull the block devices into a array eg:

Code: Select all

ls /dev/sd[a-z]
Just some ideas, but I like where this is going :) Good job.
If you find our project or support useful, then we’d really appreciate it if you’d consider contributing to the project however you can.
Donating is the easiest – you can use PayPal or become a DietPi patron.
rhkean@gmail.com

Re: over-engineered dietpi-writeimage tool...

Post by rhkean@gmail.com »

hmmm.... I've been writing to 4GB, 32GB and 64GB card... on that you got no space left on card.

can you give me the output of the following for that card:

Code: Select all

parted -s /dev/sda p
also, can you give me the output with the undocumented -g option (adds debug output)

Code: Select all

dietpi-writeimage -g rpi
as for the 7z issue...
oddly, the p7zip package provide 7zr, but the p7zip-full provide 7z and 7za ( but not 7zr ).... I'll add a test for all 3 before executing...

and the device issue...
so, you're suggesting a menu prompt if no block device is given, instead of defaulting to /dev/sda?
rhkean@gmail.com

Re: over-engineered dietpi-writeimage tool...

Post by rhkean@gmail.com »

what do you think of this block of code for prompting for block device?

Code: Select all

select device in $(lsblk -r -n --output RM,TYPE,NAME,SIZE | egrep "^1 disk" | awk '{printf("%s(%s)\n", $3, $4)}'); do
    echo "you picked:  "
    echo "    \$device:  $device"
    echo "    \$REPLY :  $REPLY"
    DEVICE="/dev/${device%%(*}"
    break
done
the RM column denotes 'removable'
the TYPE column shows 'disk' or 'part'
the egrep filters down to just "removable disks"
and the awk statement just pretty's up the select statement.
User avatar
Fourdee
Site Admin
Posts: 2787
Joined: Tue Feb 06, 2007 1:36 pm

Re: over-engineered dietpi-writeimage tool...

Post by Fourdee »

rhkean@gmail.com wrote:can you give me the output of the following for that card:

Code: Select all

parted -s /dev/sda p
Micro SD with DietPi image on it :) So 2 partitions. I use DD to clear partitions: https://github.com/Fourdee/DietPi/blob/ ... #L166-L167

Code: Select all

root@DietPi:~# parted -s /dev/sda p
Model: TS-RDF5 SD Transcend (scsi)
Disk /dev/sda: 7948MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62.9MB  58.7MB  primary  fat16        lba
 2      62.9MB  7948MB  7885MB  primary  ext4
also, can you give me the output with the undocumented -g option (adds debug output)

Code: Select all

root@DietPi:~# /DietPi/dietpi/dietpi-writeimage -g rpi
  [DEBUG]
  [DEBUG]  TESTING     :  []
  [DEBUG]  QUIET       :  []
  [DEBUG]  LASTPHASE   :  [4]
  [DEBUG]  DEVICE      :  [/dev/sda]
  [DEBUG]  VERSION     :  [110]
  [DEBUG]  WORK_DIR    :  [/var/tmp]
  [DEBUG]  IMG_FILE    :  [DietPi_v110_RPi-(Jessie).img]
  [DEBUG]  IMG_FULLPATH:  [/var/tmp/DietPi_v110_RPi-(Jessie).img]
  [DEBUG]  DIETPI_TXT  :  [/boot/dietpi.txt]
  [DEBUG]  AUTO_CUSTOM :  []
  [DEBUG]  CUSTOM_DIR  :  []
  [DEBUG]  DEBUG       :  [true]
  [DEBUG]  SBC         :  [RPi]
  [DEBUG]  IMG_URL     :  [http://dietpi.com/downloads/images/DietPi_RPi-(Jessie                                          ).7z]
  [DEBUG]  IMG_ZIP     :  [DietPi_RPi-(Jessie).7z]
  [DEBUG]  MOUNT_POINT :  [/var/tmp/mnt]
  [DEBUG]  TEST_PATH   :  []
  [DEBUG]
Download and/extract img file from dietpi.com website
  downloading /var/tmp/DietPi_v110_RPi-(Jessie).img
    deleting existing 7z file from /var/tmp
    downloading http://dietpi.com/downloads/images/DietPi_RPi-(Jessie).7z
    testing /var/tmp/DietPi_RPi-(Jessie).7z
  [DEBUG]  downloadImageFile:  0
  extracting DietPi_RPi-(Jessie).7z
    extracting /var/tmp/DietPi_RPi-(Jessie).7z
  [DEBUG]  extractImageFile: 0
Write img file to uSD card and use dd + md5sum to verify
Unable to locate /var/tmp/DietPi_v110_RPi-(Jessie).img

!!! Failed to write image to uSD card. !!!
and the device issue...
so, you're suggesting a menu prompt if no block device is given, instead of defaulting to /dev/sda?
I think its a good idea to prevent accidental use. As DietPi users range from all skill levels of linux, its worth adding a little protection with user input if we can.
what do you think of this block of code for prompting for block device?
Yeah I like it, works a treat, nice one :)

Code: Select all

root@DietPi:~# ./test
1) sda(7.4G)
#? 1
you picked:
    $device:  sda(7.4G)
    $REPLY :  1
as for the 7z issue...
oddly, the p7zip package provide 7zr, but the p7zip-full provide 7z and 7za ( but not 7zr ).... I'll add a test for all 3 before executing...
Looks like I needed the p7zip-full package. p7zip is already installed :)

Code: Select all

root@DietPi:~# 7z
-bash: 7z: command not found
root@DietPi:~# apt-get install p7zip-full
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  p7zip-rar
The following NEW packages will be installed:
  p7zip-full
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 747 kB of archives.
After this operation, 3,323 kB of additional disk space will be used.
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main p7zip-full armhf 9.20.1~dfsg.1-4.1+deb8u1 [747 kB]
Fetched 747 kB in 0s (921 kB/s)y
Selecting previously unselected package p7zip-full.
(Reading database ... 23120 files and directories currently installed.)
Preparing to unpack .../p7zip-full_9.20.1~dfsg.1-4.1+deb8u1_armhf.deb ...
Unpacking p7zip-full (9.20.1~dfsg.1-4.1+deb8u1) ...
Setting up p7zip-full (9.20.1~dfsg.1-4.1+deb8u1) ...
root@DietPi:~# 7z

7-Zip 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_GB.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)

Usage: 7z <command> [<switches>...] <archive_name> [<file_names>...]
       [<@listfiles...>]

<Commands>
  a: Add files to archive
  b: Benchmark
  d: Delete files from archive
  e: Extract files from archive (without using directory names)
  l: List contents of archive
  t: Test integrity of archive
  u: Update files to archive
  x: eXtract files with full paths
<Switches>
  -ai[r[-|0]]{@listfile|!wildcard}: Include archives
  -ax[r[-|0]]{@listfile|!wildcard}: eXclude archives
  -bd: Disable percentage indicator
  -i[r[-|0]]{@listfile|!wildcard}: Include filenames
  -m{Parameters}: set compression Method
  -o{Directory}: set Output directory
  -p{Password}: set Password
  -r[-|0]: Recurse subdirectories
  -scs{UTF-8 | WIN | DOS}: set charset for list files
  -sfx[{name}]: Create SFX archive
  -si[{name}]: read data from stdin
  -slt: show technical information for l (List) command
  -so: write data to stdout
  -ssc[-]: set sensitive case mode
  -t{Type}: Set type of archive
  -u[-][p#][q#][r#][x#][y#][z#][!newArchiveName]: Update options
  -v{Size}[b|k|m|g]: Create volumes
  -w[{path}]: assign Work directory. Empty path means a temporary directory
  -x[r[-|0]]]{@listfile|!wildcard}: eXclude filenames
  -y: assume Yes on all queries
If you find our project or support useful, then we’d really appreciate it if you’d consider contributing to the project however you can.
Donating is the easiest – you can use PayPal or become a DietPi patron.
rhkean@gmail.com

Re: over-engineered dietpi-writeimage tool...

Post by rhkean@gmail.com »

still baffled on the drive 'no space left on device' error....

you ran this output against a test box with v110 installed, but the download pulled the v109 image. You need to supply "-v 109" or "-I <Image file>" ... that's the only thing that the version is used for.... calculating the img filename...

I agree with your logic on adding the menu select for novice users. Having created dietpi, I'm certain you have had more interactions with users of every experience level. I'll add a new flag "-s" for "silent mode" and for any parameters that are not set, prompt for values or take defaults if -s flag is set.

7zip is weird... 3 different progs... all of which work on a 7z archive. If I were the package maintainer, I think that I would have done that differently. But, I'll add code to test for all 3 and prompt to install p7zip if none are found.
User avatar
Fourdee
Site Admin
Posts: 2787
Joined: Tue Feb 06, 2007 1:36 pm

Re: over-engineered dietpi-writeimage tool...

Post by Fourdee »

rhkean@gmail.com wrote:you ran this output against a test box with v110 installed, but the download pulled the v109 image. You need to supply "-v 109" or "-I <Image file>" ... that's the only thing that the version is used for.... calculating the img filename...
The 7z filenames are consistent and never change. The contents of the zip file only changes the version number. So you could use:

Code: Select all

wget DietPi_RPi-(Jessie).7z
unzip 
write_image DietPi_v*_RPi-(Jessie).img
Another option would be to delete all previous *.img files that are unzipped, wget, unzip then:

Code: Select all

write_image *.img
Not sure if thats any help, i'am probably waffling on lol :)
I agree with your logic on adding the menu select for novice users. Having created dietpi, I'm certain you have had more interactions with users of every experience level. I'll add a new flag "-s" for "silent mode" and for any parameters that are not set, prompt for values or take defaults if -s flag is set.
Probably a good idea. Stops trigger happy users screaming "DietPi wiped all my USB data" :)
still baffled on the drive 'no space left on device' error....
Did you try clearing the partition table and MBR before writing the image?
https://github.com/Fourdee/DietPi/blob/ ... #L166-L167
If you find our project or support useful, then we’d really appreciate it if you’d consider contributing to the project however you can.
Donating is the easiest – you can use PayPal or become a DietPi patron.
rhkean@gmail.com

Re: over-engineered dietpi-writeimage tool...

Post by rhkean@gmail.com »

unzip/version:
I hadn't thought of the wildcard file reference.... I'll look into that.... (probably create a tmp dir to extract into, rather than delete anything off of the user's computer that the tool doesn't put there)

change the default behaviour:
started working on the -s option... re-thinking the variables (use and naming conventions) in the process... will take a day or so... busy work week

dd space issue
no... I hadn't done the wipe-first... you did that before using the tool... which led to the out of space error? interesting...
User avatar
Fourdee
Site Admin
Posts: 2787
Joined: Tue Feb 06, 2007 1:36 pm

Re: over-engineered dietpi-writeimage tool...

Post by Fourdee »

rhkean@gmail.com wrote:dd space issue
no... I hadn't done the wipe-first... you did that before using the tool... which led to the out of space error? interesting...
I did a fresh write of the DietPi image and plugged it into a USB card reader. Attached it to the device and ran your script. It has 2 partitions, so the dd code is probably the easiest way of clearing the partition table.
If you find our project or support useful, then we’d really appreciate it if you’d consider contributing to the project however you can.
Donating is the easiest – you can use PayPal or become a DietPi patron.
rhkean@gmail.com

Re: over-engineered dietpi-writeimage tool...

Post by rhkean@gmail.com »

I've updated dietpi-writeimage for Pi-Day!!!!

the new .zip file is in the OP.

It now prompts for missing parameters if not included on the command line instead of defaulting.
Now gives a multi-select for target board
multi-select for removable disk to write image to
new -N option, NoAction, aka --dry-run
new -Y option, defaults to target board of running system, defaults to 1st removable disk, if no -d, -a, or -c options, then it defaults to all if found
creates and removes temporary working directory

This is a debug build, so it's about 25% larger with all the debug statements, but it also has a -g (debug) option that is not listed on the -h output.
Post Reply