Learn Some Linux Series -- DD for disk duplication

This should work for debian, ubuntu, dietPi on RPi and essentially any kind of Linux OS.

In this short tutorial, I am copying a 16GB mSATA SSD card on to another card for backup or setting up exact same box. I have Qty 2 of PC Engines APU-1d boxes with 2GB of DDR3 RAM, a dual core arm CPU and 3 of 1Gig ethernet ports, and I have tested them to deliver 500Mbps plus. I have done various things on this capable box, which over next few tutorials, I will share.

So after perfecting my box for the purpose, I just wanted to have same image on the other box.

dd is a utility in Linux that they claim to have meant for copy and convert and they could not name it cc, as that is used for C compiler. So we can just name dd as disk duplicator to clone our one good working disk for a backup to another.

“Just be very careful and if you accidentally use the commands in other way around for source and destination, you will lose your good working disk.”

In my APU box, I have 3 mSATA type of slots, which are mini PCI slots and one of them is used for mSATA (there must be some slight difference as I tried to add second mSATA to second or third slot and it does not work and those slots as per data sheet of the product are meant for mini PCI accessories and not for mini SSDs.

So I ended up buying from Amazon a small USB3 to mSATA adapter. I took mSATA card from second box, added it to the USB adapter, plugged USB adapter into one of the two USB ports (these ports on old boxes are 2.1) of my good working box.

I am running debian 11 latest version and patches and I logged in as normal user and then switched to root by issuing “su -” for all my subsequent work.

Further APU box has a serial (DB9) connector and it works for me at 115200 baud (bits per second) rate, 8N1 (8 bits per symbol / data frame sent, with no parity bit, which is used to add an extra bit of 0 for error correction by calculating number of 1 bits is even or odd (and accordingly called even or odd parity) used in longer error prone industrial or electrical environment in PLCs or in older phone line modems etc. with direct connected laptop to a APU box or a network switch or router or firewall, we don’t have such errors happening, so we use no parity and then in the end is a 1 for 1 stop bit to indicate end of data frame or symbol for other end to mark completion of frame received. For more on serial communications, search for PLC RS323 Serial Communications.

So in my case, I was able to simply disconnect all the I/O activity by unplugging the ethernet cables as I was logged in via serial terminal (using putty or something like that). This sped up the writing of 16G of data. Not sure how long it would have taken if I was doing it with ethernet cable plugged in to have my SSH session on. In case you are using RPi, then you have a console over HDMI cable to do the same thing.

fdisk -l

It showed me that my main drive is sda and it has 3 partitions. And second drive on the USB adapter also had 3 partitions, but of course different stuff in it and it appears as sdc (in your case it could be sdb or whatever, just make note).

Note everything in Linux is a file. Even drives are thus presented as files. These files are then mounted as disk drive or USB drive or DVD drive etc via mount points on the Linux file tree (inverted tree with root at the very top and everything branching off of that, with root of file system symbolized as /). /dev/sda etc implies a disk driver for ssd a.

So as a first step, I needed to zeroize the data on the disk that I want to copy over to (in my case sdb in USB adapter).

I issued the command below: if is for incoming file or source disk (in this case a dummy all zero content) and of is for outgoing file or destination disk to write to.

dd if=/dev/zero of=/dev/sdb bs=128k count=1 conv=notrunc

In 3 seconds I was back at my command prompt. here we used one count of destroying data, but used no truncate to keep original files of same size etc, so that geometry and thus formatting remained same. We simply removed file pointers (which give files name, set up as file hardlinks to inodes, the actual custodian of data).

confirm with command “fdisk -l” and we will see sdb with its capacity and with no partitions.

Make a note of the sector end value from the source disk (I had partitions as sda1, sda2 and sd5 and end sector was at 31277055)

Now issue this command below and it tool an hour and 15 minutes (after I unplugged the ethernet cables). This command is going to also show the the progress bar, so you know how much has been copied over and approx how much longer.

you may need to install pv (pipe viewer) utility, via

apt install pv
dd if=/dev/sda bs=128k count=31277055 conv=sync,noerror | pv -s 16G |sudo dd of=/dev/sdb

bs is block size of how many bits are copied at a time. count is to copy until that sector. no error is to continue copying even if there is any read errors (which we don’t expect anyway in a new SSD) by substituting such data with zeros for destination disk. sync is to synchronize all cached data(which kernel does while copying the disk data) to the destination before imaging is completed. pv shows progress of data copying using pipeline viewer. The pipeline is a mechanism for data passage between different processes in Linux and pv thus gets inserted in between the copying process (the pipelines as you see in command above) for visual progress bar. the size of 16G is shown as approximate size.

When it completed, it was 95% progress and a small error message saying it has no more space left. Well it actually copies 100% as I used 16G for 100% sizing of the progress bar and as we know 16G disk has some space that is not available for use after formatting from raw disk.

I use this cloned disk as such in second box, it complains of boot failure or something along the lines. Believe sync did not fully complete writing the cached data.

fdisk -l shows both disks now of 3 partitions of exact same start and end sectors and same sizes, but still does not boot.

So before we unplug it from our cloning setup, do the following and it fixes error with some free blocks and free inodes count to be wrong.

fsck -yf /dev/sdb1

it does all fixes without user interaction and within seconds, it is ready.

eject /dev/sdb

or power down the main box (shutdown now) and then unplug USB adapter, remove the mSATA or SSD whatever you have and plug into the other box and it should work.

3 Likes

Somehow I am not getting the option to edit my tutorial, I wanted to add that we can also use dd for making a copy of the image for later use, either to restore to the same or to another disk / memory card etc.

Basic format will be something like

dd if=/dev/sda of=diskbackup.img

And if someone was wondering what the f in fdisk meant, it came from old time reference of hard drives to be fixed drive, when they were huge and not portable to be moved around.

1 Like