Image Your Hard Drive using dd

Like many people here - I was having issues with corrupt restores.

I was restoring from a USB / SD Card. Using "sync" I believe what was happening is that gzip couldn't provide the 64k fast enough so it was, as the documentation suggests, filling the gaps with NULL/ZERO and causing corruption.

REMOVE sync,noerror, but it is safe to leave the bs=64k in there, as you'll get much better performance. I believe now, what happens is that dd waits for each 64K chunk rather than filling the gaps. So on my hard drive image, it started out at around 18MB/s and then as it got to the blank portions of the drive where compression was much higher - it increased to 50 MB/s.

I was trying backup my Windows system using this same strategy. To prepare the Windows partition, I created a 10G /dev/zero file in Linux, copied over to Windows, and kept coping it over and over until I filled up the disk. I then deleted them, booted a thumb drive to Linux, and tried to DD the Windows partition.
The problem I had was I couldn't get the image to be any smaller than the full 750G of the disk, piping through gzip.
I'm curious if the Windows partition doesn't present any redundancy to the Linux OS to take advantage of. Does anyone know? Does NTFS do some 'clever' Windows thing where the disk is actually discontinuous outside of Windows?

i know this is an old post but block sizes also use that exact amount of memory during the dd process.

I use 5G block sizes and you must have more that 5G of Ram available etc

this speeds up the process greatly. like 10x

Excellent Blog. Thank you so much for taking your time to write this.

Hello everybody,

I read this page twice before I started using dd.
I have a HDD with 2 partitions: Windows 7 (NTFS format) and CentOS6.3 (ext4 format) and I backup all the disk with this command:
"dd if=/dev/sda conv=sync,noerror bs=64K | gzip -c > /*/sda.img.gz"
and after the backup was finished, I got no errors.

I restored the image on another SSD and I got this error/worning:
{{{
dd: writing to `/dev/sda': No space left on device
234441649+0 records in
234441648+0 records out
120034123776 bytes (120 GB) copied, 2192.83 s, 54.7 MB/s
}}}

The SSD's type are the same, exactly the same.
The information on the SSD is around 20GB, and on the first SSD, that one which I backuped, has 5GB free space.

I used GParted soft for checking the SSD whit a Clonezilla-SysRescCD.
For ext4 partition I got:
{
e2label no such file or directory while trying to open /dev/sda3
Couldn't find valid filesystem superblock.

Couldn't find valid filesystem superblock.

dumpe2fs 1.41.10 (10-Feb-2009)
dumpe2fs: No such file or
directory while trying to open /dev/md126p1

Unable to read the contents of this file system!
Because of this some operations may be unavailable.

The following list of software packages is required for ext3 file system support: e2fsprogs.

}

For NTFS partition I got:
{
The device doesn't exist.
ntfsresize v2.0.0 (libntfs 10:0:0)
Error(2):Failed to check /dev/sda/ mount state: No such file or directory
Probably /etc/mtab is missing. It is too risky to continue. You might try another Linux distro.
Unable to read the file system contents of this file system. Because of this some operations may be unavailable. The causer might might a missing software package. The following list of support packages is required for ntfs file system support.
ntfsprogs /ntfs-3g
}

Do you think this is because I am using an old distribution of SysRescCD?

Do you have any idea what is the problem?

All the best,
Remus.

Hi there,

just wanted to draw your attention to the fact that zipping the image is quite demanding with respect to cpu power. If you have enough space its better to avoid this - in my case using compression was around 15 times slower....

this would work like:

# dd if=/dev/hda conv=sync,noerror bs=64K of=/mnt/sda1/hda.img

But if you do this over slow connections or have not much space for an image its of course fine to compress...

Hi, just want to do a miniature correction. Making the file delete.me in /tmp is a very bad idea.
/tmp is actually of the tmpfs kind. Meaning that the /tmp partition is actually your RAM.

Essentially, you'll be filling up your ram (let's agree that's never a good idea). And will wind up with none of it usable.

On top of that, it's quite possible you have other partitions!

So essentially, one would do
sudo mount | grep /dev/hda to find where the partition in question is mounted, then in that location... (let's assume it's /home/mystorage)
sudo dd if=/dev/zero of=/home/mystorage/delete.me && sudo rm /home/mystorage/delete.me and not touch that specific partition for a while.

First of all, thanks for the article.

Now about the disadvantages of brute imaging by dd. I'd actually consider it an advantage, cause commercial imaging software seems to think too high of its own intelligence and won't give an option of making a "dumb" image of a partition if it "knows" this FS. There are numerous occasions when I'd like to preserve everything verbatim, including errors. Well, come to think of it, there was Norton Ghost 8 bootable that had an option "no compression". I think that implied full dumb copy without smarting off, but I may be wrong. Thank god we have dd in Linux, cause I don't know how to deal with this in the Windows world (which I do have to deal with).

Incidentally, if someone of you knows an _obedient_ commercial imaging tool with this kind of control (preferably with GUI) available on boot media, I would very much appreciate your sharing such info.

Thanks.

I used this cat /dev/sda | gzip -9 > hdbackup.img.gz
in a shell

and want to do this too

dd if=/dev/zero of=/tmp/delete.me bs=8M; rm delete.me
and I dont know for sure what to do (do you use mount hdbackup.img.gz /image -o loop)

You can also use the cat command and pipe it through gzip with the "-9" option (for maximum compression) and then redirect the output to your image file:

cat /dev/sda | gzip -9 > hdbackup.img.gz

It works for me, but I have only been using Linux for less than a year (I like it better than windows now, witch I have been using all my life). So don't tell me if you have any problems with it...

-- A 13 year old anonymous Linux geek

I'm trying to restore an image back to the original disk, and unfortunately it's not working. It's a dual boot box, XP and Gentoo, and what I get after the restore is a bogus partition table according to my boot disk. I can see two partitions, the windows and extended like so:

*************************************************************************
fdisk -l /dev/sdc
Warning: invalid flag 0x0000 of partition table 5 will be corrected by w(rite)

Disk /dev/sdc: 160.0 GB, 160041885696 bytes
129 heads, 4 sectors/track, 605778 cylinders
Units = cylinders of 516 * 512 = 264192 bytes
Disk identifier: 0x8ff57ec4

Device Boot Start End Blocks Id System
/dev/sdc1 * 1 406425 104857648 7 HPFS/NTFS
/dev/sdc2 406426 605778 51433074 5 Extended

*********************************************************

Here's what I actually backed up:

rabbit scoutBackup # cat scoutHdFdisk.info

Disk /dev/sdc: 160.0 GB, 160041885696 bytes
129 heads, 4 sectors/track, 605778 cylinders
Units = cylinders of 516 * 512 = 264192 bytes
Disk identifier: 0x8ff57ec4

Device Boot Start End Blocks Id System
/dev/sdc1 * 1 406425 104857648 7 HPFS/NTFS
/dev/sdc2 406426 605778 51433074 5 Extended
/dev/sdc5 * 406426 406804 97780 83 Linux
/dev/sdc6 406805 414556 2000014 83 Linux
/dev/sdc7 414557 605778 49335274 83 Linux
rabbit scoutBackup #

Looks like it has something to do with restoring the extended partitions. Any tips on how to fix this?

hello , i am new to linux. i was woudering why we are using bs=64K not 4K (4096/1024) or somthing like that.
plz explain. thx for the article.

Thanks for the reminder that you can use dd to do a disk backup / restore. Sometimes we forget that the simple low-level tools are still there.

One other alternative I wanted to mention is clonezilla (http://clonezilla.org/). The documentation is a bit lacking but it's a very powerful tool that duplicates and extends the norton ghost functionality. In the past I've set up a PXE-boot clonezilla config so you can remote-boot a machine and clone the OS directly onto it over the network without using any removable media.

Just want to say this post was a great help, I also found another post at this address http://rob-ward.co.uk/blog/ddbackup.php that has some useful information that someone might find useful, it explains how to split images up to fit on a disk which is really nice and also gives examples of how to backup DVD's and stuff with it.

And again great post, saved me loads of time :-)

the method for restoring the hdd in the article doesn't work for me.

although my source image was correct, it wasn't transferred properly to the hdd. i inspected the two images, and realized that all the data was being transferred, but it was being regularly padded with a chunk of zeros every so often. Here's what I think is happening: either the pipe fills up, gzip runs too slow, or (since i'm transferring the image over the network) the transfer time is too slow. This means dd will sometimes get a chunk of data less than the 64K blocksize, and since sync is enabled, it will pad the rest of the 64K with zeros. bad!

i would recommend just leaving off sync and noerror (since they have to do with read errors, which you really shouldn't get because you're reading the image)

has anyone else run into this problem? what are your thoughts?

I just tried a backup and restore using your instructions. I was getting a "no space left on device" error on restore and found this to work instead:
# gzip -cd /mnt/sda1/hda.img.gz | dd of=/dev/hda

Thanks for the article, works great apart from that!

If the sector size of partitions are known, you can extract partition images from the main image via:

Outputs as:

Disk /dev/sda: 60.0 GB, 60011642880 bytes
255 heads, 63 sectors/track, 7296 cylinders, total 117210240 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63    16771859     8385898+  8e  Linux LVM
/dev/sda2        16771860    75489434    29358787+  8e  Linux LVM
/dev/sda3        75489435    75698279      104422+  83  Linux
/dev/sda4        75698280   117210239    20755980    5  Extended
/dev/sda5        75698343   117210239    20755948+  8e  Linux LVM

So to create an image of the second partition (sda2) from existing image of sda...

Copy /dev/sda to /dev/sdb:

- /dev/sda: Source disk
- /dev/sdb: Target disk
- sync: Use synchronized I/O for data and metadata
- noerror: Continue copy operation after read errors

# dd if=/dev/hda of=/mnt/sda1/mbr.img bs=512 count=1

This stores the first 512 bytes of the disk (contianing the MBR and the primary partition info - i.e. the first four primary entries) into the file "mbr.img".

To restore (be careful - this could destroy your existing partition table and with it access to all data on the disk):

# dd if=/mnt/sda1/mbr.img of=/dev/hda

If you only want to restore the actual MBR code and not the primary partition table entries, just restore the first 448 bytes of the MBR:

# dd of=/dev/hda if=/mnt/sda1/mbr.img bs=448 count=1.