Ubuntu: How do you monitor the progress of dd?



Question:

dd is a wonder. It lets you duplicate a hard drive to another, completely zero a hard drive, etc. But once you launch a dd command, there's nothing to tell you of its progress. It just sits there at the cursor until the command finally finishes. So how does one monitor dd's progress?


Solution:1

Update 2016: If you use GNU coreutils >= 8.24 (default in Ubuntu Xenial 16.04 upwards), see method 2 below for an alternate way to display the progress.


Method 1: By using pv

Install pv and put it between input / output only dd commands.

Note: you cannot use it when you already started dd.

From the package description:

pv - Pipe Viewer - is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

Installation

sudo apt-get install pv  

Example

dd if=/dev/urandom | pv | dd of=/dev/null  

Output

1,74MB 0:00:09 [ 198kB/s] [      <=>                               ]  

You could specify the approximate size with the --size if you want a time estimation.


Example Assuming a 2GB disk being copied from /dev/sdb

Command without pv would be:

sudo dd if=/dev/sdb of=DriveCopy1.dd bs=4096  

Command with pv:

sudo dd if=/dev/sdb | pv -s 2G | dd of=DriveCopy1.dd bs=4096  

Output:

440MB 0:00:38 [11.6MB/s] [======>                             ] 21% ETA 0:02:19  

Other uses

You can of course use pv directly to pipe the output to stdout:

pv /home/user/bigfile.iso | md5sum  

Output

50,2MB 0:00:06 [8,66MB/s] [=======>         ] 49% ETA 0:00:06  

Note that in this case, pv recognizes the size automatically.


Method 2: New status option added to dd (GNU Coreutils 8.24+)

dd in GNU Coreutils 8.24+ (Ubuntu 16.04 and newer) got a new status option to display the progress:

Example

dd if=/dev/urandom of=/dev/null status=progress  

Output

462858752 bytes (463 MB, 441 MiB) copied, 38 s, 12,2 MB/s  


Solution:2

From HowTo: Monitor the progress of dd

You can monitor the progress of dd without halting it by using the kill command.

To see the progress of dd once it's running, open another terminal and enter:

sudo kill -USR1 $(pgrep ^dd)  

This will display dd progress in the dd terminal window without halting the process. If you're on BSD or OS X, use INFO instead of USR1. The USR1 signal will terminate dd.

If you would like to get regular updates of the dd progress, then enter:

watch -n5 'sudo kill -USR1 $(pgrep ^dd)'  

watch will probe the dd process every -n seconds (-n5 = 5 seconds) and report without halting it.

Note the proper single quotes in the commands above.


Solution:3

A few handy sample usages with pv and less typing or more progress then other answers:

First you will need to install pv, with the command:

sudo apt-get install pv  

Then some examples are:

pv -n /dev/urandom | dd of=/dev/null  pv -tpreb source.iso | dd of=/dev/BLABLA bs=4096 conv=notrunc,noerror  

Note: the first sample is 5 characters less typing then dd if=/dev/urandom | pv | dd of=/dev/null.

And my favorite for cloning a disk drive:

(pv -n /dev/sda | dd of=/dev/sdb bs=128M conv=notrunc,noerror) 2>&1 | dialog --gauge "Running dd command (cloning), please wait..." 10 70 0  

http://s0.cyberciti.org/uploads/faq/2012/10/check-status-of-dd-in-progress.png

source: http://www.cyberciti.biz/faq/linux-unix-dd-command-show-progress-while-coping/

Also for archiving myself.


Solution:4

Use Ctrl+T while dd is running, and it will output the progress (in bytes):

load: 1.51  cmd: dd 31215 uninterruptible 0.28u 3.67s  321121+0 records in  321120+0 records out  164413440 bytes transferred in 112.708791 secs (1458745 bytes/sec)  


Solution:5

The best is using http://dcfldd.sourceforge.net/ it is easy to install through apt-get


Solution:6

For the sake of completeness:

Version 8.24 of the GNU coreutils includes a patch for dd introducing a parameter to print the progress.

The commit introducing this change has the comment:

dd: new status=progress level to print stats periodically

Many distributions, including Ubuntu 16.04.2 LTS use this version.


Solution:7

Native progress status was added to dd!!!

The new version of Coreutils (8.24) adds a progress status to the dd tool:

Usage on Xubuntu 15.10:

Open a terminal and type these commands:

wget ftp://ftp.gnu.org/pub/gnu/coreutils/coreutils-8.24.tar.xz  tar -xf coreutils-8.24.tar.xz  cd coreutils-8.24  ./configure && make -j $(nproc)  

Run dd as root:

sudo su  cd src  ./dd if=/dev/sdc of=/dev/sda conv=noerror status=progress  

You will see: Bytes, seconds and speed (Bytes/second).

To check the versions of dd:

Native:

dd --version  

New:

cd coreutils-8.24/src  ./dd --version  


Solution:8

If you have already started dd, and if you are writing a file such as when creating a copy of a pendrive to disk, you can use the watch command to constantly observe the size of the output file to see changes and estimate completion.

watch ls -l /pathtofile/filename  

To see only file size (h-human view):

watch ls -sh /pathtofile/filename  


Solution:9

The dd | pv | dd triad made my 50GB vm copy take 800 seconds, as opposed to 260 seconds using just dd. With this pipeline, at least, pv has no idea how big the input file is so it won't be able to tell you how far along you are so there's no disadvantage to doing it as follows- and you get a nice speed advantage:

I would avoid pv on anything large, and (if using Bash):

Control-Z the dd process

bg to put it in background. Observe that bg will give you output like [1] 6011 where the latter number is a process id. So, do:

while true; do kill -USR1 process_id ; sleep 5; done

where process_id is the process id you observed. Hit Control-C when you see something like:

[1]+  Done dd if=/path/file.qcow2 of=/dev/kvm/pxetest bs=4194304 conv=sparse  -bash: kill: (60111) - No such process  

You are done.

Edit: Silly Systems Administrator! Automate your life, don't work! If I have a long dd process that I want to monitor, here's a one-liner that will take care of the whole enchilada for you; put this all on one line:

 dd if=/path/to/bigimage of=/path/to/newimage conv=sparse bs=262144 & bgid=$!; while true; do sleep 1; kill -USR1 $bgid || break; sleep 4; done  

You can, of course, script it, perhaps make $1 your input file and $2 your output file. This is left as an exercise for the reader. Note that you need that little sleep before the kill or the kill may die trying to send a signal to dd when it's not ready yet. Adjust your sleeps as desired (maybe even remove the second sleep altogether).

Bash- FTW! :-)


Solution:10

http://linuxcommando.blogspot.com/2008/06/show-progress-during-dd-copy.html

Basically:

kill -USR1 < dd pid >  


Solution:11

On Ubuntu 16.04

Ubuntu 16.04 comes with dd (coreutils) Version 8.25 . Hence the option status=progress is Supported :-)

To use it, just add status=progress along with your dd command.

Example :

dd bs=4M if=/media/severus/tools-soft/OperatingSystems/ubuntu-16.04-desktop-amd64.iso of=/dev/null status=progress && sync  

Gives the status as

1282846183 bytes (1.2 GiB, 1.1 GiB) copied, 14.03 s, 101.9 MB/s  

enter image description here


Solution:12

I have created bash wrapper over dd that will use pv to show progress. Put it into your .bashrc and use dd as usual:

# dd if=/dev/vvg0/root of=/dev/vvg1/root bs=4M      2GB 0:00:17 [ 120MB/s] [===========================================================>] 100%              0+16384 records in  0+16384 records out  2147483648 bytes (2.1 GB) copied, 18.3353 s, 117 MB/s  

Source:

dd()  {      local dd=$(which dd); [ "$dd" ] || {          echo "'dd' is not installed!" >&2          return 1      }        local pv=$(which pv); [ "$pv" ] || {          echo "'pv' is not installed!" >&2          "$dd" "$@"          return $?      }        local arg arg2 infile      local -a args      for arg in "$@"      do          arg2=${arg#if=}          if [ "$arg2" != "$arg" ]          then              infile=$arg2          else              args[${#args[@]}]=$arg          fi      done        "$pv" -tpreb "$infile" | "$dd" "${args[@]}"  }  


Solution:13

So today I got a little frustrated with trying to run kill in a loop while dd was running, and came up with this method for running them in parallel, easily:

function vdd {      sudo dd "$@" &      sudo sh -c "while pkill -10 ^dd$; do sleep 5; done"  }  

Now just use vdd anywhere you'd normally use dd (it passes all arguments directly through) and you'll get a progress report printed every 5s.

The only downside is that the command doesn't return immediately when dd completes; so it's possible that this command can keep you waiting an extra 5s after dd returns before it notices and exits.


Solution:14

As mentioned above, at least with the 'dd' from GNU coreutils, or busybox, it will respond to a USR1 signal by printing progress info to stderr.

I wrote a little wrapper script for dd that shows a nice percent-complete indicator, and tries to not interfere with dd's process or way of functioning in any way. You can find it on github:

http://github.com/delt01/dd_printpercent

Unfortunately, this SIGUSR1 trick only works with either GNU dd (from the coreutils package) or busybox's 'dd' mode with that specific feature enabled at compile time. It doesn't work with the stock 'dd' included with most BSD systems, including FreeBSD and OS X ... :(


Solution:15

I really like ddrescue, it works as dd but gives output and doesn't fail on errors, on the contrary it has a very advanced algorithm an tries really hard to do a successful copy... There are also many GUIs for it

Project: https://www.gnu.org/software/ddrescue

Wikipedia: https://en.wikipedia.org/wiki/Ddrescue

enter image description here


Solution:16

Easiest is:

 dd if=... of=... bs=4M status=progress oflag=dsync  

oflag=dsync will keep your writing in sync, so information of status=progress is more accurate. However it might be a bit slower.


Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »