Estimate Time for Job Completion (With Progress Updates) When Tar’ing Huge Directories

For the sake of future me, I am recording this here, the coolest shell trick I’ve learned this year:

(Linux):

tar cf - /folder-with-big-files -P | pv -s $(du -sb /folder-with-big-files | awk '{print $1}') | gzip > big-files.tar.gz

(OSX):

tar cf - /folder-with-big-files -P | pv -s $(($(du -sk /folder-with-big-files | awk '{print $1}') * 1024)) | gzip > big-files.tar.gz

with output looking like:

4.69GB 0:04:50 [16.3MB/s] [==========================>        ] 78% ETA 0:01:21

Requires ‘pv’: https://github.com/icetee/pv

Reproduced from this Superuser answer here: https://superuser.com/a/665181

Leave a Reply

Your email address will not be published. Required fields are marked *