Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Get md5sum in pipe

I am creating archives of very large directories and splitting these archives in smaller parts as follows:

tar -vcz target_dir | pigz > target_dir.tar.gz

md5sum target_dir.tar.gz > md5sum.txt

split -n 10 target_dir.tar.gz target_dir.tar.gz.part-

The problem is with this approach that I basically need twice the space of the tar.gz file, which is problematic as some of the target directories are huge (TBs).

I could pipe the tar output into split to reduce the required disk space:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

tar -vcz target_dir | pigz | split -n 10 - target_dir.tar.gz.part-

But how would I calculate the md5sum of the tar.gz file before it goes into split?

>Solution :

Use tee to split a stream. Use bash process substitution to run a temporary process with input from a temporary fifo.

tar -vcz target_dir |
    pigz |
    tee >(md5sum > md5sum.txt) |
    split -n 10 - target_dir.tar.gz.part-
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading