Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

merge two text file in one without duplicate header in bash

I have two files that looks like this:

file1.txt

ID group gender age
a 1 m 55
b 1 f 54
c 2 m 53
... etc


file2.txt

ID group gender age
l 1 m 64
m 2 f 94
n 2 m 93
... etc

Desired output:

ID group gender age
a 1 m 55
b 1 f 54
c 2 m 53
.
l 1 m 64
m 2 f 94
n 2 m 93
.

Which means to the header only show once.I tried head -n 1 file1.txt > merge.txt; tail -n +2 -q *.txt >> merge.txt, In my real file1.txt has 87394750 row and file2.txt has 369765446 rows,it’s spend a lot of time.Any ideas? Many thanks!

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

>Solution :

Use cp then sed:

cp file1.txt merge.txt; sed 1d file2.txt >> merge.txt

sed 1d skips the first line.

To skip more lines, use sed 1,nd, where n is the total number of lines to skip.

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading