Ubuntu: Merging 2 csv files


I have two files:


12,34,56   11,23,67  


12,34,98  11,23,43  

I want to merge them into final.csv:

12,34,56,98  11,23,67,43   

and file size is 10gb

I have used csvtool but it doesn't work when file size is too big


The same results can also be obtained with bash commands.

paste a.csv b.csv |awk -F",||\t" '{print $1","$2","$3","$6}'  

The code gives following output:

12,34,56,98  11,23,67,43  


The perl script below reads the two files simultaneously and joins each line from file a.csv to last item in b.csv.Files are given as arguments on the command-line. Order of the files is important.

#!/usr/bin/env perl  use strict;  use warnings;    open(my $fh1,'<',$ARGV[0]) or die $!;  open(my $fh2,'<',$ARGV[1]) or die $!;    while( not eof($fh1) and not eof($fh2)){      my $line1 = <$fh1>;      my $line2 = <$fh2>;        chomp $line1;      chomp $line2;        my @words2 = split(",",$line2);      printf("%s,%s\n",$line1,$words2[-1]);    }  

Test run:

$ cat a.txt  12,34,56  11,23,67  $ cat b.txt  12,34,98  11,23,43  $ ./merge_files.pl  a.txt b.txt                                                                   12,34,56,98  11,23,67,43  

To send data to new file use shell's redirection operator > like so:

./merge_files.pl  a.txt b.txt  > final.txt  

Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Next Post »