I wrote a shell script which reads some files and copy all the data from these files to other consolidated file.
This happens multiple times and previously copied data will not be copied again. For this i am saving the last copied line in a separate file and comparing it from the second time.
Flow of logic :
sed '$d' $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
Full script:
#!/bin/bash
consolidatedCSVFile=$1
endTime=$2
curdate=$(date +%s)
CSVFiles=${@:3}
CSVFiles=${CSVFiles[@]}
lastLines=_LASTLINES
lastLine=_LASTLINE
newMerge=false
# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile ]
if [ $? == 1 ]
then
mkdir $consolidatedCSVFile$lastLines
newMerge=true
fi
testInProcess=false
if [ $endTime -ge $curdate ]
then
testInProcess=true
fi
for file in $CSVFiles
do
if $newMerge ;
then
touch $consolidatedCSVFile$1astLines5(basename $file)$lastLine
sed $d $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
availableLastLine="`cat $consolidatedCSVFile$1astLines/$(basename $file)$lastLine`"
if $testInProcess
then
sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
fi
fi
done
if ! $testInProcess ;
then
rm -rf $consolidatedCSVFile$lastLines
fi
Sometimes, i get error as sed: -e expression #1, char 26: unknown command: 'X'
My guess
I am trying to combine the files generated by JMeter. Sometimes, the files are generating data like 1439459065780,5,SOAP/XML-RPC Request,200,OK,ws test 1-3,text,true,267,3,3,5
I think the problem is while sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
at SOAP/XML
. Slash(/) could be problem here.
worked after modifying availableLastLine
to availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine)
.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments