I wrote a shell script which reads some files and copy all the data from these files to other consolidated file.
This happens multiple times and previously copied data will not be copied again. For this i am saving the last copied line in a separate file and comparing it from the second time.
Flow of logic :
sed '$d' $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
Full script:
#!/bin/bash
consolidatedCSVFile=$1
endTime=$2
curdate=$(date +%s)
CSVFiles=${@:3}
CSVFiles=${CSVFiles[@]}
lastLines=_LASTLINES
lastLine=_LASTLINE
newMerge=false
# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile ]
if [ $? == 1 ]
then
mkdir $consolidatedCSVFile$lastLines
newMerge=true
fi
testInProcess=false
if [ $endTime -ge $curdate ]
then
testInProcess=true
fi
for file in $CSVFiles
do
if $newMerge ;
then
touch $consolidatedCSVFile$1astLines5(basename $file)$lastLine
sed $d $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
availableLastLine="`cat $consolidatedCSVFile$1astLines/$(basename $file)$lastLine`"
if $testInProcess
then
sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
fi
fi
done
if ! $testInProcess ;
then
rm -rf $consolidatedCSVFile$lastLines
fi
Sometimes, i get error as sed: -e expression #1, char 26: unknown command: 'X'
My guess
I am trying to combine the files generated by JMeter. Sometimes, the files are generating data like 1439459065780,5,SOAP/XML-RPC Request,200,OK,ws test 1-3,text,true,267,3,3,5
I think the problem is while sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
at SOAP/XML
. Slash(/) could be problem here.
worked after modifying availableLastLine
to availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine)
.
Working Script :
#!/bin/bash
consolidatedCSVFile=$1
endTime=$2
curdate=$(date +%s)
CSVFiles=${@:3}
CSVFiles=${CSVFiles[@]}
lastLines=_LASTLINES
lastLine=_LASTLINE
newMerge=false
# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile ]
if [ $? == 1 ]
then
mkdir $consolidatedCSVFile$lastLines
newMerge=true
fi
testInProcess=false
if [ $endTime -ge $curdate ]
then
testInProcess=true
fi
for file in $CSVFiles
do
if $newMerge ;
then
touch $consolidatedCSVFile$1astLines5(basename $file)$lastLine
sed $d $file >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
## Replaced below line
#availableLastLine="`cat $consolidatedCSVFile$1astLines/$(basename $file)$lastLine`"
availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine)
if $testInProcess
then
sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
else
sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
fi
fi
done
if ! $testInProcess ;
then
rm -rf $consolidatedCSVFile$lastLines
fi
Possible error cases handled while merging of multiple files
#!/bin/bash
echo -e "\n******* $(date) *******"
echo "Arguments :: $@"
consolidatedCSVFile=$1
echo "consolidatedCSVFile :: $1"
endTime=$2
curdate=$(date +%s)
echo -e "endTime :: $2\n"
CSVFiles=${@:3}
CSVFiles=${CSVFiles[@]}
echo -e "Individual Files : ${CSVFiles[@]}\n"
lastLines=_LASTLINES
lastLine=_LASTLINE
newMerge=false
# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile ]
if [ $? == 1 ]
then
newMerge=true
fi
echo "newMerge? :: $newMerge"
if $newMerge ;
then
if [ ! -d $consolidatedCSVFile$lastLines ]
then
echo "creating LASTLINES directory"
mkdir $consolidatedCSVFile$lastLines
fi
fi
testInProcess=false
if [ $endTime -ge $curdate ]
then
testInProcess=true
fi
echo -e "testInProcess? :: $testInProcess\n"
for file in $CSVFiles
do
echo -e "\nCurrent file in loop :: $(basename $file)"
if $newMerge ;
then
# Returns 1 if file is empty
[ -s $file ]
if [ $? == 1 ]
then
echo "File empty !!"
continue
fi
echo "creating LASTLINE file"
touch $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
#sed '$d' $file >> $consolidatedCSVFile
sed '1d' $file | sed '$d' >> $consolidatedCSVFile
echo "Last line to save :: $(tail -1 $consolidatedCSVFile)"
tail -1 $consolidatedCSVFile > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
else
# Returns 1 if file is empty
[ -s $file ]
if [ $? == 1 ]
then
echo "File empty !!"
continue
fi
if [ ! -f $consolidatedCSVFile$lastLines/$(basename $file)$lastLine ]
then
echo "creating LASTLINE file"
touch $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
#sed '$d' $file >> $consolidatedCSVFile
sed '1d' $file | sed '$d' >> $consolidatedCSVFile
echo "Last line to save :: $(tail -1 $consolidatedCSVFile)"
tail -1 $consolidatedCSVFile > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
continue
else
availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine)
echo "Last line from file :: $availableLastLine"
fi
if $testInProcess
then
tempVar=`sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d'`
#echo -e "\ntempVar ::\n$tempVar\n"
[ -z "$tempVar" ]
if [ $? == 1 ]
then
#sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
echo -e "$tempVar" >> $consolidatedCSVFile
echo "Last line to save :: $(tail -1 $consolidatedCSVFile)"
tail -1 $consolidatedCSVFile > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine
else
echo "No last line to save"
fi
else
sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
fi
fi
done
if ! $testInProcess ;
then
echo -e "\nRemoving LASTLINES folder"
rm -rf $consolidatedCSVFile$lastLines
fi
echo -e "\n*************"
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments