AWK condition failing for a large file but works on a smaller file

kevin mugwe

I have a script that is comparing two files and outputting the values if a condition is met.The script works well if I reduce the records in my second file but fails if I use the actual file with 100000 records. What could I be doing wrong? Here is my code

awk -F '|' 'NR==FNR{a[$1"|"$2]=$3}NR!=FNR{if(exist a[$1"|"$3]) {print "update inventory set amount=" a[$1"|"$3] " where balance_id=" $2 " and balance_type_id=3019;"}}' cash.txt test.txt > update.sql

Cash.txt
0|3019|0
1|3019|1
2|3019|2
3|3019|3
4|3019|4
5|3019|5
6|3019|5

test.txt
0|0|3019
1|1|3019
2|2|3019
3|3|3019
4|4|3019
5|5|3019
6|6|3019
7|7|3019
8|8|3019
9|9|3019
10|10|3019
11|11|3019
12|12|3019



anubhava

You may try this awk:

awk -F '[|\r]' 'NR == FNR {
   map[$1 "|" $2] = $3
   next
}
$1 "|" $3 in map {
   print "update inventory set amount=" map[$1 "|" $3] " where balance_id=" $2 " and balance_type_id=3019;"
}' cash.txt test.txt > update.sql

By not using next in first block your awk is evaluating FNR != NR condition for each record in first file and then for each record of second bigger file.

Also second block can be just $1 "|" $3 in map so that we process only matching records.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Java

How to split a large text file into smaller files with equal number of lines?

From Dev

Apache/PHP large file download (>2Gb) failing

From Dev

AWK split a large file into thousand mini files error

From Dev

time delay after printing lines with sed or awk in large file

From Dev

Can I split a large HAProxy config file into multiple smaller files?

From Dev

How to do multi-row calculations using awk on a large file

From Dev

Split large text file using AWK, given specific parameters

From Dev

optimizing awk command for large file

From Dev

VBScript Split Large excel file into smaller File after every 50,000 rows

From Dev

awk failing to read contents of a large file

From Dev

Split Large XML File Into Smaller Files

From Dev

Symfony2, RestFul API, Error The uploaded file was too large. Please try to upload a smaller file

From Dev

Creating smaller chunks from large file and sort the chunks

From Dev

Split File With positional value condition with awk

From Dev

Produce Large Object file from Smaller source file

From Dev

awk with variables in condition and in output redirection file

From Dev

busybox tar or split large file to multiple smaller files

From Dev

Awk - Separate one .txt file to files by condition

From Dev

Retrieve text from a large file using a smaller matching header

From Dev

awk to consolidate large tabular file?

From Dev

Break a large file into smaller pieces

From Dev

Split a CSV file into smaller files based on some condition

From Dev

Evaluating condition of if statement in awk using a second file

From Dev

Awk programming: Split large file into smaller ones based on pattern

From Dev

Break large file into smaller files

From Dev

Bash split large file of 2 line chunks into smaller files

From Dev

How can I compress a large file into smaller parts?

From Dev

Take smaller file path

From Dev

Can Webpack helps me separate a large single file into smaller pieces?

Related Related

  1. 1

    How to split a large text file into smaller files with equal number of lines?

  2. 2

    Apache/PHP large file download (>2Gb) failing

  3. 3

    AWK split a large file into thousand mini files error

  4. 4

    time delay after printing lines with sed or awk in large file

  5. 5

    Can I split a large HAProxy config file into multiple smaller files?

  6. 6

    How to do multi-row calculations using awk on a large file

  7. 7

    Split large text file using AWK, given specific parameters

  8. 8

    optimizing awk command for large file

  9. 9

    VBScript Split Large excel file into smaller File after every 50,000 rows

  10. 10

    awk failing to read contents of a large file

  11. 11

    Split Large XML File Into Smaller Files

  12. 12

    Symfony2, RestFul API, Error The uploaded file was too large. Please try to upload a smaller file

  13. 13

    Creating smaller chunks from large file and sort the chunks

  14. 14

    Split File With positional value condition with awk

  15. 15

    Produce Large Object file from Smaller source file

  16. 16

    awk with variables in condition and in output redirection file

  17. 17

    busybox tar or split large file to multiple smaller files

  18. 18

    Awk - Separate one .txt file to files by condition

  19. 19

    Retrieve text from a large file using a smaller matching header

  20. 20

    awk to consolidate large tabular file?

  21. 21

    Break a large file into smaller pieces

  22. 22

    Split a CSV file into smaller files based on some condition

  23. 23

    Evaluating condition of if statement in awk using a second file

  24. 24

    Awk programming: Split large file into smaller ones based on pattern

  25. 25

    Break large file into smaller files

  26. 26

    Bash split large file of 2 line chunks into smaller files

  27. 27

    How can I compress a large file into smaller parts?

  28. 28

    Take smaller file path

  29. 29

    Can Webpack helps me separate a large single file into smaller pieces?

HotTag

Archive