Out of memory exception when using xlsx module with large files

sreepurna

I am trying to get number of rows and columns present in excel sheet

const xlsx = require('xlsx')
let dothis= (path) =>{
    let workbook = xlsx.readFile(path)
    let sheetsList = workbook.SheetNames

    for (let i in sheetsList) {
        var ref = workbook.Sheets[sheetsList[i]]["!ref"];
        var range = xlsx.utils.decode_range(ref);
        console.log(range);
    }
}
dothis('../storage/1million records- Copy.xlsx')

If the file contains less number of records then this is working fine. But if it contains 1million records I am getting the following error:

<--- Last few GCs --->

[10020:00000000004E7540]    88573 ms: Mark-sweep 1110.2 (1341.3) ->
1110.2 (1341.3) MB, 4542.4 / 0.0 ms  allocation failure GC in old space requested [10020:00000000004E7540]    92546 ms: Mark-sweep
1110.2 (1341.3) -> 1110.2 (1326.3) MB, 3973.0 / 0.0 ms  last resort GC in old space requested [10020:00000000004E7540]    97207 ms: Mark-sweep 1110.2 (1326.3) -> 1110.2 (1317.8) MB, 4660.5 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 00000350676A57C1 <JSObject>
    1: parse_ws_xml_data [e:\eg\node_modules\xlsx\xlsx.js:~12315] [pc=00000280234400A9](this=000003CD0C00C211 <JSGlobal Object>,sdata=000002BA7A4A2089 <Very long string[287008182]>,s=000002BA7A4A21B1 <Object map = 00000294315E5519>,opts=000003279F33D969 <Object map = 00000294315E5CA9>,guess=000002BA7A4A2189 <Object map = 00000294315E9401>,themes=000003279F33DA41 <Object map = 00000294315...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory  1: node_module_register  2: v8::internal::FatalProcessOutOfMemory  3: v8::internal::FatalProcessOutOfMemory  4: v8::internal::Factory::NewFixedArray  5: v8::internal::Dictionary<v8::internal::SeededNumberDictionary,v8::internal::SeededNumberDictionaryShape>::AtPut 6: v8::internal::HashTable<v8::internal::SeededNumberDictionary,v8::internal::SeededNumberDictionaryShape>::IsKey 7: v8::internal::HashTable<v8::internal::SeededNumberDictionary,v8::internal::SeededNumberDictionaryShape>::IsKey 8: v8::internal::LookupIterator::ApplyTransitionToDataProperty  9: v8::internal::JSReceiver::class_name 10: v8::internal::SourcePositionTableIterator::Advance 11: v8::internal::SourcePositionTableIterator::Advance 12: 00000280232843C1
Jibesh Patra

You may want to run your script using --max-old-space-size flag and allocate more memory.

node --max-old-space-size=8000 hello.js

will allocate 8GB memory to the current running instance.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Out of Memory Exception when handling large files in C#

From Java

Out of memory exception when decrypt large file using Cipher

From Dev

Download large files using large byte array causes "Out of memory"

From Dev

Compress large file using SharpZipLib causing Out Of Memory Exception

From Dev

Bash script using gzip and bcftools running out of memory with large files

From Dev

Process out of memory error using nodejs when nothing should be that large

From Dev

ADO Large DataSet throws out of memory exception

From Dev

Writing Large File To Disk Out Of Memory Exception

From Dev

Out of memory when creating large number of relationships

From Dev

Rapidminer - Out of memory when working on large datasets

From Dev

Rapidminer - Out of memory when working on large datasets

From Dev

Hibernate out of memory exception while processing large collection of elements

From Dev

C++ allocating large array on heap gives "out of memory exception"

From Dev

C# Large JSON to string causes out of memory exception

From Dev

Entity framework large data set, out of memory exception

From Dev

Hibernate out of memory exception while processing large collection of elements

From Dev

EF - Query Large Data Sets Causes Out Of Memory Exception

From Dev

out of memory when saving a large string data into a text file using open() function

From Dev

"Out of memory" error when using TTask.Run for a large number of job running in parallel

From Dev

Out of memory exception while using threads

From Dev

Out Of Memory Exception - Using SqlDataReader and OpenXML

From Dev

Out of memory exception using PdfSharp in a azure webjob

From Dev

Count number of pages in large set of pdf files : Out of memory

From Java

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory when processing large files with fs

From Dev

Geeting exception finding no. of sheets in xlsx files using Apache poi

From Dev

Extracting ISO files in 7z.dll- Out of Memory Exception

From Dev

Out of memory when loading a large picture (5 MB) or GIFs to WebView

From Dev

Web worker out of memory when processing large array

From Dev

Out of memory when loading a large picture (5 MB) or GIFs to WebView

Related Related

  1. 1

    Out of Memory Exception when handling large files in C#

  2. 2

    Out of memory exception when decrypt large file using Cipher

  3. 3

    Download large files using large byte array causes "Out of memory"

  4. 4

    Compress large file using SharpZipLib causing Out Of Memory Exception

  5. 5

    Bash script using gzip and bcftools running out of memory with large files

  6. 6

    Process out of memory error using nodejs when nothing should be that large

  7. 7

    ADO Large DataSet throws out of memory exception

  8. 8

    Writing Large File To Disk Out Of Memory Exception

  9. 9

    Out of memory when creating large number of relationships

  10. 10

    Rapidminer - Out of memory when working on large datasets

  11. 11

    Rapidminer - Out of memory when working on large datasets

  12. 12

    Hibernate out of memory exception while processing large collection of elements

  13. 13

    C++ allocating large array on heap gives "out of memory exception"

  14. 14

    C# Large JSON to string causes out of memory exception

  15. 15

    Entity framework large data set, out of memory exception

  16. 16

    Hibernate out of memory exception while processing large collection of elements

  17. 17

    EF - Query Large Data Sets Causes Out Of Memory Exception

  18. 18

    out of memory when saving a large string data into a text file using open() function

  19. 19

    "Out of memory" error when using TTask.Run for a large number of job running in parallel

  20. 20

    Out of memory exception while using threads

  21. 21

    Out Of Memory Exception - Using SqlDataReader and OpenXML

  22. 22

    Out of memory exception using PdfSharp in a azure webjob

  23. 23

    Count number of pages in large set of pdf files : Out of memory

  24. 24

    FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory when processing large files with fs

  25. 25

    Geeting exception finding no. of sheets in xlsx files using Apache poi

  26. 26

    Extracting ISO files in 7z.dll- Out of Memory Exception

  27. 27

    Out of memory when loading a large picture (5 MB) or GIFs to WebView

  28. 28

    Web worker out of memory when processing large array

  29. 29

    Out of memory when loading a large picture (5 MB) or GIFs to WebView

HotTag

Archive