Essentially, I have a 6.4GB XML file that I'd like to convert to JSON then save it to disk. I'm currently running OSX 10.8.4 with an i7 2700k and 16GBs of ram, and running Python 64bit (double checked). I'm getting an error that I don't have enough memory to allocate. How do I go about fixing this?
print 'Opening'
f = open('large.xml', 'r')
data = f.read()
f.close()
print 'Converting'
newJSON = xmltodict.parse(data)
print 'Json Dumping'
newJSON = json.dumps(newJSON)
print 'Saving'
f = open('newjson.json', 'w')
f.write(newJSON)
f.close()
The Error:
Python(2461) malloc: *** mmap(size=140402048315392) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "/Users/user/Git/Resources/largexml2json.py", line 10, in <module>
data = f.read()
MemoryError
Many Python XML libraries support parsing XML sub elements incrementally, e.g. xml.etree.ElementTree.iterparse and xml.sax.parse in the standard library. These functions are usually called "XML Stream Parser".
The xmltodict library you used also has a streaming mode. I think it may solve your problem
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments