我想以编程方式下载一些文件,但MemoryError
较大的文件却出现异常。例如,当我尝试下载一个较小的文件时,代码很好,但是当我尝试下载较大的文件时,却捕获了一个MemoryError
。
这是我的代码:
def __download_gpl_file(accession):
try:
bin_string = __get_response(accession)
if bin_string is None:
return False
string = __unzip(bin_string)
except MemoryError:
print 'Out of memory for: ' + accession
return False
if string:
filename = DOWNLOADED + accession + '.txt'
with open(filename, 'w+') as f:
f.write(string)
return True
return False
def __get_response(attempts=5):
url = __construct_gpl_url(accession) # Not shown
response = None
while attempts > 0:
try:
response = urllib2.urlopen(url)
if response and response.getcode() < 201:
break
else:
attempts -= 1
except urllib2.URLError:
print 'URLError with: ' + url
return response.read()
def __unzip(bin_string):
f = StringIO(bin_string)
decompressed = gzip.GzipFile(fileobj=f)
return decompressed.read()
我可以做些什么来下载较大的文件吗?提前致谢。
而不是一次写入整个文件,而是逐行写入:
file = urllib2.urlopen('url')
with open('filename','w') as f:
for x in file:
f.write(x)
如果您想使其更快:
file = urllib2.urlopen('url')
with open('filename','w') as f:
while True:
tmp = file.read(1024)
if not tmp:
break
f.write(tmp)
本文收集自互联网,转载请注明来源。
如有侵权,请联系[email protected] 删除。
我来说两句