I am populating a local database using a third party service. I have a list of urls (around 500). I am calling each url in a loop, and updating my database with the returned data. The code flow looks like this:
for url in urllist:
req = urllib.urlopen(url)
data = json.loads(req.read())
req.close()
#update the db using data here
Whenever I run this piece of code, the script fails at random points with the error message "Name or service not known". This doesn't have anything with the urls because the script fails at random points (i.e. at 50th iteration in one run, and at 60th iteration in another)
What could be the reason for this?
if you use a bad proxy or there are network problems you can try this:
for url in urllist:
retry = 0
while True: # retry request
try:
req = urllib.urlopen(url)
resp_data = req.read() # in call read() network still processing
except Exception as e: # TODO need more detailed handling
if retry > 3: # 3 this is serious problem. exit
raise e
retry += 1 # retry
else:
data = json.loads()
req.close() # not needed
break
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments