I tried to fetch data from API endpoints which has multiple parameters and build big pandas dataframe from it. In my attempt, I passed the root URL and iterate the thought API endpoint's multiple parameters with a customized values list. My goal is to fetch the data from specific API endpoints but passing different values to its multiple values and finally generate one pandas dataframe. Currently, my code returned me an empty dataframe. I am afraid I might do something wrong in my attempt. Can anybody point me out what went wrong in my code? Does anyone suggest a possible way of doing this in pandas? Any thoughts?
my current attempt
import pandas as pd
import json, requests
commodityCodes = [101, 102, 103, 104, 105]
countyCodes = [1220, 2010, 2050, 2110, 3330, 3370, 5210, 5460, 5490, 5520]
marketyear= list(range(2010,2020))
api_key = 'ec95a478-e46e-47f9-b57d-3d19012d527d'
rooturl = 'https://apps.fas.usda.gov/OpenData/api/esr/exports/'
headers = {'API_KEY': '{key}'.format(key=api_key)}
finaldf=pd.DataFrame()
for cc1 in commodityCodes:
for cc2 in countyCodes:
for year in marketyear:
jsonData = requests.get(rooturl+'commodityCode/{}'.format(cc1)+ '/countryCode/{}'.format(cc2)+'/marketyear/{}'.format(year), headers=headers).json()
df= pd.read_json(json.dumps(jsonData))
finaldf.append(df)
but my above code returned me an empty dataframe. Maybe I used a nested loop to do this and that might cause the problem. Also running the above code is takes some minutes.
What would be an efficient way of doing this? Where was I wrong in my attempt? How can I correctly fetch data from API endpoints? Can anyone suggest a possible way of doing this? Any ideas?
Two issues:
finaldf = pd.DataFrame()
for cc1 in commodityCodes:
for cc2 in countyCodes:
for year in marketyear:
jsonData = requests.get(rooturl+'commodityCode/{}'.format(cc1)+ '/countryCode/{}'.format(cc2)+'/marketyear/{}'.format(year), headers=headers).json()
df = pd.read_json(json.dumps(jsonData))
finaldf = finaldf.append(df)
Suggestion
Build your url in one step, it's more readable
rooturl = 'https://apps.fas.usda.gov/OpenData/api/esr/exports/commodityCode/{}/countryCode/{}/marketyear/{}'
url = rooturl.format(cc1, cc2, year)
この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。
侵害の場合は、連絡してください[email protected]
コメントを追加