How to correctly fetch data from API endpoint with multiple parameter in python?

kim

I tried to fetch data from API endpoints which has multiple parameters and build big pandas dataframe from it. In my attempt, I passed the root URL and iterate the thought API endpoint's multiple parameters with a customized values list. My goal is to fetch the data from specific API endpoints but passing different values to its multiple values and finally generate one pandas dataframe. Currently, my code returned me an empty dataframe. I am afraid I might do something wrong in my attempt. Can anybody point me out what went wrong in my code? Does anyone suggest a possible way of doing this in pandas? Any thoughts?

my current attempt

import pandas as pd
import json, requests

commodityCodes = [101, 102, 103, 104, 105]
countyCodes = [1220, 2010, 2050, 2110, 3330, 3370, 5210, 5460, 5490, 5520]
marketyear= list(range(2010,2020))

api_key = 'ec95a478-e46e-47f9-b57d-3d19012d527d'
rooturl = 'https://apps.fas.usda.gov/OpenData/api/esr/exports/'
headers = {'API_KEY': '{key}'.format(key=api_key)}

finaldf=pd.DataFrame()

for cc1 in commodityCodes:
    for cc2 in countyCodes:
        for year in marketyear:
            jsonData = requests.get(rooturl+'commodityCode/{}'.format(cc1)+ '/countryCode/{}'.format(cc2)+'/marketyear/{}'.format(year), headers=headers).json()
            df= pd.read_json(json.dumps(jsonData))
    finaldf.append(df)

but my above code returned me an empty dataframe. Maybe I used a nested loop to do this and that might cause the problem. Also running the above code is takes some minutes.

What would be an efficient way of doing this? Where was I wrong in my attempt? How can I correctly fetch data from API endpoints? Can anyone suggest a possible way of doing this? Any ideas?

Maximilian Peters

Two issues:

  1. Pandas' append returns a new object, it doesn't modify the original dataframe
  2. you append in the outer loop, not in the inner loop

finaldf = pd.DataFrame()

for cc1 in commodityCodes:
    for cc2 in countyCodes:
        for year in marketyear:
            jsonData = requests.get(rooturl+'commodityCode/{}'.format(cc1)+ '/countryCode/{}'.format(cc2)+'/marketyear/{}'.format(year), headers=headers).json()
            df = pd.read_json(json.dumps(jsonData))
            finaldf = finaldf.append(df)

Suggestion

Build your url in one step, it's more readable

rooturl = 'https://apps.fas.usda.gov/OpenData/api/esr/exports/commodityCode/{}/countryCode/{}/marketyear/{}'
url = rooturl.format(cc1, cc2, year)

この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。

侵害の場合は、連絡してください[email protected]

編集
0

コメントを追加

0

関連記事

分類Dev

How to correctly fetch data from API endpoint with multiple parameter in python?

分類Dev

How to fetch specific data from hash if there are multiple matches for the key?

分類Dev

Fetch data from local API

分類Dev

How to make fetch data from web api Angular 6

分類Dev

JS fetch API: How to fetch content from multiple files with one async function?

分類Dev

How to fetch data dynamically from an xml file using python?

分類Dev

fetch data using multiple ids from firebase

分類Dev

How to set up API server for multiple fetch requests of SQL Server data?

分類Dev

Apiary API endpoint returning null data when called from web

分類Dev

how to send data received from laravel controller to another page through ajax fetch api response

分類Dev

How to fetch data from array and insert to database

分類Dev

ROR: How to fetch data from View to Controller

分類Dev

How to fetch multiple twitter user's feed in API 1.1

分類Dev

how to get fetch data from sqlite from the given string?

分類Dev

How to fetch multiple column from mysql using go lang

分類Dev

Spring Boot accessing MongoDB - multiple records/documents update from rest api endpoint

分類Dev

How to create a multiple options from a single data coming from the API for displaying it as a select/dropdown in VueJS

分類Dev

How to get the value from previous value in json using fetch api?

分類Dev

How to get distinct values from an array of fetch API output

分類Dev

How to fetch the latest log events from the CloudWatch API?

分類Dev

How to use react native fetch() to retrieve malformed json from api

分類Dev

How to fetch data from joined tables straight to a class?

分類Dev

How to fetch or get data from Cloud Firestore Firebase in android

分類Dev

How to fetch partial data from a large yaml file?

分類Dev

How to fetch data from table and compare it and display the result in laravel?

分類Dev

Active Record: How to fetch data from 3 tables at once?

分類Dev

How to do pagination of data fetch from database in laravel in the backend

分類Dev

How to fetch data from one to many relationship using Django queries

分類Dev

Angular: How to use a Service to fetch data from server and store in an array?

Related 関連記事

  1. 1

    How to correctly fetch data from API endpoint with multiple parameter in python?

  2. 2

    How to fetch specific data from hash if there are multiple matches for the key?

  3. 3

    Fetch data from local API

  4. 4

    How to make fetch data from web api Angular 6

  5. 5

    JS fetch API: How to fetch content from multiple files with one async function?

  6. 6

    How to fetch data dynamically from an xml file using python?

  7. 7

    fetch data using multiple ids from firebase

  8. 8

    How to set up API server for multiple fetch requests of SQL Server data?

  9. 9

    Apiary API endpoint returning null data when called from web

  10. 10

    how to send data received from laravel controller to another page through ajax fetch api response

  11. 11

    How to fetch data from array and insert to database

  12. 12

    ROR: How to fetch data from View to Controller

  13. 13

    How to fetch multiple twitter user's feed in API 1.1

  14. 14

    how to get fetch data from sqlite from the given string?

  15. 15

    How to fetch multiple column from mysql using go lang

  16. 16

    Spring Boot accessing MongoDB - multiple records/documents update from rest api endpoint

  17. 17

    How to create a multiple options from a single data coming from the API for displaying it as a select/dropdown in VueJS

  18. 18

    How to get the value from previous value in json using fetch api?

  19. 19

    How to get distinct values from an array of fetch API output

  20. 20

    How to fetch the latest log events from the CloudWatch API?

  21. 21

    How to use react native fetch() to retrieve malformed json from api

  22. 22

    How to fetch data from joined tables straight to a class?

  23. 23

    How to fetch or get data from Cloud Firestore Firebase in android

  24. 24

    How to fetch partial data from a large yaml file?

  25. 25

    How to fetch data from table and compare it and display the result in laravel?

  26. 26

    Active Record: How to fetch data from 3 tables at once?

  27. 27

    How to do pagination of data fetch from database in laravel in the backend

  28. 28

    How to fetch data from one to many relationship using Django queries

  29. 29

    Angular: How to use a Service to fetch data from server and store in an array?

ホットタグ

アーカイブ