Uploading Multiple Web Files to Cloud Storage With Python

Data Science Acolyte

I am trying to upload multiple web files to a storage bucket using python. I have a service account set up to enable the credentials and so it should be working. However, every time I try to run this code I receive this error:

Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/voterfile-oh?projection=noAcl: [email protected] does not have storage.buckets.get access to voterfile-oh.

Can I receive help on this issue? I was not able to gather any helpful information from this question nor did the questioner receive an answer that resolved the issue.

def upload_voterfile(bucket_name):
    from google.cloud import storage
    from google.oauth2.service_account import Credentials
    import os
    import json

    credentials_dict = {credentials}
    with open('credentials.json', 'w') as json_file:
        json.dump(credentials_dict, json_file)

    credentials = Credentials.from_service_account_file('credentials.json')

    client = storage.Client(credentials=credentials, project='oh-data-pipeline')
    bucket = client.get_bucket(bucket_name)
    county_list= ['Adams','Allen', 'Ashland', 'Ashtabula', 'Athens', 'Auglaize', 'Belmont', 'Brown', 'Butler', 'Carroll', 
                  'Champaign', 'Clark', 'Clermont', 'Clinton', 'Columbiana', 'Coshocton', 'Crawford', 'Cuyahoga', 'Darke', 
                  'Defiance', 'Delaware', 'Erie', 'Fairfield', 'Fayette', 'Franklin', 'Fulton', 'Gallia', 'Geauga', 'Greene', 
                  'Guernsey', 'Hamilton', 'Hancock', 'Hardin', 'Harrison', 'Henry', 'Highland', 'Hocking', 'Holmes', 
                  'Huron', 'Jackson', 'Jefferson', 'Knox', 'Lake', 'Lawrence', 'Licking', 'Logan', 'Lorain', 'Lucas', 'Madison',
                  'Mahoning', 'Marion', 'Medina', 'Meigs', 'Mercer', 'Miami', 'Monroe', 'Montgomery', 'Morgan', 'Morrow', 
                  'Muskingum', 'Noble', 'Ottawa', 'Paulding', 'Perry', 'Pickaway', 'Pike', 'Portage', 'Preble', 'Putnam', 
                  'Richland', 'Ross', 'Sandusky', 'Scioto', 'Seneca', 'Shelby', 'Stark', 'Summit', 'Trumbull', 'Tuscarawas', 
                  'Union', 'Van Wert', 'Vinton', 'Warren', 'Washington', 'Wayne', 'Williams', 'Wood', 'Wyandot']

    for f, g in zip(range(1, 89), county_list):
        file = urllib.request.urlopen('https://www6.sos.state.oh.us/ords/f?p=VOTERFTP:DOWNLOAD::FILE:NO:2:P2_PRODUCT_NUMBER:{}'.format(f))
        blob = bucket.blob('{}'.format(g))
        blob.upload_from_string(data=file.read(), content_type="text/plain")

if __name__ == "__main__":

    upload_voterfile(bucket_name='voterfile-oh')
Happy-Monad

I have been able to reproduce your issue, here are the steps I have followed:

  1. Created a service account and assign Storage Object Creator permission.
  2. Ran gcloud iam service-accounts keys create [FILE_NAME].json --iam-account [NAME]@[PROJECT_ID].iam.gserviceaccount.com to get account credentials file.
  3. Erased lines 7 to 9 of your script and changed the filename of line 11 to match the name of the downloaded credentials file.
  4. Ran the script. Here I get the same error as you.

The reason behind this error is that on line 14 you're getting your bucket object through get_bucket method. This method queries cloud Storage, requiring get permissions on your bucket but the Storage Object Creator role does not include get permissions.

To solve your issue just change line 14 with this code bucket = client.bucket(bucket_name) which directly creates a bucket object without interacting with Cloud Storage and therefore it does not raise the permission error, see corresponding reference.

Another solution might be to change the service account role to Storage Object Admin because it includes get permissions.

이 기사는 인터넷에서 수집됩니다. 재 인쇄 할 때 출처를 알려주십시오.

침해가 발생한 경우 연락 주시기 바랍니다[email protected] 삭제

에서 수정
0

몇 마디 만하겠습니다

0리뷰
로그인참여 후 검토

관련 기사

분류에서Dev

Uploading multiple files with iOS SDK and deleting them locally when done

분류에서Dev

Drupal uploading files to server

분류에서Dev

CodeIgniter uploading files no file

분류에서Dev

CodeIgniter uploading files

분류에서Dev

What is the benefit of using google cloud storage instead of google cloud sql for saving files

분류에서Dev

using python tool on multiple files

분류에서Dev

Apahce Website - Uploading files and download

분류에서Dev

In your web app, when uploading files to S3, is it safe to upload them directly to S3 from the client side?

분류에서Dev

Google Appengine Cloud Storage

분류에서Dev

Primefaces 5 FileUploadEvent, not all files are uploading

분류에서Dev

Send files with phpmailer before uploading them?

분류에서Dev

Uploading/Deleting files from a database in Django

분류에서Dev

Google Cloud 함수 (Python)에서 Google Cloud Storage 버킷에 새 파일 쓰기

분류에서Dev

Google Cloud Storage와 Google Cloud CDN

분류에서Dev

Google Cloud Storage create_upload_url-App Engine Flexible Python

분류에서Dev

iOS 용 Google Cloud Storage

분류에서Dev

Architecture for uploading Images to Azure Blob Storage without Mobile Services

분류에서Dev

How to create a generator of lines from multiple files in Python

분류에서Dev

Generate function calls tree from multiple Python files

분류에서Dev

Python을 사용하여 Cloud Storage에 여러 웹 파일 업로드

분류에서Dev

Beam Python SDK에서 모든 Cloud Storage 파일의 크기 재 계산 방지

분류에서Dev

파일의 공개 URL 가져 오기-Google Cloud Storage-App Engine (Python)

분류에서Dev

GAE Python이 Cloud Storage에 파일을 제대로 쓸 수 없습니다.

분류에서Dev

Google Cloud Storage에 게시 할 때 $ _FILES 변수가 비어있는 이유는 무엇인가요?

분류에서Dev

Python 스크립트에서 Google Cloud Functions를 사용하여 Google Cloud Storage에서 CSV 읽기

분류에서Dev

Google Cloud Storage: Download partial audio file

분류에서Dev

Google cloud storage client api not working for patch

분류에서Dev

Google Cloud Storage - Knowing who uploaded

분류에서Dev

How to upload milions object to google cloud storage

Related 관련 기사

  1. 1

    Uploading multiple files with iOS SDK and deleting them locally when done

  2. 2

    Drupal uploading files to server

  3. 3

    CodeIgniter uploading files no file

  4. 4

    CodeIgniter uploading files

  5. 5

    What is the benefit of using google cloud storage instead of google cloud sql for saving files

  6. 6

    using python tool on multiple files

  7. 7

    Apahce Website - Uploading files and download

  8. 8

    In your web app, when uploading files to S3, is it safe to upload them directly to S3 from the client side?

  9. 9

    Google Appengine Cloud Storage

  10. 10

    Primefaces 5 FileUploadEvent, not all files are uploading

  11. 11

    Send files with phpmailer before uploading them?

  12. 12

    Uploading/Deleting files from a database in Django

  13. 13

    Google Cloud 함수 (Python)에서 Google Cloud Storage 버킷에 새 파일 쓰기

  14. 14

    Google Cloud Storage와 Google Cloud CDN

  15. 15

    Google Cloud Storage create_upload_url-App Engine Flexible Python

  16. 16

    iOS 용 Google Cloud Storage

  17. 17

    Architecture for uploading Images to Azure Blob Storage without Mobile Services

  18. 18

    How to create a generator of lines from multiple files in Python

  19. 19

    Generate function calls tree from multiple Python files

  20. 20

    Python을 사용하여 Cloud Storage에 여러 웹 파일 업로드

  21. 21

    Beam Python SDK에서 모든 Cloud Storage 파일의 크기 재 계산 방지

  22. 22

    파일의 공개 URL 가져 오기-Google Cloud Storage-App Engine (Python)

  23. 23

    GAE Python이 Cloud Storage에 파일을 제대로 쓸 수 없습니다.

  24. 24

    Google Cloud Storage에 게시 할 때 $ _FILES 변수가 비어있는 이유는 무엇인가요?

  25. 25

    Python 스크립트에서 Google Cloud Functions를 사용하여 Google Cloud Storage에서 CSV 읽기

  26. 26

    Google Cloud Storage: Download partial audio file

  27. 27

    Google cloud storage client api not working for patch

  28. 28

    Google Cloud Storage - Knowing who uploaded

  29. 29

    How to upload milions object to google cloud storage

뜨겁다태그

보관