Accessing files from Google cloud storage in RStudio

yabtzey

I have been trying to create connection between the Google cloud storage and RStudio server(The one I spinned up in Google cloud), so that I can access the files in R to run sum analysis on. I have found three different ways to do it on the web, but I don't see many clarity around these ways so far.

  1. Access the file by using the public URL specific to the file [This is not an option for me]
  2. Mount the Google cloud storage as a disc in RStudio server and access it like any other files in the server [ I saw someone post about this method but could not find on any guides or materials that shows how it's done]
  3. Using the googleCloudStorageR package to get full access to the Cloud Storage bucket.

The step 3 looks like the pretty standard way to do it. But I get following error when I try to hit the gcs_auth() command

Error in gar_auto_auth(required_scopes, new_user = new_user, no_auto = no_auto, : Cannot authenticate - options(googleAuthR.scopes.selected) needs to be set to includehttps://www.googleapis.com/auth/devstorage.full_control or https://www.googleapis.com/auth/devstorage.read_write or https://www.googleapis.com/auth/cloud-platform

The guide on how to connect using this is found on https://github.com/cloudyr/googleCloudStorageR but it says it requires a service-auth.json file to set the environment variables and all other keys and secret keys, but do not really specify on what these really are.

If someone could help me know how this is actually setup, or point me to a nice guide on setting the environment up, I would be very much grateful.

Thank you.

Dan

You probably want the FUSE adaptor - this will allow you to mount your GCS bucket as a directory on your Server.

  1. Install gcsfuse on the R server.
  2. create a mnt directory.
  3. run gcsfuse your-bucket /path/to/mnt

Be aware though that RW performance isnt great vis FUSE

Full documentation

https://cloud.google.com/storage/docs/gcs-fuse

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Accessing files in the Google Cloud Storage from two different google cloud projects

From Dev

streaming gzipped files from google cloud storage

From Dev

Exception accessing Google Cloud Storage from newly created Compute Engine

From Dev

Accessing google cloud storage bucket from cloud functions throws 500 error

From Dev

How to download files from Google Cloud Storage using curl

From Dev

Access private files from a google cloud storage bucket

From Dev

Using Dask to read parquet files from a google cloud storage

From Dev

Add partition columns of Parquet files from Google Cloud Storage to BigQuery

From Dev

Transfer files from dropbox/drive to Google cloud storage

From Dev

Restore Previous Versions of Files From Google Cloud Storage

From Dev

Loading multiple files from Google Cloud Storage into a single Pandas Dataframe

From Dev

How to remove extension name from multiple files in google cloud storage?

From Dev

Uploading files to Google Cloud Storage from Localhost or external server

From Dev

How to copy files from colab or cloud storage to google drive?

From Dev

Struggling to read csv files from Google Cloud Storage bucket

From Dev

Copy Files from S3 bucket to Google Cloud Storage

From Dev

Downloading Files from Google Cloud Storage to Remote Server

From Dev

Google Cloud Storage - files not showing

From Dev

How to load numpy npz files in google-cloud-ml jobs or from Google Cloud Storage?

From Dev

Accessing and using csv file from Cloud Storage in Cloud Run instance

From Dev

Accessing Mount from Azure Files in Cloud Service

From Dev

Transfer files from Google Cloud Storage to Windows VM instance using Cloud Functions

From Dev

Google Cloud Platform: accumulate data from Pub/Sub to files in Cloud Storage without Dataflow

From Dev

Google Cloud: Auto-deploy static files from Source Repository to Cloud Storage

From Dev

Dowload google cloud storage files from cloud bucket continuously looping when using npm start

From Dev

Accessing google cloud storage using hadoop FileSystem api

From Dev

How to concatenate a million files in Google Cloud Storage

From Dev

Google Cloud Storage Joining multiple csv files

From Dev

Where Are the Files in My Google Cloud Storage Bucket?

Related Related

  1. 1

    Accessing files in the Google Cloud Storage from two different google cloud projects

  2. 2

    streaming gzipped files from google cloud storage

  3. 3

    Exception accessing Google Cloud Storage from newly created Compute Engine

  4. 4

    Accessing google cloud storage bucket from cloud functions throws 500 error

  5. 5

    How to download files from Google Cloud Storage using curl

  6. 6

    Access private files from a google cloud storage bucket

  7. 7

    Using Dask to read parquet files from a google cloud storage

  8. 8

    Add partition columns of Parquet files from Google Cloud Storage to BigQuery

  9. 9

    Transfer files from dropbox/drive to Google cloud storage

  10. 10

    Restore Previous Versions of Files From Google Cloud Storage

  11. 11

    Loading multiple files from Google Cloud Storage into a single Pandas Dataframe

  12. 12

    How to remove extension name from multiple files in google cloud storage?

  13. 13

    Uploading files to Google Cloud Storage from Localhost or external server

  14. 14

    How to copy files from colab or cloud storage to google drive?

  15. 15

    Struggling to read csv files from Google Cloud Storage bucket

  16. 16

    Copy Files from S3 bucket to Google Cloud Storage

  17. 17

    Downloading Files from Google Cloud Storage to Remote Server

  18. 18

    Google Cloud Storage - files not showing

  19. 19

    How to load numpy npz files in google-cloud-ml jobs or from Google Cloud Storage?

  20. 20

    Accessing and using csv file from Cloud Storage in Cloud Run instance

  21. 21

    Accessing Mount from Azure Files in Cloud Service

  22. 22

    Transfer files from Google Cloud Storage to Windows VM instance using Cloud Functions

  23. 23

    Google Cloud Platform: accumulate data from Pub/Sub to files in Cloud Storage without Dataflow

  24. 24

    Google Cloud: Auto-deploy static files from Source Repository to Cloud Storage

  25. 25

    Dowload google cloud storage files from cloud bucket continuously looping when using npm start

  26. 26

    Accessing google cloud storage using hadoop FileSystem api

  27. 27

    How to concatenate a million files in Google Cloud Storage

  28. 28

    Google Cloud Storage Joining multiple csv files

  29. 29

    Where Are the Files in My Google Cloud Storage Bucket?

HotTag

Archive