Building Google Cloud Platform Data Catalog on unstructured data

nsk Published at Dev

nsk

I have unstructured data in the form of document images. We are converting these documents to JSON files. I now want to have technical metadata captured for this. Can someone please give me some tips/best practices for building a data catalog on unstructured data in Google Cloud Platform?

mesmacosta

This answer comes with the assumption that you are not using any tool to create schemas around your unstructured data and query your data, like BigQuery, Hive, Presto. And you simply want to catalog your files.

I had a similar use case, Google Data Catalog has an option to create custom entries.

Some tips on building a Data Catalog on unstructured files data:

Use meaningful file names on your JSON files. That way searching for them will become easier.
Since you are already using GCP, use their managed Data Catalog, and leverage their custom entries API to ingest the files metadata into it.
In case you also want to look for sensitive data in your JSON files, you could run DLP on them.
Use Data Catalog Tags to enrich the files metadata. The tutorial on the link shows how to do it on Big Query tables, but you can do the same on custom entries.

I would add some information about your ETL jobs that convert these documents in JSON files as Tags. Like execution time, data quality score, user, business owner, etc.

In case you are wondering how to do the step 2, I put together one script that automatically does that: link for the GitHub. Another option is to work with Data Catalog Filesets.

So between using custom entries or filesets, I'd ask you this, do you need information about your files name?

If not then filesets might easier, since at the time of this writing it does not show any info about your files name, but are good to manage file patterns in GCS buckets: It is defined by one or more file patterns that specify a set of one or more Cloud Storage files.

The datatalog-util also has an option to enrich your filesets, in case you just want to have statistics about them, like average file size, types, etc.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at2020-12-15

Comments

0 comments

From Dev

Related Related

Article

Building Google Cloud Platform Data Catalog on unstructured data

Building Google Cloud Platform Data Catalog on unstructured data

Error Python API GCP Data Catalog - Google Cloud Platform

Google Cloud Platform Data Display

Delta Lake - Building Data Catalog

Not able to read data from Google Cloud Platform in StreamSets Data Collector

Storing images and structured data together (Google Cloud Platform)

How to configure specific data pipeline in Google Cloud Platform?

Read unstructured data pandas

Unlabeled vs unstructured data

Importing unstructured data into hadoop

Cassandra and unstructured data

Knockoutjs mapping and unstructured data

Ideal way to read data in bucket stored batches of data for Keras ML training in Google Cloud Platform?

Using Google Cloud Platform for scheduled recurring API data pull that then loads the data to BigQuery

Google Cloud Platform: accumulate data from Pub/Sub to files in Cloud Storage without Dataflow

How to import data from Google Cloud Platform (BigQuery/Cloud SQL) into R?

How to convert this unstructured data to structured?

Convert unstructured data into a Python Dictionary

Parsing mixed structured and unstructured data

How to create unstructured data in terraform

Unstructured data to find a column count

Why videos are unstructured data in context of Big data?

data mining with unstructured data how to implement?

how can i download my data from google-cloud-platform using python?

How to fetch data from Google Cloud Platform via API from a mobile app on Cordova?

how to configure Google Cloud Platform Data Loss Prevention client library for python to work behind a SSL proxy?

Best way to transfer data between vm instaces in different projects in Google cloud platform

Manage Google Data Catalog in Terraform - Set Tag Template's visibility

How to insert/append unstructured data to bigquery table

How to structure unstructured data using apache pig