Synchronize data from MySql to Amazon RedShift

Gavriel

We do some aggregation on huge datasets in Amazon RedShift, and we have some relatively small amount of data in MySQL. For some of the joins in RedShift we need the data in MySQL. What is the best way to synchronize the MySql data to RedShift? Is there such a thing in redshift like the remote view in oracle? Or should I programatically query MySql and insert / update in RedShift?

altermativ

When MySQL data is required for joins in Redshift, we usually just send it over from one to another.

It implies:

  1. Redshift: Creating an analogous table schema (bearing in mind Redshift/PSQL's particularities)
  2. MySQL: Dumping the data table (in csv format)
  3. Zipping the export, and sending it to S3
  4. Redshift: Truncating the table, and importing all data using COPY

Steps 2 to 4 can be scripted, and allow you to send fresh data over to Redshift when necessary or regularly.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Synchronize data from MySql to Amazon RedShift

From Dev

Export from Amazon Redshift into an RDS MySQL database

From Dev

Migrating Data From Amazon Redshift into DynamoDB

From Dev

Incremental loading into amazon redshift from local mysql database - Automation process

From Dev

Out of Memory:Transferring Large Data from Amazon Redshift to Pandas

From Dev

Copying data from MySQL to Amazon DynamoDB

From Dev

Amazon Redshift: Copying Data Between Databases

From Dev

Amazon Redshift - How to extract previous month data

From Dev

connecting amazon redshift server from tableau server

From Dev

Export query from MySQL to Redshift

From Dev

Synchronize data from Steam Api and IGDB

From Dev

How to Load Data into Amazon Redshift via Python Boto3?

From Dev

Amazon Redshift how to get the last date a table inserted data

From Dev

Amazon Redshift C# client to query data without ODBC/JDBC

From Dev

How do I write a Luigi workflow for loading data into Redshift from MySQL?

From Dev

Loading Unicode data from mysql to Redshift fails with "Bad UTF8 hex sequences"

From Dev

Copying specific Columns in Amazon Redshift from S3 databucket

From Dev

Error connecting to amazon redshift from R - windows 10

From Dev

How to convert FROM sas date format TO int in postgres (Amazon redshift)?

From Dev

When to use Amazon Redshift spectrum over AWS Glue ETL to query on Amazon S3 data

From Dev

Append and Overwrite in Amazon Redshift

From Dev

Pivot a table with Amazon RedShift

From Dev

Amazon RedShift Copy Command

From Dev

Unsigned field in Amazon Redshift?

From Dev

Amazon Redshift error in Tableau

From Dev

Amazon Redshift rank group

From Dev

Amazon Redshift terminology

From Dev

Java connect to Amazon Redshift

From Dev

Getting JSON Data from Redshift table

Related Related

  1. 1

    Synchronize data from MySql to Amazon RedShift

  2. 2

    Export from Amazon Redshift into an RDS MySQL database

  3. 3

    Migrating Data From Amazon Redshift into DynamoDB

  4. 4

    Incremental loading into amazon redshift from local mysql database - Automation process

  5. 5

    Out of Memory:Transferring Large Data from Amazon Redshift to Pandas

  6. 6

    Copying data from MySQL to Amazon DynamoDB

  7. 7

    Amazon Redshift: Copying Data Between Databases

  8. 8

    Amazon Redshift - How to extract previous month data

  9. 9

    connecting amazon redshift server from tableau server

  10. 10

    Export query from MySQL to Redshift

  11. 11

    Synchronize data from Steam Api and IGDB

  12. 12

    How to Load Data into Amazon Redshift via Python Boto3?

  13. 13

    Amazon Redshift how to get the last date a table inserted data

  14. 14

    Amazon Redshift C# client to query data without ODBC/JDBC

  15. 15

    How do I write a Luigi workflow for loading data into Redshift from MySQL?

  16. 16

    Loading Unicode data from mysql to Redshift fails with "Bad UTF8 hex sequences"

  17. 17

    Copying specific Columns in Amazon Redshift from S3 databucket

  18. 18

    Error connecting to amazon redshift from R - windows 10

  19. 19

    How to convert FROM sas date format TO int in postgres (Amazon redshift)?

  20. 20

    When to use Amazon Redshift spectrum over AWS Glue ETL to query on Amazon S3 data

  21. 21

    Append and Overwrite in Amazon Redshift

  22. 22

    Pivot a table with Amazon RedShift

  23. 23

    Amazon RedShift Copy Command

  24. 24

    Unsigned field in Amazon Redshift?

  25. 25

    Amazon Redshift error in Tableau

  26. 26

    Amazon Redshift rank group

  27. 27

    Amazon Redshift terminology

  28. 28

    Java connect to Amazon Redshift

  29. 29

    Getting JSON Data from Redshift table

HotTag

Archive