Migrate AWS S3 Bucket to Google Cloud Storage - Storage Transfer Service

Written By Guest User
Google Cloud Storage Database Migration

On Jun 14, 2021 we saw Migrate AWS RDS to GCP Cloud SQL in Five Simple Steps in our Evonence GCP’s Blog Series. As a continuation, this blog will explore how AWS S3 contents can be easily transferred to GCP’s Cloud Storage using Storage Transfer Service for Cloud, a Free Service except the charges associated with network usage(AWS Network Egress) and operations performed by the transfer service at GCP Cloud Storage’s Class A PUT operation charge and AWS S3 Bucket’s storage cost.

Prerequisite

Existing AWS S3 Bucket with some data and User with Programmatic access to AmazonS3FullAcess

Prerequiisite for Database Migration
setting Permission

Make a note of Access Key ID and Secret Access Key.

Secret access key

Lastly, create a S3 and have some objects in it

AWS S3 Bucket

For the demonstration purpose three video files each with ~400MB size are uploaded to AWS S3 bucket.

The game starts now...

Make sure you have a GCP Project with Billing enabled and Cloud Storage API enabled.

GCP to AWS Migration

  1. Create a Cloud Storage Bucket.

  2. Now let’s create a Transfer Job using Storage Transfer Service wizard with source as AWS S3 bucket with the Access Key ID and Secret Access Key which was generated in the prerequisite step.

Storage Transfer Services

3. Define Cloud Storage Bucket as the destination

Cretae Cloud Storage Bucket as the destination

4. Provide an apt description for the Job

Creating a Transfer Job

5. Select the preferred Scheduling option

Preferred Scheduling option

Finally, 1.2GB of data got transferred is 5 min 55 sec with an average speed of 3.67 MB/s

Serverless Migration tool
Additional scheduler GCP Facts

Benefits

  1. Easy-to-use and Serverless Migration tool

  2. Built-in scheduler hence not need of any additional scheduler solutions

GCP Facts

  1. You can use Cloud Secret Manager to save this sensitive information like AWS Access Key ID & Secret Access Key

  2. Alway use Regional/ Standard Region Storage Class for any Data Processing 

  3. Storage Transfer Service for cloud currently does not support CLI option. Only GCP Console & REST API options are available

Conclusion

Easily migrate AWS S3 Bucket contents to GCP Cloud Storage Bucket in a simple and straightforward fashion using Storage Transfer Service in a wizard-style. 

Now, we have seen how to migrate AWS RDS & AWS S3. The next target would be to migrate AWS EC2 to Containers using Migrate for Anthos...Stay tuned...

Guest User
Previous

Deploy Magento on Google Kubernetes Engine
Next

Migrate AWS RDS to GCP Cloud SQL in Five Simple Steps