How I have transferred 1000 GB data from AWS using aws s3 sync in few Hours

Have you ever tried to migrate GB’s of data from aws s3 to your local PC or another s3 bucket? If so, you’ll know that it’s difficult because the enormous size of data and small chunks of the file could have taken days to do the data transfer, and meanwhile new data piled up in the S3 Bucket.

Many people have found solutions by simply copying the folder and files using any s3 client software manually, but there are a few things you should know before you start this data transfer activity that could save your hours & days of effort. This post will tell you what you need to know to make sure you do the data transfer in the most efficient, automated way.

Before you begin the activity, here are a few prerequisites.

Requisites: (aws-cli, s3 bucket, IAM user with programmatic access)

  1. install the aws-cli.

  1. Make sure you have an aws IAM user with proper access to read/write on the s3 buckets.
  2. Configure the AWS 
  3. IAM user credentials Accesskey / secret key in the system

Once the above-mentioned steps are complete to ensure everything is set up properly. Open the terminal or command prompt and type the below command if everything is good, then you should be able to see the list of buckets of your account.

aws s3 ls

As quick automated way to do the data sync from aws s3 to local PC, I have create the windows batch script , so you can change the value and  run in your windows system. and it will do the task for you.

Save the below code as s3sync.bat

rem Download AWS CLI

echo 'setting the variables'

rem source bucket name with folder ex.

rem destinaion directory (Local PC) name with folder ex. C:\Users\abc\Newfolder.. Do not keep space in the directory.
set DESTN_DIR=c:\data\s3

rem Access Key & Secret Key. 

rem AWS CLI installation path
set AWS_PATH="C:\Program Files\Amazon\AWSCLI\bin\aws"

rem Max. requrent requests
%AWS_PATH% configure set default.s3.max_concurrent_requests 200

rem Download/Sync command

Please not all the aws configurations will be store in your local PC folder in the path C:\Users\Yourname\.aws

in case you need some professional help , please contact at [email protected]in

Leave a Reply