blog sysoft how to transfer 1000 gb data from aws s3 using s3 sync quickly

How I have transferred 1000 GB data from AWS S3 in few Hours

Have you ever tried to migrate GB’s of data from aws s3 to your local PC or another s3 bucket? If so, you’ll know that it’s difficult because the enormous size of data and small chunks of the file could have taken days to do the data transfer, and meanwhile new data piled up in the S3 Bucket.

Many people have found solutions by simply copying the folder and files using any s3 client software manually, but there are a few things you should know before you start this data transfer activity that could save your hours & days of effort. This post will tell you what you need to know to make sure you do the data transfer in the most efficient, automated way.

Before you begin the activity, here are a few prerequisites.

Prerequisites: (aws-cli, s3 bucket, IAM user with programmatic access)

  1. Install the aws-cli refer amazon document for installation steps
  2. Make sure you have an aws IAM user with proper access to read/write on the s3 buckets.
  3. Configure the AWS
  4. IAM user credentials Accesskey / secret key in the system

Once the above-mentioned steps are complete to ensure everything is set up properly. Open the terminal or command prompt and type the below command if everything is good, then you should be able to see the list of buckets of your account.

aws s3 ls

💡Quick Tip: As quick automated way to do the data sync from aws s3 to local PC, I have create the windows batch script , so you can change the value and run in your windows system. and it will do the task for you.

Save the below code as s3sync.bat

em Download AWS CLI https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html#install-msi-on-windows

echo 'setting the variables'

rem source bucket name with folder ex. aws.XXXX.com.001/folder
set BUCKET_NAME=aws.yoursources3bucket.com/folder

rem destinaion directory (Local PC) name with folder ex. C:\Users\abc\Newfolder.. Do not keep space in the directory.
set DESTN_DIR=c:\data\s3

rem Access Key & Secret Key. 
set AWS_ACCESS_KEY_ID=AKIXXXX
set AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX

rem AWS CLI installation path
set AWS_PATH="C:\Program Files\Amazon\AWSCLI\bin\aws"

rem Max. requrent requests
%AWS_PATH% configure set default.s3.max_concurrent_requests 200

rem Download/Sync command
echo %BUCKET_NAME% %DESTN_DIR%
%AWS_PATH% s3 sync s3://%BUCKET_NAME% %DESTN_DIR%

Please not that all the aws configurations will be store in your local PC folder in the path C:\Users\Yourname\.aws

Join to become subscribers

Stay in the loop with everything you need to know.

Discover more from syssoft

Subscribe now to keep reading and get access to the full archive.

Continue reading