Have you ever tried to migrate GB's of data from aws S3 to you local PC or another s3 bucket? If so, you'll know that it's difficult because the huge size of data and small chucks of file could have taken days to do the data transfer and meanwhile new data have been piled up in the S3 Bucket .
So what do you do? Many people have found solutions by simply copying the folder and files using any S3 client software manually , But there are a few things you should know before you start this data transfer activity which could save your hours & days of effort.
This post will tell you what you need to know to make sure you do the data transfer in most efficient , automated way.
Before begin the activity here are few prerequisites.
Prerequisites: ( aws-cli, s3 bucket, IAM user with programmatic access )
- install the aws-cli. https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html#install-msi-on-windows
- make sure you have aws iam user with proper access read / write on the s3 buckets
- configure the aws IAM user credentials accesskey / secretkey in the system
Once the above mentioned steps are complete to ensure everything is setup properly . open the terminal or command prompt and type the below command if every thing is good then you should be able to see the list of buckets of your account.
aws s3 ls
As quick automated way to do the data sync from aws s3 to local PC i have create the windows batch script , so you can change the value and run in your windows system. and it will do the task for you.
save the below code as s3sync.bat
rem Download AWS CLI https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html#install-msi-on-windows echo 'setting the variables' rem source bucket name with folder ex. aws.XXXX.com.001/folder set BUCKET_NAME=aws.yoursources3bucket.com/folder rem destinaion directory (Local PC) name with folder ex. C:\Users\abc\Newfolder.. Do not keep space in the directory. set DESTN_DIR=c:\data\s3 rem Access Key & Secret Key. set AWS_ACCESS_KEY_ID=AKIXXXX set AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX rem AWS CLI installation path set AWS_PATH="C:\Program Files\Amazon\AWSCLI\bin\aws" rem Max. requrent requests %AWS_PATH% configure set default.s3.max_concurrent_requests 200 rem Download/Sync command echo %BUCKET_NAME% %DESTN_DIR% %AWS_PATH% s3 sync s3://%BUCKET_NAME% %DESTN_DIR%
Please not all the aws configurations will be store in your local PC folder in the path C:\Users\Yourname\.aws
in case you need some professional help , please contact at email@example.com