Transfer terabytes of data between AWS s3 buckets cross-account, cross-region and cross-vpc



Create s3cross-account IAM user :

Create s3-cross-account user’s policy :

Change bucket’s policies:

# test bucket A access
$ aws s3 ls --profile s3-prod-cross-account s3://bucket-A --region eu-west-1
$ aws s3 cp --profile s3-prod-cross-account s3://bucket-A/some-file . --region eu-west-1
-> some-file downloaded locally# test bucket B access
$ aws s3 ls --profile s3-prod-cross-account s3://bucket-B --region eu-west-3
$ touch dummyfile
$ aws s3 cp --profile s3-prod-cross-account ./dummyfile s3://bucket-B --region eu-west-3
-> dummyfile uploaded to s3

Transferring the data :

Dealing with small buckets:

aws s3 cp --profile s3-prod-cross-account s3://bucket-source/ s3://bucket-target/ --source-region source-region --region target-region --recursive
aws s3 cp --profile s3-prod-cross-account s3://bucket-A/ s3://bucket-B/ --source-region eu-west-1 --region eu-west-3 --recursive

Dealing with bigger buckets:

An error occurred (AccessDenied) when calling the CopyObject operation: VPC endpoints do not support cross-region requests

Final results:

NAME                 COMPLETIONS   DURATION   AGEs3-copy-bucket-a      1/1           66m        22ds3-copy-bucket-a1     1/1           77m        22ds3-copy-bucket-a2     1/1           74m        22ds3-copy-bucket-a3     1/1           72m        22ds3-copy-bucket-a4     1/1           65m        22ds3-copy-bucket-a5     1/1           64m        21ds3-copy-bucket-a6     1/1           66m        22ds3-copy-bucket-a7     1/1           67m        22ds3-copy-bucket-a8     1/1           77m        22ds3-copy-bucket-a9     1/1           74m        22d

Final thoughts



Original post:

Leave a Reply

Your email address will not be published. Required fields are marked *