If you want to store multiple copies of objects in your S3 buckets in different regions, S3 can be set up to automatically replicate objects from a source bucket into replica buckets around the world, increasing performance and access latency of your applications.
It should be noted that if the objects you’re replicating are publically available over the internet, such as a media bucket for uploaded content, you can simply use AWS’s CloudFront CDN to cache content on edge locations everywhere.
How Does Replication Work?
Replication basically runs whenever you upload or update an object in a bucket. You create a replication rule, which defines the source and destination bucket. Objects uploaded into the source bucket will be uploaded into the destination bucket. For replication to work, you’ll need Versioning enabled. This will increase storage costs slightly, depending on how much you overwrite data.
There are some limitations. AWS doesn’t replicate deleted objects, so you’ll need to delete each copy manually. And, while replication usually happens fairly quickly, it can take up to 15 minutes or more if you’re replicating large objects.
When an object is replicated, a metadata tag is set in the source object. All of the replicas will have the REPLICA tag, and the source object will be tagged COMPLETED. This means you can have bdirectional replication—two buckets with replication rules pointing at each other. S3 won’t replicate uploaded objects with the replica tag, otherwise you’d be stuck in an infinite replication loop.
This is called multi-master replication, but there’s no system in place for ensuring that objects uploaded later will be replicated later. In this setup, it’s possible for data to get out of sync if you’re updating two objects simultaneously, and it’s possible to be operating on stale data as it syncs from one master to another.
Setting Up Replication
From the S3 Management Console, navigate to the source bucket. You’ll need to turn on Versioning under “Properties:”
Once that’s turned on, you can add a new replication rule under Management > Replication > Add Rule.
By default, it will replicate all objects, though you can set a filter if you’d like.
Under Destination, select the destination bucket. You can also configure the storage class for the replica objects, as well as setting the owner if you’re transferring across accounts. There’s also an option for “S3 Replication Time Control,” which guarantees objects will be replicated within 15 minutes, backed by a service level agreement.
On the next screen you’ll need to select or create an IAM role for the replication. The permissions are handled automatically, just enter a name.
Click “Next,” and click “Save.” The rule should be active immediately; you can test uploading an object, and you should see it replicated to the destination bucket, then you’ll see the replication status tag change to COMPLETED.
