An Amazon S3 lifecycle rule can delete objects automatically after a set number of days, or move them to a cheaper storage class. This helps you follow a data retention plan and avoid paying to store data you no longer need. This article covers the deletion (expiration) rule.
UCR researchers get AWS accounts through ITS under the University of California agreements; see Cloud accounts. This article is for Amazon S3. CephRDS, the on-campus S3 storage, does not support automatic deletion (lifecycle) rules; delete data there yourself when your retention period ends.
Expiration is permanent. Once a rule deletes an object, it cannot be recovered unless you keep another copy. Check the rule's scope carefully, and check your data retention obligations (funder, journal, and UC records rules; see Records retention) before you set one.
Step 1: Create a bucket (if you need one)
- Sign in to the AWS Management Console and open the S3 console.
- Click Create bucket.
- Enter a bucket name and choose the AWS Region.
- Review the other settings, then click Create bucket.
Step 2: Add a lifecycle rule
- In the S3 console, open your bucket and select the Management tab.
- Under Lifecycle rules, click Create lifecycle rule.
- Enter a Lifecycle rule name.
- Choose the rule scope:
- Limit the scope of this rule using one or more filters, and enter a prefix (for example
scratch/) to apply it to one folder; or - Apply to all objects in the bucket, and tick the acknowledgement.
- Limit the scope of this rule using one or more filters, and enter a prefix (for example
- Under Lifecycle rule actions, select Expire current versions of objects.
- Enter the number of Days after object creation (for example,
90). - Review the summary and click Create rule.
S3 runs lifecycle rules in the background. Objects are usually removed within a day or two of reaching the set age, not at an exact time.
If versioning is turned on
In a versioned bucket, expiring the current version only adds a delete marker; the older versions remain and are still billed. To remove them too, also select Permanently delete noncurrent versions of objects and set the number of days.
Command-line alternative
The same rule with the AWS CLI. Save this as lifecycle.json (replace the prefix and days as needed):
{
"Rules": [
{
"ID": "expire-scratch-after-90-days",
"Filter": { "Prefix": "scratch/" },
"Status": "Enabled",
"Expiration": { "Days": 90 }
}
]
}
Then apply it to your bucket:
aws s3api put-bucket-lifecycle-configuration --bucket my-lab-bucket --lifecycle-configuration file://lifecycle.json
This replaces any existing lifecycle configuration on the bucket. To see what is already set, run aws s3api get-bucket-lifecycle-configuration --bucket my-lab-bucket first.
More information
- Managing the lifecycle of objects (AWS)
- Setting a lifecycle configuration on a bucket (AWS)
- For Google Cloud Storage, see Object Lifecycle Management and KB012: Using Ursa Major archive storage.