Versioning, lifecycle rules and tags¶

Versioning¶

A bucket starts unversioned: writing a key replaces what was there, and deleting it removes it. Turning versioning on keeps every version instead.

aws --endpoint-url https://s3.example.org s3api put-bucket-versioning \
    --bucket my-bucket --versioning-configuration Status=Enabled

A bucket can be Enabled or Suspended, and can't go back to having no versioning, as in S3.

• Enabled: each write gets a new version ID. A delete leaves a delete marker, which has an ID too, in front of the versions it hides.
• Suspended: a write or a delete replaces the version named null, leaving the other versions in place.
• Reading: versionId reads any version. A key whose current version is a delete marker answers 404 NoSuchKey. The marker read by its ID answers 405. Both answers include x-amz-delete-marker. A plain listing shows current objects only, while ListObjectVersions shows every version and marker.
• Erasing: DELETE with a versionId removes that version for good. If it was the current one, the next newest takes its place.
• Restoring: copy an older version onto the same key, with x-amz-copy-source naming the version.

Older versions are stored as they were, and stay until erased, by hand or by a lifecycle rule. Reading and listing current objects cost the same with versioning on: at 100,000 keys, no difference was measurable.

On a scoped key, changing a bucket's versioning and erasing a version both need admin. A key with write can delete an object, which leaves a marker, but can't destroy the versions behind it. See Access keys.

ListObjectVersions also works on a bucket that was never versioned, where each object is version null. Tools that empty buckets use it, such as boto3's object_versions.delete() and Terraform's force_destroy.

Lifecycle rules¶

A lifecycle rule removes objects on a schedule. A bucket's rules can:

• expire current objects some days after they were written, or from a date;
• erase older versions some days after they were replaced, keeping the newest few if asked;
• remove a delete marker left with no versions behind it;
• abort multipart uploads some days after they started.

A rule applies to a whole bucket, or to objects matching a prefix, a size range or tags. In a versioned bucket, expiring an object goes through the same path as a client's delete: it leaves a delete marker. The version is kept until a rule on older versions erases it.

<LifecycleConfiguration>
  <Rule>
    <ID>logs</ID>
    <Status>Enabled</Status>
    <Filter><Prefix>logs/</Prefix></Filter>
    <Expiration><Days>30</Days></Expiration>
  </Rule>
  <Rule>
    <ID>old-versions</ID>
    <Status>Enabled</Status>
    <Filter></Filter>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>7</NoncurrentDays>
      <NewerNoncurrentVersions>3</NewerNoncurrentVersions>
    </NoncurrentVersionExpiration>
    <AbortIncompleteMultipartUpload>
      <DaysAfterInitiation>2</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
  </Rule>
</LifecycleConfiguration>

With the AWS CLI, the same rules go in JSON to aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://rules.json.

What a rule may contain¶

A bucket holds at most 1,000 rules. A rule on Days also removes a lone delete marker of that age, as AWS documents.

What is refused¶

A rule deletes data, so leBLOB refuses a rule it doesn't fully understand. Skipping a condition inside a filter would apply the rule to more objects than intended.

• An unknown element, a repeated one, an empty number or a tag missing its key or value: 400 MalformedXML.
• Transition and NoncurrentVersionTransition: 501 NotImplemented. There is one storage class.
• A rule with neither Filter nor Prefix: MalformedXML. A rule over a whole bucket says so, with an empty <Filter>.
• An And next to other conditions in the same Filter: MalformedXML, since it isn't clear which was meant.
• A rule aborting uploads that filters on size or tags: 400, since an upload has neither yet. It can filter on a prefix.
• A rule removing lone delete markers that filters on tags: 400, since markers have no tags.

Setting or removing rules needs admin on a scoped key, and reading them read.

When rules run¶

Rules are applied in a pass that runs once a day, so a new rule can take up to a day to act. An object can go up to a day after it is due. Days count from the object's time exactly, where AWS rounds up to the next midnight UTC. Neither removes anything early.

A pass walks about 2,000 versions per second and removes about 200 per second, so that clients' writes keep priority: a bucket of 10 million keys takes about 80 minutes. After a restart, the pass resumes where it stopped.

Not built: the x-amz-expiration header, which tells a client when an object will expire.

Object tags¶

An object can have up to 10 tags, each a key and a value:

aws --endpoint-url https://s3.example.org s3api put-object-tagging --bucket my-bucket \
    --key report.pdf --tagging 'TagSet=[{Key=project,Value=apollo}]'

• PutObjectTagging, GetObjectTagging and DeleteObjectTagging work on the current version, or on another with versionId.
• Tags can also come with a PUT, a copy or a multipart upload, in the x-amz-tagging header. A copy keeps the source's tags unless x-amz-tagging-directive: REPLACE says otherwise.
• A read reports how many tags an object has in x-amz-tagging-count.
• Tagging changes no time, ETag or version, so a lifecycle rule doesn't see a tagged object as newer.

Limits are AWS's: keys of up to 128 characters, values of up to 256, using letters, numbers, spaces and + - = . _ : / @. A request that sends x-amz-tagging where tags can't apply gets 501, so the tags aren't dropped without a word.

On a scoped key, tagging requires write. Tagging an older version by ID requires admin.

Bucket tagging isn't built.