Object Storage lifecycle policies failure causes data loss
I am using lifecycle policies:
<noncurrentversionexpiration> <noncurrentdays>5</noncurrentdays> </noncurrentversionexpiration>
Using this option will delete objects that are not the newest, most current version, is this right, but the deleted objects (marked with DeleteMarker), the latest version of the file is DeleteMarker, so it deletes all my versions and keeps only DeleteMarker, causing data loss.
To fix this problem you should be able to use "NewerNoncurrentVersions" according to the documentation https://docs.aws.amazon.com/en_us/AmazonS3/latest/API/API_NoncurrentVersionExpiration.html
but it seems that this is not implemented in the linode object storage, how to fix this data loss?
1 Reply
hphillips
Linode Staff
According to https://www.linode.com/blog/cloud-storage/s3-compatible-object-storage-for-your-business/, Linode Object Storage is built on Ceph.
Searching through GitHub in https://github.com/ceph/ceph, I do not see NewerNoncurrentVersions implemented.
If you would like to retain previous versions of a file that may be deleted AND use the noncurrentversionexpiration, you could create two copies of your file whenever you create it, and only delete the current one when doing your delete operations:
Current/ExampleFile
Backups/ExampleFile
You would then do something like:
<LifecycleConfiguration>
<Rule>
<ID>RemoveNonCurrentVersions</ID>
<Filter>
<Prefix>Current/</Prefix>
</Filter>
<Status>Enabled</Status>
<NonCurrentVersionExpiration>
<NonCurrentDays>1</NonCurrentDays>
</NonCurrentVersionExpiration>
</Rule>
<Rule>
<ID>RemoveNonCurrentVersionsOfBackups</ID>
<Filter>
<Prefix>Backups/</Prefix>
</Filter>
<Status>Enabled</Status>
<NonCurrentVersionExpiration>
<NonCurrentDays>30</NonCurrentDays>
</NonCurrentVersionExpiration>
</Rule>
</LifecycleConfiguration>