Figuring out the cost of versioning on Amazon S3

We all love versioning on Amazon s3. It gives us peace of mind and the ability to recover our data if something goes wrong. As for the Amazon documentation:

Versioning is a means of keeping multiple variants of an object in the same bucket. You can use versioning to preserve, retrieve, and restore every version of every object stored in your Amazon S3 bucket. With versioning, you can easily recover from both unintended user actions and application failures.

Storage facility
Photo by Steve Johnson on Unsplash

But what about costs?

Keeping many variants of an object will not be free. And as for S3 FAQ, the billing for once is obvious:

How am I charged for using Versioning?

Normal Amazon S3 rates apply for every version of an object stored or requested


Versioning has a simple price structure but how do you monitor the impact on your storage costs? If you spend 10K every month on S3, it is useful to know if versioning is 5%, 10% or 20% of the bill.

Knowing that normal rates apply for every version of an object does not answer the questions:

  • what percentage of your storage is used by versioning?
  • how much enabling versioning is costing you at the end of the month?

If you have 100s of TB or PB of data, millions or billions of objects, you cannot list the bucket and check the items one by one. It would be anyway too costly to do that. There is no direct way to find the added cost of the versioned objects using Cost Explorer. And the Cost and Usage Report provides only the total cost of your bucket.

Photo by Jose Antonio Gallego Vázquez on Unsplash

What about CloudWatch?

A metric in CloudWatch would be the perfect solution, as it would be easy to track and monitor. Unfortunately there is no metric that covers versioning. The current metrics available for a S3 bucket refer only to the number of objects or to the different storage classes, for example:

Available metrics for the bucket

CloudWatch could only help if we knew upfront the ratio between current and previous versions.

What about using the AWS CLI?

You can do almost everything using the AWS CLI. Including listing all the objects in your bucket and figure out their size and versioning status. A very dummy command could be:

$ aws s3 ls s3://YourBucketName --summarize --human-readable –recursive

The output can be processed and the compared with the total size from the CloudWatch metric to get the estimate of the total version size.

Unfortunately, the approach does not scale when you have a large bucket. Besides, listing periodically all the objects has its own costs and you might end up paying more than what you spend in versioning.

​​S3 Inventory then!

A not immediate but very accurate way to determine the cost of versioning is using S3 Inventory. What is it?

Amazon S3 inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet (Parquet) output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or a shared prefix (that is, objects that have names that begin with a common string).

We can generate two inventory reports in which the first report includes all versions of object and the second includes only the current versions. After calculating the size from both the reports, subtract the current version objects size from all version objects size.

But even easier, we can generate a single inventory report including only the current versions, compute the storage and compare it to the total bucket size from CloudWatch.

Sampling?

If you have a well distributed bucket, you can analyze only a meaningful subset of your data. It would be faster and cheaper too. For example, if the prefix of all your objects is randomly distributed between aa and zz, you can use a subset of objects using S3 Inventory or AWS CLI to estimate your costs.

For example, out of the 676 prefixes (26 × 26) below you can sample just a handful of them.

What about Cost Explorer?

We said that Cost Explorer does not expose the cost of versioning. But it can be used as a proxy, if you have a lifecycle rule in place to permanently delete previous versions .

Let us assume you have a retention of 30 days for previous versions. You can temporarily increase it and use the daily view of S3 costs in Cost Explorer to see if and how much it affects your billing. For example, you can increase the retention to 40 days, wait 10 days and rollback the change to 30 days.

If the cost of your versioning is significant, you will an increase for those 10 days with a drop when you rollback the change. If not, you know that the percentage of costs of versioning in your S3 bill is not significant, as in the following example.

Example of a daily view of S3 costs in Cost Explorer

To recap…

There is no simple way to determine the percentage cost of using versioning on S3 but there are a few options to have a reliable estimate. Use S3 inventory for an accurate value, play with a sample of your data or change a lifecycle rule for a reasonable guess.

What is an Availability Zone destruction? Should I care?

There are six different S3 storage classes on S3 and they differ in costs for storage, transition requests and retrieval requests. This is to cover all use cases or to make your AWS bill hard to forecast. Or both.

The old S3 Reduced Redundancy Storage is the past, and all the current classes share the same 11 9’s. They have different availabilities and SLAs but, they all have 6+ copies of the data stored on AWS servers.

Performance across the S3 Storage Classes
Performance across the S3 Storage Classes

Standard-IA or One Zone-IA?

But wait, how can Standard-IA and One Zone-IA share the same durability if the latter keeps all the copy of the same number of copies but only in a single AZ?

The devil is in the details: do you see that little asterisk in the table? The apparently insignificant note states:

Because S3 One Zone-IA stores data in a single AWS Availability Zone, data stored in this storage class will be lost in the event of Availability Zone destruction.

So even if the number of copies is the same and you can still boast your amazing 11 9’s to your customers, the resilience of the One Zone-IA option is not the same. Even Amazon suggests it for backups or easily re-creatable data only.

Unlike other S3 Storage Classes which store data in a minimum of three Availability Zones (AZs), S3 One Zone-IA stores data in a single AZ and costs 20% less than S3 Standard-IA. S3 One Zone-IA is ideal for customers who want a lower-cost option for infrequently accessed data but do not require the availability and resilience of S3 Standard or S3 Standard-IA. It’s a good choice for storing secondary backup copies of on-premises data or easily re-creatable data.

With that little asterisk, Amazon lets you figure it out yourself the risk and the cost of it. As it might vary across regions and zones too, they give you 11 9’s with a caveat. You pay 20% less but you are now in charge of risk management.

What is an Availability Zone destruction? How likely is that?

AZ destruction?

It might sound a silly question but if you have to assess yourself the likelihood, you need to know what they are talking about. There is no further asterisk or note, so you have to dig deeper and ask support to have the definition.

(…) If you would like I can open a request to have the definition ‘Availability Zone destruction’ added

OK, there is nothing specific in the S3 documentation but they still state that

Availability Zone destruction would include any event that causes multiple data centers to become unavailable including but not limited to fires, earthquakes, a flood, etc. A destructive availability zone event can be summarized as any event that would impact multiple data centers or a sizeable portion of the region the resources are located in.

Should I take the risk?

For business critical services it is always advisable to architect your solution to be highly available and fault tolerant to an AZ outage. But this is just risk management.

If you heavily rely on Standard-IA and think the chance of your “next unicorn” crashing are higher than an Availability Zone destruction, you might gamble on lower costs and higher margins. But bet a few dollars on an AZ destruction too. In both cases you would have a profitable exit (*)

An unicorn or Standard-IA?

(*) Talking about asterisks, if you try that or find out a bookmaker that takes the bet please let me know.

S3 Reduced Redundancy Storage is (almost) dead

As a software developer and architect I have spent countless hours discussing the benefits, the costs and the challenges of deprecating an API or a service in a product. AWS has the opportunity and the business model to skip the entire discussion and simply use the pricing element to make a service useless.

Let’s take the Reduced Redundancy Storage option for Amazon S3. It has been around since 2010 and the advantage versus standard storage is (actually, was) the cost. As for the AWS documentation:

It provides a cost-effective, highly available solution for distributing or sharing content that is durably stored elsewhere, or for storing thumbnails, transcoded media, or other processed data that can be easily reproduced. The RRS option stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive, but does not replicate objects as many times as standard Amazon S3 storage

Again, the only benefit of Reduced Redundancy Storage is the cost. Once you remove the cost discount, it’s a useless feature.

imageedit_1_2408963963

If you make it more expensive than standard storage you are effectively deprecating it without having to change a single SDK or API signature. And that’s exactly what AWS did lowering the price of standard storage class without changing the one for the Reduced Redundancy Storage option.

These are the prices for US East (N. Virginia) but similar differences apply in other regions:

Amazon S3 Reduced Redundancy Storage
imageedit_4_5408119950

The only real change was then moving the RRS from the main S3 page (where now the options do not include RRS anymore) to a separate one.

Amazon S3

imageedit_7_4172100111

S3 Reduced Redundancy Storage is still there, they did not even increase the price of the service. But it’s a dead feature and you have no reason to use it anymore. An amazing approach to the challenge of deprecation.

Transparent server side encryption on S3

I would like to take advantage of Server-Side Encryption with Amazon S3-Managed Encryption Keys for some existing buckets where I have some legacies applications storing data without S3 encryption. Encryption of data at rest is of course important and with the chance of doing it on AWS with a simple flag (or one line of code) there is not much of an excuse not using it while working with S3. But how does it work for old legacy applications where you might not able to change the client code soon? Unfortunately there is not a simple way to achieve it using S3 configuration only.

Ideally I would love to simply find a simple bucket property in the console but unfortunately there is not one. With a bucket policy I can of course lock PUT requests without server-side encryption but my goal is to convert them to PUT with server side encryption, not simply reject the requests. A bucket policy can only check for permissions on the object that is uploaded to S3 and compare to the rules set, it cannot transform data on the fly.

Any other option?

You can implement a Lamdba function that performs a new PUT for the same objects on every PUT of  requests without the server-side encryption attribute: it has some implications on the costs but it’s an easy short term workaround while you adapt the legacy application and it’s entirely transparent to the existing clients, whatever they are using SDKs or directly performing HTTP requests (check the doc to get a general idea about S3 integration with Lambda)

Of course the long-term solution should be to implement Server Side Encryption with the SDK changing client code but a Lambda function can be your short-term hack.