IBM Cloud Object Storage S3API Tutorial: How to save money with ranged read in small object public cloud archives.

With our public cloud offering we see many clients with very small object workloads struggling with the transaction costs, although they are minimal (<5%) in many object storage cases there are still some archival cases that have a huge amount of very small objects which highers the transaction costs to a higher number than storing the data in a vault or cold-vault bucket. Especially when using these "infrequent access" and "glacier" tiers the transaction cost goes up so the less PUT transactions the cheaper cost for your archive.

(For more information on all our Cloud Object Storage Public Prices : CLICK HERE)

I quickly created a small overview on how you could save a lot in transactions by doing a ranged read, in this small example I will put an objects which I created from three pictures and than read every picture 1-by-1.

1) Create our combined object

cat 2017-18de-guzziclassic-otwe-1.jpg >> 1fotoinobject

cp 1fotoinobject 2fotoinobject

cat 2017-18de-guzziclassic-otwe-2.jpg >> 2fotoinobject

cp 2fotoinobject 3fotoinobject

cat 2017-18de-guzziclassic-otwe-3.jpg >> 3fotoinobject

Above I created three objects, 1 with 1 picture, 1 with 2 pictures and 1 with 3 pictures. For this tutorial it makes it easy to know the byte ranged to read the pictures separately :

2) Upload the three objects to my bucket in Bluemix

aws s3 cp --endpoint-url http://s3-api.us-geo.objectstorage.softlayer.net s3partialtest/1fotoinobject s3://s3rangedread/

aws s3 cp --endpoint-url http://s3-api.us-geo.objectstorage.softlayer.net s3partialtest/2fotoinobject s3://s3rangedread/

aws s3 cp --endpoint-url http://s3-api.us-geo.objectstorage.softlayer.net s3partialtest/3fotoinobject s3://s3rangedread/

3) List the objects and review byte offset

Since our Bluemix GUI is by default reporting in human readable format, lets request the details through the AWS S3API CLI :

aws s3api list-objects --endpoint-url http://s3-api.us-geo.objectstorage.softlayer.net/ --bucket s3rangedread

{
    "Contents": [
        {
            "LastModified": "2017-06-19T21:32:53.064Z",
            "ETag": "\"cd6a74a40c4fcf2fe6fe9f57088e1a9d\"",
            "StorageClass": "STANDARD",
            "Key": "1fotoinobject",
            "Owner": {
                "DisplayName": "622287036f1f440c9f163977a851d647",
                "ID": "622287036f1f440c9f163977a851d647"
            },
            "Size": 315055
        },
        {
            "LastModified": "2017-06-19T21:33:50.286Z",
            "ETag": "\"7866af7a5da52cdca6af4cd0e1c4df8d\"",
            "StorageClass": "STANDARD",
            "Key": "2fotoinobject",
            "Owner": {
                "DisplayName": "622287036f1f440c9f163977a851d647",
                "ID": "622287036f1f440c9f163977a851d647"
            },
            "Size": 608314
        },
        {
            "LastModified": "2017-06-19T21:33:59.333Z",
            "ETag": "\"d34aec9ae9ea161a5a4e956152bb822b\"",
            "StorageClass": "STANDARD",
            "Key": "3fotoinobject",
            "Owner": {
                "DisplayName": "622287036f1f440c9f163977a851d647",
                "ID": "622287036f1f440c9f163977a851d647"
            },
            "Size": 912615
        }
    ]
}

Now we have enough information for our small test. In the object with key "3fotoinobject" the first picture is 1-315055, the second picture is 315055-608314 and the third picture is 608314-912615.

4) Perform ranged read to review all three pictures :

First picture :

aws s3api get-object --bucket s3rangedread --endpoint-url http://s3-api.us-geo.objectstorage.softlayer.net/ --range bytes=0-315055 --key 3fotoinobject final-foto1.jpg
{
    "AcceptRanges": "bytes",
    "ContentType": "binary/octet-stream",
    "LastModified": "Mon, 19 Jun 2017 21:33:59 GMT",
    "ContentRange": "bytes 0-315055/912615",
    "ETag": "\"d34aec9ae9ea161a5a4e956152bb822b\"",
    "ContentLength": 315056,
    "Metadata": {}
}

Second picture :

aws s3api get-object --bucket s3rangedread --endpoint-url http://s3-api.us-geo.objectstorage.softlayer.net/ --range bytes=315055-608314 --key 3fotoinobject final-foto2.jpg
{
    "AcceptRanges": "bytes",
    "ContentType": "binary/octet-stream",
    "LastModified": "Mon, 19 Jun 2017 21:33:59 GMT",
    "ContentRange": "bytes 315055-608314/912615",
    "ETag": "\"d34aec9ae9ea161a5a4e956152bb822b\"",
    "ContentLength": 293260,
    "Metadata": {}

}

Third picture :

aws s3api get-object --bucket s3rangedread --endpoint-url http://s3-api.us-geo.objectstorage.softlayer.net/ --range bytes=608314-912615 --key 3fotoinobject final-foto3.jpg
{
    "AcceptRanges": "bytes",
    "ContentType": "binary/octet-stream",
    "LastModified": "Mon, 19 Jun 2017 21:33:59 GMT",
    "ContentRange": "bytes 608314-912614/912615",
    "ETag": "\"d34aec9ae9ea161a5a4e956152bb822b\"",
    "ContentLength": 304301,
    "Metadata": {}

}

And since a picture says more than text :

5) Conclusion

As you can see the ability to perform ranged read can greatly reduce the amount of objects which has a direct impact on the PUT transaction cost. In some cases this could save up-to thousand of euro's a month due to the higher transaction costs in archive tiers like our vault and cold-vault offerings.

Feel free to contact me it you are having issues creating your internal business case for object storage, I am sure there are workloads that can profit from it! Luckily we are delivering the same technology for on-premise, hybrid cloud, dedicated cloud and public cloud use-cases since we have a 1-solution fits all but we deploy our solution in different configurations so no need to re-test/verify before moving your workload.