Access to "trapped" Data requires Data Transformation

Years ago I wrote an article using Microsoft Word. I kept the file in my folder until recently when I wanted to repurpose it. I went to retrieve the file and open it with the current version of Microsoft Word. The software asked me whether I was sure I wanted to do that since some of the formats may not transfer. This is not an uncommon occurrence, yet when we want to move data where the source and target are different, we think we can do that without any data transformation and the application should just be able to access it as when it was written.

The reality is that when moving data either between applications or versions of applications or between storage systems or protocols, data must be transformed. Here are a few examples of data transformations Interlock had to perform:

Custom application data moved from CAS object to S3 object.

80TB of data written by a custom application had set retentions. Data had to be transformed from CAS to S3 formats. The customer wanted to change retentions on the data; these had to be modified prior to committing data to the target system. Since there was no change to the custom application, data had to be presented with all metadata add-ons associated with the application. Since metadata is written differently in CAS versus S3, accommodations had to be made to ensure the application could access the metadata and content.

Move data and snapshots from NAS to S3 with versions.

Data stored on a NAS used snapshot technology for data protection. Snapshots provided views of a file at various points of time. If the file was unintentionally modified or corrupted, a snapshot could be used to retreat the most relevant instance. This data had to be moved to the S3 storage system to accommodate changes in the application being used. There were two main transformations that had to happen:

• Data had to be transformed from a file format to an object format, along with retentions, permissions, metadata, etc.
• Snapshots had to be unraveled and converted to object versioning. It was necessary to ensure that users could revert to older instances of the file/object if needed.

Migrate data with extensive metadata that had to be carried forward.

The application didn’t maintain a separate database for storing metadata. All metadata was appended to the file/object. In some cases, there were 100s of metadata items that had to be carried over to the new system. These had to be converted to the new formats while maintaining the application’s ability to discover the metadata.