Big Data, the Data Lifecycle and data curation

This morning I came across a paper which provoked a chain of thought that I hadn't explicitly considered -- curation of these amazing big data resources we are building.

Management of data has been guided for a couple of decades (an era in information systems terms) by the Data Lifecycle, which has had modifications suggested over the years, but is essentially unchanged since it became popular.

The paper, (Pouchard, 2016) raises the issue of how the entry of something as radical as Big Data onto the scene is going to affect attitudes and processes.

This is something we need to get on top of very rapidly while Big Data is in the development stages and has not been locked down. It's much at the stage that structured databases were before SQL dominated -- remember? Bespoke data structures and conventions, hierarchical structures, hard coded reporting. There was also the urban legend of the collection by a national library of data used by public instrumentalities. Unfortunately the collection didn't include metadata, not did it record charactersets or even the hardware that the collection, largely of tapes, had been stored from. Well meaning, but not even really useful as landfill.

The potential for this to happen and the scale of the impact, as Big Data takes hold and inevitably rapidly evolves as new potential is discovered and realised, can only be imagined.

Pouchard focuses on research data, but this applies equally to other organisations using Big Data. His proposal is aligned to the research cycle, but he proposes a number of salient questions for practitioners, which he relates to the "four V's" of Big Data and his model.

The two dimensions create a matrix of pertinent questions which cover most aspects of dealing with this problem. They are very down to earth and free of academic jargon and well worth looking at.

The research team intends to work with real projects to test and refine their model and the issues that arise from it. Pouchard emphasises that the curation issue needs to be considered in the planning stages of a Big Data project, so have a look now, get a feel for the issues, don't wait for the final results. Let's face it my colleagues and I in the research game, never finalise anything!

Download it from the link below.

Pouchard L (2015)  Revisiting the Data Lifecycle with Big Data Curation, International Journal of Digital Curation, 10(2) 176-192, doi: http://dx.doi.org/10.2218/ijdc.v10i2.342