Dark Data problem

Organizations are amassing and storing more data than ever before and in the world of business maintaining and managing data often presents many challenges for growing companies though there are benefits apart from compliance and other data requirements.

Most data garnered are going to be used to improve something about the way organizations do business either from security or gaining insights to provide better experience. Data collected that are considered not valuable poses a real tactical value how they are managed and deemed as "Dark Data". Realistically as organizations collect vast amount of this unstructured information which they fail to use and being stored is costly to manage and potentially risky endeavor of protecting it. Major portion of data captured across enterprise are never used.

To get an idea, here are some shades which are considered as Dark Data which gets collected but fail to be used for other purposes - Log Files, Previous employe Data, Notes, Emails, Customer Information, Survey Data, Information assets etc., Useful data can be, the information we tend to hold for is simply collateral, in case we need to prove something occurred in the past, which becomes obsolete afterwards for any other use.

Risks and Problems with Dark Data

As with potentially rewarding and intriguing information assets, organizations must also be aware what potential risks that the dark data posses especially when it is stored in cloud outside of immediate control and management. This leads to serious security risks as no one wants to go through piles of old information that doesn't mean anything to them, but hackers aren't going think same way as they will be willing to mine that old data for data breaches that may have company sensitive information. Given the kind of information organizations collect, few potential risks the dark data can bring are

• Legal and regulatory risk : If data covered by mandate or regulation – such as confidential, financial information (credit card or other account data) or patient records – appears anywhere in dark data collections, its exposure could involve legal and financial liability. If dark data encompasses proprietary or sensitive information reflective of business operations, competitive advantages, important partnerships and joint ventures, and so forth, inadvertent disclosure could adversely affect or compromise important business activities and relationships.
• Reputation risk : Any kind of data breach reflects badly on the organizations affected thereby. This applies as much to dark data (especially in light of other risks) as to other kinds of breaches.
• Opportunity cost: Given that the organization has decided not to invest in analysis and mining of dark data by definition, concerted efforts by third parties to exploit its value represent potential losses of intelligence and value based upon its contents.

How Data Data can be managed

While Dark Data is not necessarily a bad thing, we cannot completely get rid of legacy data. The goal should be having set of strategies and process in place along with using advanced tools to manage and organize in order to keep risks and costs associated with dark data at a reasonable limit. Few approaches that could help

• Ongoing assessment : Dark data should be recognized and should be subjected to reconnaissance. They should drive research into new tools and technologies to help extract value from such data. Yesterday's dark data might be tomorrow's shinning source for insights with current advanced technologies and analytic techniques. Performing regular inventory provides understanding of where data resides and how its stored, how its protected and what access controls are in place to help maintain its security for regulatory and compliance.
• Auditing and Monitoring: Most organizations of any size conduct periodic security audits, evaluating risks, exposures, incident response and policy. Dark data needs to be folded into this process and visited sufficiently often to manage risks of exposure as well as potential loss or harm. As with auditing proper monitoring of the dark data should be enabled for better insights on who is actually accessing the data and the purpose it is being used.
• Ubiquitous Data Obfuscation : Any digital asset with potential value and possible risk must be stored in encrypted form, whether on the organization's premises and equipment or elsewhere in the cloud. No dark data should be readily accessible to casual inspection, under any circumstances. Strong encryption with proper access controls will make it difficult for those who want to obtain dark data for whatsoever purposes.
• Retention polices and safe disposal : Proper retention policies should be in place and after which it should be disposed depending on requirements and subject to government guidelines approved methods.

Given the right appreciation for both potential value and possible risk, organizations can deal with dark data to balance one against the other. By digging into its dark data collections, keeping those holdings whose potential value outweighs its risks and deleting those whose risks outmatch its potential returns, an organization can be sure it's proactively keeping what may prove worthwhile in the future while dropping what may poison future productivity or profitability.

On the other end of the dark data spectrum, organizations though build up dark data steadily may also be missing out on some great opportunities by not using dark data effectively. With advancement in tools and technologies, there are some untapped potential inside that mass of information as organizations thinks of their non used stuff. In the end organization should strike balanced approach to find the information that is truly valuable amid a giant mass of unstructured dark data.