Data Migration Fundamentals

Part 1: The necessity of data migration. (Part 1 of a series)

In our IT driven world, data is the new gold. Data (along with it's smarter nephew called Information) determines more and more how successful your business can become. Only organizations that manage their data effectively stand a chance to survive in a globalized world. Customer and Sales data, Vendors, Assets, Products and Inventory are real life entities that are stored in your information system in the form of objects, tables and fields. In a sense, these real-life entities "become" data in the day to day business. If properly managed, your data propels you into increasing profits. If neglected, it may become your downfall.

As it stands, most companies don't have their data in order.

Think about it. Without reliable information about sales, it's impossible to design a new commercial strategy. If the stock numbers in your system are not up to date, it's impossible to deliver products on time. But even internal data processes can be problematic. Data is strewn across a complex landscape without any consistency or consolidation. The data team has little control over what data flows into the system, and who is responsible for maintaining its quality. Data models and quality criteria are not documented, or, reversely, exist in many different versions that are all obsolete. The master data organization produces a lot of paper, but does not really add value. There is no proper cleanup process in place, so that existing data problems are not addressed effectively. Worse, when problems are identified, no one feels responsible to act. Stakeholders hide behind procedures, teams pull up bureaucratic obstacles, and no one takes ownership or makes decisions.

In many cases, the only solution is to clean up the chaos by starting a new data project. Often this is in the form of a business improvement initiative, whereby the data is cleaned up and moved into an integrated data platform. Data is analyzed, harmonized, de-duplicated and cleaned up. Then its loaded or entered into the new system. Viola, job done. Unfortunately, it's not that easy. This is because defining business processes and setting up the new architecture is not enough. A supporting activity also pops up that happens to consume up to 40% (according to Bloor research) of your project budget: data migration.

First let's come up with a clear definition. What is data migration? Although understanding has increased over the past years due to the whole big data trend in IT, many businesses still lack a firm grasp of what data migration actually is:

Data migration is a one-off sub-project in the overall project with the objective to move existing data records from a source system into a target system. An important part of this move is the transformation of the data records into the target data structure. The move is almost always accompanied by source data profiling and subsequent cleansing, so the resulting target data will meet the quality standards.

This definition is by no means perfect, and better ones are quite possible. At least we have achieved to define the core activity: moving data between IT systems or landscapes, at the same time changing its shape. With this understanding, we can exclude some activities that can be related to data migration, but are not data migration, being:

• Data Management or cleansing of productive data.
• Moving data via operational interfaces.
• Copying data sets between servers.

Now some caution should be applied here. Although the activities listed above are not data migration themselves, they are closely linked. Sometimes these activities are a prerequisite for, or a consequence of, the data migration project. For instance:

• When new data is moved to the new system, data stewards have to be re-trained to handle the data.
• Sometimes temporary or even permanent interfaces are required to make the data migration work. This is the case when the old and new system run in parallel, or when the old system is phased out gradually rather than in a big bang. (Data objects will therefore be managed and synchronized between the platforms)
• Data migrations are often tested several times before the cutover phase. To be able to test all the objects, a copy of the production system on a separate server is made available for these tests.

These are the main characteristics of data migration. To close this article, allow me to list 4 key success factors that determine the outcome of your data migration project.

1. Data Migration must be managed as a (sub-)project, and good communication is essential.
2. Data Migration is a business activity (with important technical components), and the business must be involved in every step.
3. Data Migration is about measurable data quality, so the objectives of the data migration can be measured and appraised.
4. Data Migration requires a sufficiently skilled and high-performance team, with a right mix of technical, functional and project management expertise.

Often, data migrations go wrong. This is a reality in many projects. The reason they go wrong is almost always tied to one of these four success criteria. Therefore, in the next articles in this series, I will address these criteria one by one, followed by an approach to make your data migration project a success.

Author: Lars Oberink,

Codex Dataensis - Our mission is Data