Managed Migration Middleware

There are multiple ways as we can classify a migration project.

Today I want to talk about two types of migration projects in regards to the way as the involved parties approaches it.

Standard Source or Standard Target projects

Happens when the responsible for the data migration owns the source or target application and have been involved in multiple projects where the data is extracted/loaded from/to its system. In many of those cases, commercial applications have a built-in procedure (batch import/export, APIs, data stream, etc.) to extract the data from its own database to a standard format or to load pre-formatted data into its own database in its totality or in parts (specific entities/tables or filtered data), as a full/partially automated process. It is an advantage considering the impact of development/test of a migration solution.

In this type of project one team will be responsible for either extract or load the data using standard/pre-defined source data extraction or target data loading process that is already pre-defined by the source or target application owner. Other teams may support this partner but it is always one of the two ways: the responsible team will extract (or “export”) data in standard way and creating a process to ingest it in the target system (it usually requires transformations and the owner of the “T and L” of ETL process is the target application) or it will read the data in the original source format and prepare the data to be loaded in a pre-defined format (transformations are also required, but the target application have a pre-defined process for loading data – the “L” of the ETL loads data in a “predefined input format” for this process).

Unfortunately, it is generally limited to smaller projects or cases where the application capabilities are mapped from one source application to one target application.

Managed Migration Middleware projects

To explain the Managed Migration Middleware, we need to look at the reasons for its existence. Whenever the system capabilities are mapped from multiple sources to multiple targets it increases the level of the overall complexity and the risks involved on migration. Some critical factors are:

·         Number of partners/areas/users who will need to access/process data

·         Increased complexity on data transformations

·         Time required to process the whole dataset

·         Coordination of the E2E process

·         Customer impact

·         Operational impact

Each of those factors brings risks that needs to be properly addressed as part of a well-defined migration strategy. In order to properly address this complexity and mitigate the risks it is very important to introduce some critical elements to the project like a consistent data profiling, a clear interface between the teams (a Data Transfer Protocol), a data reconciliation (between different sources, different targets and from sources to targets), data validations with clear quality gates, fallback process and data governance to name a few.

Considering all those critical factors, the risks brought by each of them and the elements introduced to mitigate those risks you will need a framework that allows you to have end to end visibility and control of migration process. This framework is a Managed Migration Middleware.

The owner of migration process is also the ultimate owner of the Managed Migration Middleware and it may be implemented by the team performing the extraction of data from source systems, the one loading the data on target systems or both. It is very important to identify the boundaries of each team’s activities and ensure there is no gaps on it.

Having the right partner or team, with experience on migration projects, is another key factor for the success of the data migration project. The right team/partner will influence the strategy, approach and tools used to have a migration without losing data, attending all the business requirements and protecting the customer from any outage or unwanted impact.

Let me know in the comments if you want to read more about Managed Migration Middleware or how to define the best migration strategy!

Don't rely on "good luck" for your next migration!