Integration - doing it right

In today's connected world, its impossible to imagine a standalone application which does not have integration needs or capabilities. Applications require connectivity with other applications for variety of purposes but essential function of the integration is to facilitate data flow between systems.

Even though Integration plays such a critical role for all applications, it is an afterthought for many implementations, which results in a wrongly designed system. This is mainly because Integration by itself has no meaning for an organization unless it is a consulting/services company providing integration expertise. For a bank or insurance provider or logistics provider or healthcare provider or a manufacturer, Integration by itself is not a business fulfillment application, even though it is a mission critical application.

Integration systems are never built the same way any company would build their fulfillment systems. Majority of Integration system end up getting built "interface by interface" while trying to fit in important aspects such as reusable components, API's, Micro-services, B2B, EAI, component based architecture etc.

When solving any organizations integration problems or establishing integration for future, below are some of the important aspects which needs to be considered

Use the right tools

Building of reliable and useful integration system starts with choosing a methodology and a tool which can fulfill your current as well as future integration requirements. There is no one size fit all, every methodology & tool has its own benefits and drawbacks including in-house middle-ware based integration, hybrid integration, PaaS, IaaS, or SaaS. It is very important that a careful assessment of current and future integration needs is done. Each requirement should be weighted for its importance and then strengths / weaknesses of different tools & methodologies should be tested against those requirements. This approach will ensure unbiased selection of right methodology and tool.

Most enterprise grade fulfillment applications have their own integration technology built into their system, which is often product specific. It is very important that integration tool you choose is extensible to cover for complex & non-standard scenario's. You must build a extensible and loosely coupled solution to reap the full benefit of integration system which delivers.

Building integration system as a product

Every organization where I have stood up integration system had one common theme and that was to build integration system as a whole and not interface by interface. Building system as a whole would mean your outlook should be as though you are building a product and not merely building a interface or implementing a project. By following this approach, we could produce awesome results every-time and not just technological debt was settled but also turn around time was reduced by up to 75% in some cases. In the next article I will explain the key consideration of building a integration system as a product and what it actually means. For today's topic just remember not to view or build integration systems interface by interface or for a project.

Architect an implementable system and implement a supportable system

This sounds very basic but often overlooked and this is not just true for integration system but true for any application. However it is critical for integration systems that transactions can be traced quickly, failure can be reported at real time and recovering from failure is easy. Integration systems are highly changeable systems and the one's with very high possibility of transient errors but still this aspect is often missing or overlooked or never rightly provisioned, when architects design the system. Integration systems are arbiters between source and destination systems. Traceability of data flow and ability to pinpoint it based on business key is crucial for a sustainable and maintainable system which has right tools for support resources. Architect the system which your support teams can maintain.

Non functional requirements (NFR's)

While lot of attention is given to functional requirements of a project/product, non-functional requirements are no less important and they also impact systems architecture as well as how its implemented. Non-functional requirements such as up time, track & trace, security, GDPR, data archival & retrieval, performance (read/write), language support, etc are equally important for an integration system.

Buzz-words

Hybrid-integration, API or API economy, Digital transformation, Micro-services, and cloud technology etc are some of the many aspects of Integration, which are dominant. These technologies or approaches or strategies are not one size fit all. These will take care of your technical debts and make sure you have latest and greatest tools & connectors available to support today's and future growth. It will also make your organization more agile and easy to integrate. It is very important that companies take up digital transformation initiatives but must also evaluate the benefits and return on investment carefully to avoid disappointments.

Its important to understand the context in which these technologies will be used and problem they will solve. For example creation of an API is only addressing mechanism of data movement not the business processes which make API's useful. So just converting data movement aspect from say sFTP to a REST API is not going to resolve business problem associated with the integration.

In the end, its very important that a organization has holistic approach towards the integration. Many times application teams end up making integration decisions due to the void left by absence of a proper integration solution or endless wait for a integration expert. This results in, usage of tools which are not fit for purpose and cause overall negative sentiment towards systems integration.

Even if your organization has endless supply of resources (monitory, system and employees), its in your benefit to create a stable and sustainable integration strategy as early as possible. Requirement to integrate your system/product/application with other systems (A2A & B2B) is only going to grow in future.