Integration Platform on Cloud – Essentials

In my earlier article on “Design Patterns for Next Generation Data Platform on Cloud”, I have attempted to encourage enterprises to adopt Data platform as part of their digital journey. While industry moving towards building the data platform, reimagining the integration layer and define new integration platform become necessary to reap the benefit from the investment. Capabilities associated with the integration platform are expected to drive the modernization of enterprise integration landscape.

This article is focused on the essential capabilities enterprise should target while planning for the next gen integration platform on cloud. Enterprise Architects, CIOs and CTOs can refer the article if they are planning to build their cloud native integration platform.

Approach towards integration on cloud is lot different from on-premises. As per my experience from last few years, there are 3 essential capabilities that should be part of the build of next generation native integration platform. In next section, such capabilities have been described with sample use cases and design patterns with Azure Reference Architecture. 

1.  Responsive:

Standing on the post digital era, responsive platform powered by realtime event driven architecture is new normal. Information should be propagated to systems near realtime to improve customer experience, avoid adverse incident, serve the transient demand etc.

Use Case:

• Collaborative Business that runs with wide range of affiliates like mortgage
• Lead management across CRM and other end user service
• IoT based manufacturing operations

Key Patterns:

• Asynchronous: Message and event based patterns like Topic/Pub-Sub, Queue/P2P, Event streaming (IoT)
• Synchronous: API based request-response pattern

2.  Data Platform:

Streamlining the data flow from various heterogeneous source systems to different target systems is critical to avoid issues with data integrity and data consistency across organization. It is observed that migration of applications to cloud is taking longer than expected due to multiple such integrations between systems. Building this capability will enable enterprises to decouple the data from both source and target applications.

Use Case:

• Consistent data capture - sourcing data from internal system, 3rd parties, real time sources (such as IoT devices), CRM, ERP and new customer applications like inventory data, product information, flight data etc.
• Build analytical insight to improve operations like sales, customer management etc.
• Use centralized data platform as source of truth across enterprise and exposing using appropriate RBAC.

Key Patterns:

• Ingestion: Stream and Batch ingestion to central Raw Data lake from system of records
• Consumption: Operational DB, API endpoint, Reporting/Analytic DB

3.  Service Platform:

API-led point to point integration has established its value in industry long back, however enterprises still not able to get it commoditized within their companies. Establishing a strategic API platform for ecosystem leverage is key in today’s digital age where APIs to be designed as products, business channels and even business models. Adoption of API culture and building API Management platform are 2 key foundational blocks of such platform.

Use Case:

• Centralized APIs for master data like Product, Customer, etc.
• Exposed APIs for 3rd party collaboration
• Open Banking APIs

Key Patterns:

• API Management including API Gateway, Management Plane and Developer Portal

Further extending the capability to reach the next level of maturity, enterprises should aim to achieve at least the following:

Integration Catalog: Integration is backbone of any IT system today and having a complete catalog benefits enterprise in many folds. While it helps understand the relationships among applications, it also serves the purpose of reusability of the already built integrations and the endpoints avoiding many problems stated already.

Self-service: Exposing the above catalog using decision tree can empower the portfolio and citizen developers and increase the reusability of the base foundation layer.

As we see around, every successful digital enterprise has harnessed integration as key enabler of their digital journey. Building a strong strategy and establishing the integration platform cannot be overlooked any more. Begin your journey with small steps toward building the above capability and unfold the new value in every step.