Microservices surviving the Enterprise data layer

Technologies facilitating the adoption of micro-service architectures in the Enterprise IT, where it hurts more: the data layer

A "plastic" polymorph feature for a smart anti-corruption layer

[Disclaimer: the views expressed in this post are my own and do not necessarily reflect the views of Oracle]

0. mission statement, first ;)

The adoption roadmap of MicroServices architecture in the context of non-greenfield Enterprise IT is a painful trouble as it requires to cope not just with the "new" but also with its sustainable relationship with what layered in IT landscape (technologies, practices, data): Enterprise companies have deployed over decades a number of web, customer-facing applications (e-commerce portals, CRM, caring, etc) that won't be ported in a single step to the new paradigm, while the (digital) customer instead expects a uniform experience across the diverse services on the web: the Products, Orders, Shipping, Records, Assistance entities he interacts with shall be as consistent as possible whatever he lands on a brand-new Microservices oriented e-commerce service or whether he's dealing with a traditional Contact Center leveraging Siebel CRM, or browsing on the flagship JEE portal.

For IT this means keeping key IT entities and functions (as for example "Customer", "Catalog", "Orders", "Shipping", "Contacts", whether shared or duplicated on new or on traditional frontends) to run

• as "consistent" and as "insightful" as possible (which means reciprocally timely up-to-date for transactional data, and timely accessible for the analytics to be more and more meaningful and actionable
• as "seamless" as possible, despite their different representation model and technologies (mostly relational on one side, mostly non-relational on the Microservices side)

Despite being a data design and integration concerns, most publications still treat Microservices-oriented architecture as an "island", with no relationships with any other "traditional" system. And even if Domain Driven Design is generally considered the leading discipline to shape Microservices' functional decomposition, replica and partitioning dimensions, still I hear and read a number of misconceptions on very basic principles of properties, design and technologies of state management of the new architecture, that in turn muddle the conversation on integration design with that RDBMS-centric architecture which fuels the "traditional" transactional and analytical systems of Enterprises: if you heard statements as that existing relational data platforms are not adequate as a persistence layer for that hierarchical “rich” modelling of Microservices, or that – on the opposite side of the array -  “post-relational” architectures cannot provide any of ACID behaviour and properties that are indeed still desirable for some critical Enterprise customer-facing services, or that a Microservices architecture does not cope with any ACID properties, misquoting the actual meaning of "Consistency" with Brewer’s definition of “atomic consistency”, you'll have a feeling of what I mean.

Another side-effect of this level of debate, which more directly affects my daily duties as a technology advisor of some large Enterprise IT, is that in this situation our customers (Oracle customers, in my case) likely miss the chance to get a clear understanding of “what’s in” their current technology portfolio they can leverage to facilitate the inclusion of new paradigms into their landscape, and its architectural value, which is a key ingredient for a sustainable modernization too, as we'll see later.

So, with the upcoming series of posts, I am sharing my personal perspective on a number of data-related subjects and prejudices, sorted among those I believe most relevant, taking chance to draw, step-by-step, a prejudice-free architectural blueprint that puts together the new paradigm and the existing transactional and analytical areas, and to provide a technology view of what customers I daily deal with (Oracle customers) can leverage of their install base to achieve a sustainable and realistic integration roadmap.

Today we'll start trying to make practical (technological I mean) sense of "timely" seamless integration between new and traditional transactional and analytical foundries, talking about some existing features of Oracle install base as smart anti-corruption layers.

“plastic”…what? 

One of the architectural principles that Microservices somehow brought back to the attention is polymorphism (e.g. both in languages and in data design) which is not just at the base of Object Oriented Design, but –mainly- a concept used in Biology to model the strategies of adaptation of native species to changed environmental conditions, and of reciprocal relationship with new “disruptive” entrants... Sounds familiar? Let’s see whether under this “biological” perspective we can find some inspirational principles for our aim....(disclaimer: I’m not even close to be a biologist, so get please the raw metaphor and try to fly over the details… ;)

In Biology, the "morph" assumed by an organism in its niche is defined as its "phenotype". The phenotype is the result of its genetic capabilities (structural) and its "plastic" behavioural abilities too (hence those abilities that may not be permanent during the life cycle), both contributing to shape the best “form” the organism can leverage to respond to environment pressure. 

Now, when in an ecosystem different phenotypes (morphs) appear within the same species, that’s polymorphism. Such variety in phenotypes, can be then determined by some genetic variance (“genetic” polymorphism) or rather as a result of  "plastic" adaptation capability in that species; in this second case, that’s a specific kind of polymorphism defined as “polyphenism”, or phenotypic "plasticity": different morphs appearing on top of same genotype. 

So…what?

Let's for a moment keep playing with this metaphor looking at Enterprise IT as a kind of ecosystem, consisting of two main “niches” (it’s a deliberate over simplification, thus omitting any governance and planning function): the Transactional (that of "actions") and the Analytical (the "decisions" area)

In the former, some stable environmental conditions over decades (a "bounded" market - not a web-scale one - a bounded community of customers and users, a IT delivery model, "monolithically" shaped after the "siloed" IT organizations), have shaped "species" (Systems and Data) with a specific "morph", genetically geared towards ACID behaviour (properties that has been mostly delegated to relational databases capabilities):

• transactionally-wise, aiming at strict and "atomically" consistent behaviour of functions and coherence of their state across the whole application domain (most of us still are comfortable when a web UI keeps our tran$actions atomically consistent with our orders or bookings...)
• organizationally-wise, aiming at a "vertical" cohesion of the application development and deployment design (most IT organizations still are comfortable when they can “see” the application domain as a single shape they are testing and operating, as a single deployment unit, attributable to a specific project’s budget, with a specific accountability and responsibility chain, etc etc…)

The sudden raise of a digital, web-scale market - a competitive arena for Enterprise companies too - has put a growing urgence indeed on scalability, resilience and agility, relaxing the value of strict consistency, thus creating the condition for the raise of a new IT “morph”, the Microservice, based on a structurally "genetic" different model of distribution, decoupling of both services and their data’s representation (a “BASE” genotype...although on its meaning I'd like spend some time in a separate post, as to me is one of the most misunderstood concept I read about), in turn enabling scale of agility, speed, frequency in the Delivery cycle as well (leveraging DevOps practice)

a Polymorph landscape

Our Enterprise landscape is now polimorph: two "genetic" variance of the same species (applications). However the two morhps are not living in isolated niches but rather competing for the same resources (as said, Enterprise entities as "Customer", "Product", "Order") and both need survive the competition (for the sake of business continuity). In short, their relationship must be "cooperative", rather then competitive, where access to reciprocal resources shall be at same time "timely" and "seamless" but without requiring reciprocal modification in respective original structure.

Here comes "plastic" polymorphism to play.

"plastic" polymorphism at data level...

Let's try now to shape the meaning of polymorphism previously given, to data: data polymorphism (multiple data representations) can then either result from a "genetic" variance (many different "single modal" data structures) or rather from a "plastic" capability of each single genotype (where a structure - whether relational or not - can project different representations if it's internal structure), so in this latter case

• for a relational system (SQL and ACID) to project its "structure" in a NoSQL, “schema-on-read”, and REST morph
• and, reciprocally, to project a SQL “structure” into Collections of post-relational datastores

Among the two options, "plastic" polymorphism is the one enabling a true seamless access to reciprocal resources, because allow both morphs to adapt reciprocally, hence to play the role of a valid anti-corruption layer between the two

Oracle RDBMS customers can find such "plastic" capability in alternative ways:

1) either as the ability to project its existing relational structures as "non-relational" morphs,

• both as REST JSON one: through Oracle Rest Data Services (a REST-SQL gateway embedded in the relational db, configured and deployed “as code” via SQL)
• and as Message / Stream one: leveraging polyglot Change Data Capture technologies (as GoldenGate for BigData or HotCache for Coherence In Memory DataGrid). We'll see in a while why this message-oriented morph is also key for our aim...

or, reverting the perspective, projecting a structure (a SQL access) onto non-relational Collections (such as with SODA, Simple Oracle Document Access API)

...and in the "Big Picture"

To appreciate the value of a technology we shall evaluateits architectural value, otherwise it's just a "feature"...so we shall understand how, where and why it will provide value to the landscape: this exercise in our case would require a blueprint of the Microservices "niche", so let's standby a while and succintly draw one (at a very high level...): by now - for the sake of “getting the big picture” at first, I’ll fly well over the details, giving for granted a number of properties and principles that - I swear - I’m going to deep dive in greater detail in next posts...

building blocks of the Microservices "niche"

As we known, micro-services represents decomposed parts of a broader application domain, self-contained in terms of functionality, representation and persistence of their status, and of their lifecycle (hence the terms of "sub domains"). These services can be individual functions or rather set of functions (hence "applications"), serving as backend service for some channel-specific frontends developed on specific frameworks as Angular or Oracle JET, or low-code platforms as Oracle's Visual Builder or Chatbot.

Whatever the aim, in order to be independent and self contained, they must define or be provided their own infrastructure resources too (its runtime interpreter, its Operating System - both packaged nowadays as a "container" - Storage, Network, etc) and its non-functional *ilities (load balancing policies, replica and fault tolerance rules, access and authorization, etc): these "definition" is a contract with underlying operating environment (as stated in 12 Factor Manifesto) that will actually look after and "instantiate" them when application or function gets deployed.

This mean that, depending on the level of abstraction provided to the developer, a Microservice runtime can typically be classified as: a Function-as-a-Service platform, as FNProject (the developer pushing a single function to the platform, while the platform implementing and looking after any further resources needed underneath), as an Application-as-a-Service one, as CloudFoundry or Oracle's Application Container Cloud Service (the developer pushing an application, while the platform implementing underneath resources), or as a Container-as-a-Service one, as Kubernetes or Oracle's Container Engine (the developer pushing a whole Docker Container to a Registry, the platform pulling it, creating any other infrastructure dependency and looking after its lifecycle).

Although functions and services are self-contained and decoupled components, however, being sub-domains of a broader application domain, they can deliver business wide functionalities only by collaborating with each other, thus establishing relationships, sometime explicitly (e.g. one upstream service consumes a supplier's "published language" - an API) other times "opportunistically" (e.g. one downstream service subscribes to a message type, that can be eventually published by some upstream services he does not possibly even know): that means that our Microservices platforms has to provide, on top of the orchestrator, two kinds of fabrics, a RPC-oriented (REST) and a Message-oriented (a distributed broker as Kafka):

• the RPC oriented (REST), with network and DNS for services reciprocal discovery, weaving the API-based "consumer-supplier" pattern (e.g. for "has-a" relationhips, lookup, enrichment): imagine an OrderManagement services looking up its ProductItems from a decoupled ProductCatalog service. For external, public consumers of some internal, private suppliers, then the weaving is further delegated to an API Platform (like APICS) so as to publish API (and made searchable for prospect consumers), and enforcing protection policies (both in terms of Security – e.g. with Authentication, Authorization, Auditing, Confidentiality, Privacy, etc - and in terms of Back-Pressure – e.g. with Throttling, Timeouts, SLA management, etc). Instead, for external suppliers (typically Backing Services) of internal consumers, a Service Broker technology would ensure the binding of supplier's configuration (e.g. access credentials, connection url) in a way transparent to consumer's implementation. This is particularly the case of those "backing services", “per-service” polymorph persistence datastores (whether relational or post-relational, SQL or REST), such as Oracle NoSQL, MySQL, Coherence IMDG or the same Oracle RDBMS, available as Docker images. However, the complexity of relationships weaved through this kind of fabric, can easily grow to a degree of complexity that this fabric is usually managed by a Service Mesh as Istio, to keep a declarative control over load balancing, security, circuit breaking, fault management, but also - which is another hot subject for such distributed architectures - in terms of telemetry and distributed tracing of a mesh of relationships that could likely grow into an Operations' nightmare.
• the Message oriented, for the "publisher-subscriber" pattern, where Microservices can broadcast their “domain events” to cascade side-effects of its own state change to remore components: to a "child" Entity owned by a remote Service (imagine a change in a customer's shipping address from an OrderManagement service that has to be cascaded to the "root", upstream CustomerManagement service - thus keeping eventual consistency between the two), or to cascade it to a peer Service (thus keeping a “choreographic” integration between the two, a "end-start" relationship: imagine for example a successful Order submission that needs to trigger a Shipping), or for "caching" locally some state broadcasted by remote Entities (e.g. as to build a "materialized view" of a remote state, as the "query" view of a CQRS pattern - imagine our CustomerManagement service that business requirements prescribe to provide the listOfCustomerOrders too - or maybe just for logging business or technical events). An important caveat of this layer is its strict atomicity requirement: obviously the event (and its message) shall be published only when state gets actually changed in the data store: we'll see in next dedicated post which patterns - as Event Sourcing - has been designed to handle this need, but for the sake of our conversation, just keep it in mind for now

Finally, for those Services representing a more complex flow then a simple 2-step relationships, a possibly long running, conditional flow, atomically fault tolerant across diverse Services (read: where all modifications shall occur in an “all or nothing” way across different Services, as in the classic case of TravelBooking), our "niche" will hopefully include some kind of lifecycle support for for SAGA orchestration services, implemented on purpose-oriented SDK as FN Flow

In summary, our Microservices “niche” technology basic equipment would roughly layered as

a “plastic” cooperation 

let's now fasten the hanging threads: by plugging the "plastic" morph capabilities of existing Oracle relational data platform, right into Microservices fabric, our Enterprise transactional reservoir can now "morph" as a "timely" and "seamless" member of its relationship mesh:

• an upstream member of a "has-a" relationship, plugging Oracle Rest Data Services as a REST anti-corruption layer into Microservices' API public or private RPC (REST) pipes: following our previous example, an OrderManagement microservice would now be able to switch its "ProductItems lookup" child entity seamlessly right from the Catalog still running in traditional "niche"
• but also a downstream member of asynchronous, "cascading" relationships, plugging GoldenGate for BigData as a stream-oriented anti-corruption layer into Microservices' “domain event” fabric: a change in customer's address operated through traditional CallCenter, will now be easily cascaded on CustomerManagement microservice of the new "digital" frontend (estabilishing eventual consistency between the two), or that a customer complains recorded in the traditional Siebel could now trigger a Marketing action from a CommunicationManagement microservice on the other side of the block (in this case, a "end-start" choreography)

We mentioned however that "Domain Events" shall be triggered with strict atomicity with change in Microservices internal state, but not every NoSQL stores can offer distributed transaction support (note: in this case strictly restricted to Microservices internal store and the Message broker...something like making sure that - under any kind of fault or error circumstance - the "order submitted" event is never forwarded to Shipping unless the Order is safely persisted...). Hosting Document Stores natively into an RDBMS as Oracle (leveraging SODA Collections) can then provide an easier path towards strict Atomicy decreasing complexity in implementing Event Sourcing patterns (again, that's out of scope for this post, we'll dig into it in an upcoming one)

Our resulting integration would be look like a "cross-border" relationship fabric where access and manipulation of Entities' state can occur seamlessly across the two "niches" leveraging REST, Messaging and SQL anti-corruption layers

a "plastic" integration with Analytics

At the beginning of this post, I referred "timely" and "seamless" properties for analytics insight as well: let's see if and what kind of "plastic" polymorphism technology can be exploited there, and how and where it can help.

Modern “All Data” Warehouses already rely on a polymorphic approach: at its core, it's already a "data lake" of integrated polymorphic stores (streaming vs static, relational vs. non-relational, structured vs. unstructured, etc) and management technologies (Kafka, Hadoop, Spark, Relational)

And - as for the "timely" property - it can already rely on streaming solutions, both from "post-relational" systems (streaming Microservices Event fabric right into its Kafka store), and from traditional transactional one (leveraging “polyglot” CDCs such as GoldenGate for BigData), 

But, regarding the the "seamless" access (which in turn contributes to "timely" factor as well), current approaches to “all data” manipulation inside the Analytic lake are still somehow limited in scope and mostly geared towards copy or move across their native areas: sometimes because desired algorithms are only available in specific engines, other times because data presentation shall be funnelled into a relational, SQL based engine

in other terms: even if you can stream your Customers complaints from traditional CRM or from a Microservice Caring service, promptly right into the datalake, it can still some take time - and design effort - before the right analytical engine can produce an insight for a Data Scientist or for a user of the analytical frontend

A “plastic” morphing technology instead, enabling an immediate seamless access to different morph, can then be key in both increasing the depth and promptness of navigation, discovery and visualization, and in reducing the time of such phase.

In the analytical world, such technologies are converging towards "projecting" a uniform SQL structure to 'any data' repositories, either from central hubs as Oracle RDBMS (in the case of BigData SQL) or from a Spark one (as in the case of SparklineData’s SNAP OLAP Cubes). In Oracle ecosystem, a similar approach can be even leveraged for "tactical" needs (as a limited scale reporting) using Table metaphor for Oracle NOSQL or plain SQL to access JSON Collections hosted on the RDBMS

Plugged into the Big Picture, here is how our plastic polymorph architecture will finally look like: distinct and polymorphic niches (to preserve reciprocal evolution) leveraging REST, Messaging and SQL anti corruption layers to keep still timely and seamlessly connected.

in conclusion

Adoption of Microservices paradigm in the Enterprise IT starts at integration, for its adoption roadmap will be inevitably iterative and incremental. Today I represented - in a rather provocative reasoning, I know ;) one of multiple options, a technology feature that - when in the big picture - can represent a quick way to implement a valuable anti-corruption layers between niches that inevitably need to grow in parallel. In the next post I will indeed figure out some more concrete example it, shaping some a more concrete implementation inspired by one of available example demo app, to see those principles in action.