Ab Initio Lambda - Overview

Nathan Marz, along with James Warren wrote the seminal 'Big Data' book a few years ago describing a new architecture that deals with the volume and velocity of our modern data world. (ISBN 9781617290343).

Our modern data world has one stand out, a compelling business driver, and that's digital transformation.

This series of posts describes the layers to enable your digital transformation journey using Ab Initio Lambda.

The common Lambda architecture

Nathan's original common Lambda architecture describes the acquisition of new data flowing through the Speed, Batch and Serving Layers to answer questions from the business.

The Ab Initio Lambda architecture

The Ab Initio Lambda architecture is an expansion of the original Lambda architecture to encapsulate the rise in the maturity of the digital revolution. The global patterns we observe in many a digital transformation are included and keeps true to Lambda architecture by adding layers.

Acquisition Layer Overview

Today we use Ab Initio's Acquire>It extensible framework to serve any data, at any time, to both the Real-Time and Batch Layers.

Built upon global patterns for data acquisition at scale and fully configurable, you'll find an order of magnitude increase in productivity and reliability. That’s the benefit of the underlying graph model – metadata drive and parallel from the get-go.

Real-Time Layer Overview

With a combination of Continuous>Flows, Active Data Cluster and the Business Rules Environment, we support the rapid feedback and sophisticated decisions required by modern data transformations. At a technical level, this means data and event processing must be low-latency, in-memory and materially scalable.

Batch Layer Overview

The Batch Layer's focus is on serving business intelligence activities, and the majority of us will find comfort in this layer. The customers want a massive number of observations to make informed business decisions.

How does it get there?

As one of the targets within the Acquisition Layer, we use Ab Initio Acquire>It extensible framework to deliver both the raw, immutable data along with the cleansed data. Ab Initio's Data Discovery and Semantic Discovery frameworks give us the meaning and association of data and information. The historically massive transformation to deliver conformed and authoritative data is relieved by leveraging Spec-to-Graph. Finally, the Semantic, Data Marts and Business Views are also quickly and effectively performed using Spec-to-Graph.

With all that data your customers want a scalable, robust, easy and secure way to obtain it, right? This is where Query>It comes in, Ab Initio's high-performance SQL engine.

Interface Layer Overview

It’s best to describe the Interface Layer with a business context, later in this series we see a Customer Interaction Platform and how Ab Initio fits and feeds back data into the Acquisition Layer.

Governance, Audit and Control Layer

We add the Governance, Audit and Control Layer because without it we have data anarchy, siloed risk and little security.

This layer provides a holistic view of our architecture in-flight. I'll discuss how we harmonise Operational metadata from Control>Center in conjunction with both Technical and Business Metadata within the Metadata>Hub to achieve this in a later post.

In the next post, we look at the Acquisition Layer in more detail