Big Data Fabric - a Reference Architecture

In my last blog, Big Data Fabric, I quoted Forrester, which defines a Big Data Fabric as

a platform that helps users discover, prepare, curate, orchestrate, and integrate data across sources by leveraging big data technologies in an automated manner

I also proposed that a Big Data Fabric must make the process of generating insights from data easy, fast, trusted and available to all. This requires a number of capabilities, from ingesting and integrating data, to cataloging it, analyzing it and visualizing it, as well as capabilities to make sure it's secure, trusted and so on. So, what would all these capabilities look from an architectural standpoint?

The picture above is a Reference Architecture for Big Data Fabric. It's a logical architecture that can be used to instantiate particular solution architectures, using products from any combination of software and cloud vendors. It's deliberately fairly detailed whilst also being at a high enough level of elaboration to fit on a page. The idea of this Reference Architecture is not to propose that every organization using a Big Data Fabric will need every component of the architecture. It's to be used as a reference guide for Enterprise and Solution Architects in the space. If you'd like to see a larger version, see the slides here Big Data Fabric Reference Architecture.

To walk through the architecture, from left to right - various sources of data are integrated and ingested into a data landscape, a set of data repositories optimized for different kinds of data. The data can then be analyzed by a set of analytical processes and is supported by a number of required services such as cataloging and security. All of this is controlled by workflow elements. The fabric is managed and governed and all of its components are hosted on some IT infrastructure. Three sets of users interact with the fabric - consumers, governance staff and operations. Interactions with the fabric occur through various channels.

I hope readers find the Big Data Fabric Reference Architecture useful. In my next blog, I'll be presenting a Big Data Fabric Capability Maturity Model which will help organizations formulate their journey towards becoming The Continuous Enterprise.