Reflective Data Modeling

‘Bend, don’t break,’ as they say. We’ve all heard it repeated ad nauseam, this adage that seemingly applies to everything from warfare to skincare. It would have long been a cliché, if it didn’t just feel so instinctively sound. But given the universal appeal of this principle, why is it then — we ask looking at IT systems in use today — it appears not to apply to data modeling?

This article is the second installment in a series outlining the principles behind the Reflective Platform, a master data management platform for building data models that never go stale. Read the mission statement for a brief overview.

Most, if not all, enterprise-scale IT systems, trusted to handle key data by businesses around the world, rely on data stores that don’t adapt well to change. These systems often have a relational database with a fixed set of tables and columns, or they might employ a NoSQL database with or without schemas. Some systems even offer ad hoc data modeling that customers — sometimes guided by consultants — can use to build and maintain models of their own.

But whichever the flavor, changing the data model in any of these systems is quite an ordeal. Data migration from one version to the next is never without its share of problems, and once completed, the history of what the data used to look like is usually gone. This strategy is flawed, in that it introduces a series of discrete breaking points, across which data lineage is lost. When each of these systems was originally designed, the data model was in all likelihood carefully chosen. But unless the data store used to hold the data offers a means of gradually evolving the data model over time, it will inevitably grow too far out of date, and the introduction of a new breaking point is needed to ‘fix’ the data model.

At Reflective, this has led us to the realization that a new way of modeling data is necessary, in order to create a reliable platform for expert systems. We call our own approach to this Reflective Data Modeling, and it embodies the following properties:

• Data is stored as objects with attributes that are strongly typed via schemas
• Schemas are also objects and as such may evolve over time
• Objects may be tagged with a new schema
• Previously recorded information is never lost

Central to this is the bitemporal nature of the Reflective Platform. We will expand further on this topic in a later article but simply put, it means that a fact can be recorded independently of when it becomes active or valid.

Data as objects — or entities — is important in order to create meaningful data models. Ultimately, any data model is a reflection of certain aspects of the real world, and the real word deals with entities.

Strongly typed attributes are necessary for the platform to be intelligent with the data it stores. It needs to know about relations between objects in order to ensure a high degree of data consistency. In the Reflective Platform, data consistency is achieved through the application of business rules written in a human-friendly language. Look for an in-depth explanation in a later article.

Schemas are data too and they need to evolve over time in order to reflect the real world — both now and later. Think about all the systems out there that still don’t have a place for storing people’s cell phone numbers, twitter handles and whatnot. Thankfully users often find creative ways to store the information anyway, but this then prevents the system from securing any form of data consistency — it doesn’t know the data is there, or what it means.

Changing the schema of an object. Wait, what? Are we saying that a car may become a person? — well, perhaps not. But consider an organizational system that distinguishes between for-profit companies and nonprofit organizations. It’s not inconceivable that a given company would eventually become nonprofit, or vice versa. And in this case, we’re effectively left with three options if we want to keep our data in line with the real world:

1. Throw away the existing data and record a new object in the system. This would lose any historic data and be almost guaranteed to cause problems with internal references as the new object would have a new id
2. Migrate the existing object somehow. This would again lose track of the fact that the entity used to be something else, not to mention that the attributes of different schemas usually don’t match up one-to-one
3. Simply accept that the schema of an object can change and record this as a fact in the data store

Never lose information. We’re choosing option 3. The bitemporal nature of the Reflective Platform saves the day, because we can record the new information without destroying what has already been recorded. We can ensure type safety and data consistency at any given point in time — if a schema changes, or an object is tagged with a different schema, this just means that the platform will impose a different set of requirements depending on whether the given point in time lies before or after the change.

Summary

Reflective Data Modeling is a paradigm for creating and working with data models in a way that adapts well to change and doesn’t lose previously recorded information. Working with objects stored in this manner makes perfect sense because it behaves just like in the real world; at any given point in time, we can present a consistent data model with a single version of the truth. But what happens to be the gospel truth one day, doesn’t have to be forever.

About the author

Rasmus Borgsmidt is a cofounder of Reflective, a recent Danish startup aiming to provide public and private organizations with an open platform that allows them to control and manage their own data.

Reflective is currently also working on Reflective Organization, an expert system that builds on the Reflective Platform to manage organizational processes and information.