Avoid the common mistake of over-engineering

Following on from our previous blog on data cloud migration and management, it is timely to highlight a common mistake that many organisations make, that of over-engineering their data platform in the cloud.

In previous blogs I have discussed data lakes, data warehouses, data vaults and data fabrics, examining the pros, cons and differences between each.

Unfortunately, many companies fall prey to the variety of options available to them. As Willem Conradie, CTO and Technical Director at PBT Group notes, in an effort to keep up to date, they make the understandable mistake of implementing variations of each of the above.

This can be a double-edged sword. On one hand, the accessibility of technology in the cloud makes it easy to mix and match, deploying these different approaches. On the other hand, this actually introduces greater complexity into their data platform, making its architecture more unwieldy. Conradie explains that this approach also leaves users with a data architecture consisting of many layers, and several components in each layer, all of which must integrate, store, transform, serve and transport data at various frequencies.

The real risk in taking an ''everything but the kitchen sink'' approach, and over-engineering one's data architecture, is that the complications it brings are only fully experienced during implementation.

Furthermore, over-engineering brings several problems. Firstly, it actually reduces agility, which is a primary benefit of moving to the cloud. Conradie points outs that over-engineered solutions don’t only bring greater intricacy in the development and implementation stages; it also makes it more difficult to evolve the solution as needed too. Furthermore, it complicates maintenance and support of the solution during its lifecycle.

Another disadvantage that over-engineering presents is that it requires more skills - at a point now when the demand for IT skills in South Africa and globally outweighs the availability.

If that is not enough to sway organisations from over-engineering, then the next disadvantage may be. The more services that are used within a solution in the cloud, the more expensive it will be to run the solution.

Essentially over-engineering mitigates the benefits of using cloud in the first place, as it increases complexity, drives up expenses and reduces agility.

One reason why many fall into this trap is because, Conradie explains, vendor guidance tends to be quite general and does not necessarily take into consideration a client's specific environment and needs.

The solution to over-engineering is similar to that of dealing with large amounts of data: effective management.

Conradie offers a few suggestions to this end:

• Implement the correct architecture governance and controls to ensure solutions deployed in the cloud are "fit-for-purpose" from a business value perspective.
• Make use of the well architected frameworks cloud service providers offer to establish these controls but add your own flavour to them to ensure they make sense in your specific environment.
• Keep it simple.... It is very easy to make a solution complex, it takes a lot more forethought and experience to make it simple.

Instead of over-engineering, taking a measured approach to one's data platform is a wiser move. Conradie urges organisations to add any requirements - such as real-time capabilities - to their data platform when the decision to do so is driven by actual business value. He stresses that introducing more capabilities should be purely driven by business demand, not the availability of technologies.

He also encourages organisations to design their processes in such a way that adding near real-time capabilities, when needed, will not require any re-architecture, or extensive redevelopment of any existing processes.

Ultimately, resisting the urge to over-engineer one's data platform can be summed up by adopting a simple maxim: just because we can, doesn't mean we should.