eCommerce has a data problem
Summary: I have left IBM and stepped back from eCommerce to focus on the fundamental issues of information technology. My journey continues at JUXT (https://juxt.pro) where I will be working on forward-thinking database systems alongside a group of passionate Clojure experts.
During my 7 years of working with IBM WebSphere Commerce I have had the privilege of getting to know a huge range of client organisations—from some of the biggest names in retail, automotive and healthcare, to vibrant segment leaders who are pioneering with their strategies for customer engagement. The core trend I have observed throughout this time is that ecommerce businesses are increasingly demanding simplicity from their ecommerce technology, above all other qualities, and they do so in the hope that it will enable them to focus more of their vital energies on improving their operations, brands and products. Simplicity in this context can be appreciated at many levels (e.g. user experience, developer experience, licensing, support, and so on), however I am referring to simplicity here in its broadest possible sense.
We're all very limited compared to the complexity we can create—Rich Hickey
eCommerce businesses find themselves navigating through a relentless sea of technology investment decisions, and the key to staying ahead is in figuring out which technologies will propel you forward and which technologies are likely to drag you under in a tangle of complexity over the long term. For instance, multi-tenant cloud-based ecommerce platforms are incredibly easy to get up and running with and can propel a small business very fast. However, when a need arises to implement some highly unique functionality that the platform cannot yet support, you will invariably find that you have exchanged short-term convenience for a huge loss of flexibility and ultimately, autonomy. Easy is not the same as simple. If you choose technology because it is easy you will more often than not find that you have created a complex dependency on a system that you have little direct control over. Of course, sometimes such an exchange can still be worthwhile for a limited period where the immediate business value justifies accruing technical debt.
The key challenge for vendors of ecommerce platforms is in striking a sustainable balance in architecture: to remove as many limitations as possible (outwards simplicity) whilst retaining an uncompromised pace of development (inwards simplicity). Users of these platforms need to be able to confidently create extensions without unwittingly incurring any technical debt for either themselves or the vendors. One commonly pursued solution to achieving this balance is to implement all platform capabilities via heterogeneous microservices, but I believe that in most cases this will only serve to dramatically increase the internal complexity for the vendor. The biggest problem with microservices from a user perspective is that any sense of having a canonical store of your most important business data is typically non-existent. What was once a five-line SQL query is now a feat of asynchronous programming across an eventually consistent system, and a small extension to a monolithic database schema is now a whole sprint of coordinated activities. It seems clear to me that having a single canonical store of your data is the height of simplicity and businesses must not lose sight of that as they move forward into cloud environments.
Whilst the microservice technologies underpinning cloud-based ecommerce platforms are far from finalised, the platforms have evidently established plausible long-term commercial models (pricing, service level agreements, support etc.) and have validated the beneficial network-effects generated by providing a pre-integrated marketplace of 3rd-party extensions. Where these platforms cannot yet support a comprehensive range of functionality themselves it is critical that users are free to extend and create their own integration patterns. Therefore, the platforms are in a race to figure out how to support the greatest breadth of flexibility at a scale necessary to satisfy the enterprise market. For many segments of ecommerce businesses this relative immaturity of the extension options available is inevitably going to be a source of complexity and the simplest option for such businesses in 2019 is probably to stick with ecommerce technology that doesn't constrain future development, even if it isn't as "easy".
Integration is undoubtedly the biggest source of complexity in an ecommerce implementation because it is hard to predict how systems will need to interoperate without significant upfront validation. This is why reducing integration risks is one of the major benefits of using an ecommerce platform. However, no platform can accommodate every possible integration path across the various payment gateways, content management systems, ERPs, multiple instances of your application serving globally distributed markets, etc. The only real solution to integration is in thoughtfully using appropriate tools that keep the system as a whole as simple as possible. I would argue that focusing on rich database schemas has historically been a great strength of the enterprise ecommerce platforms to address advanced integration and extension requirements. However, the relative costs of maintaining custom schema extensions has increased dramatically over time and I believe this is predominantly due to the inadequate information modelling capabilities of the underlying database technologies used within these platforms. These databases and the application logic they support are facing an uphill battle to maintain internal development simplicity and provide effortless forwards-compatability within this highly complex domain.
In summary, ecommerce businesses should be using simple systems built on top of simple databases. This isn't specific to eCommerce though—it is true of nearly all software systems—but I know ecommerce businesses feel the pain of the status quo very acutely. This is what I will be working on at JUXT: creating simple systems. When the ecommerce platform market eventually settles on a de facto standard for ecommerce data (made possible by a simple database!), businesses everywhere will benefit from a vastly simplified landscape of integration and a Cambrian explosion of new cloud-based ecommerce possibilities to navigate.
To the IBMers, clients and partners reading—thank you and best wishes for the future!
If you are curious about the kind of technology we're working with at JUXT, I would recommend reading more about the following topics:
