Rise of Data Apps and the data application engineers

Today even more than before we are all interacting with applications which are extremely data intensive and data driven thanks to the power of cloud infrastructure which can offer infinite (slightly exaggerated) compute and storage. Just think of all the apps you have on your phone tracking your sleep, workout, streaming etc. which predicts your behavior, next action and even your thoughts probably.

So, what is a data application? How is it different from a regular good old application? Does it require a different type of engineering team for development?

Data applications are software applications which are visually oriented, data intensive and process large amounts of data to generate actionable information allowing users to consume data interactively around a specific need or use case. They differ from static BI reports or data warehouses as they offer ad hoc interaction with data through an intuitive interface tailored to the specific use case. They are more event driven and real time than static reporting and analytics platform. Some of the trends in data applications include AIML powered automation, embedded analytics and prescriptive actions, rich user experience, event driven, and cloud powered serverless infrastructure.

Modern data applications are different from standard software applications and they are also different from standard data warehouse and reporting platforms. In fact, these applications blur the difference between software applications and analytical platforms. What does this mean for development teams? Should development teams have both software engineers and data engineers?

I believe there is a need for a new type of engineers - data application engineers who are capable of using software engineering techniques and concepts to data engineering and vice versa to build data applications. Repeatability and automation of interactive insights presented with a rich interface is key to data applications. For example, embedded analytics (powered with AIML) will require collaboration and communication between data scientists and engineers to improve automation to streamline repeatable machine learning end to end, including deploying to production environments. So, software engineering concepts of services oriented architecture (micro services based on IDEALS), event driven architecture, functional engineering (pure task and immutability) and automated continuous integration, testing and deployment (DevOps) are critical to the building of robust data applications.

What does this mean for data engineers and software engineers? They can no longer shy away from each other's skillsets but rather embrace each other because the data applications are here to stay and become the dominant applications in future, if not the only type of application. Data application engineers will not only be building batch or real time data pipelines processing billions of data points as part of the data layer but they will also be involved in building and securing APIs to expose the results of the data processing into the application layer, and ultimately packaging all the components for automated testing and deployment. Data application engineers need to think of the end user and work with front end designers and engineers while designing and developing the backend and data stack to deliver a rich user experience.