Data Engineering - A product approach

Typically data engineering teams are focused on Extract Transform Load (ETL) processes; developing robust data pipelines that are consumed by applications or technical users like Data Scientists, Data Analysts, or Application Developers. Modern data engineering teams add a missing element to the process, Access. The view on access is that it should be accessible to a wide range of users, regardless of their level of technical ability, and in recent years the role of Analytics Engineer has arisen to fulfil this need. Here, however, we describe an alternative approach: development and deployment of self-serve data products. Owing to the wide range of technical skills amongst users, products are aligned to various personas that indicate the level of technical ability required to best interact with the product.

Enable informed decision making through self-serve data products

Audience

Informed decisions can be made by many different people within an organization, either directly or indirectly. A decision can be made directly from a source, or through a chain of interactions and users, each having their own perspective, purpose, needs, and abilities.

Personas

Data engineering teams can leverage philosophies from different areas within the tech sector, specifically human centric design and design thinking as it relates to building products for user personas. As said by Anthony Salerno

Personas depict the attitudes, behaviours, and motivations of customers. Unlike traditional marketing segments which focus on demographic attributes, personas focus on how customers think and why they do or think the way they do

Following that idea of personas, data engineering teams can structure their product vision around the following five personas.

Technical Ability

Knowing that users have different levels of technical understanding and ability, products should be designed to meet a range of technical ability; from low to high. Designed and developed products according to the target audience and their respected technical ability following three levels

Target Area

Data engineering aims to develop tangible solutions for customers, both internal and external with the intent to enable and empower users to leverage data to help inform decisions efficiently, effectively, repeatable, and at scale.

External customer: Organizations or individuals that exist outside of company

Internal customer: Lines of business, departments, or individuals that are within the company

Challenges

Within organizations there are many different challenges that can impede progress related to informed decision making; governance, organizational structure, scope, knowledge, and many others. For data engineering teams the following three are the most common challenges that are presented.

Products and Services

Presentations, spreadsheets, and analytic notebooks are very powerful and can be the life blood in many organizations, however relying on them make it very difficult to scale, nor does it typically result in something a customer is willing to purchase. A product or service is something that is consumed by users to complete a single or multiple tasks. Products and services in the field of data analytics can vary greatly, but data engineering teams can provide significant value to their organization by creating robust solutions within meaningful areas of data analytics.

Data Engineering Iceberg

Most users access a final product and have little understanding to what is required to produce and support the products. Whether a product is complex or simple there are many different elements that are required to develop something that is maintainable, reliable, adopted, and endorsed by a user.

Typically the elements are not glamourous, rather mundane, and highly complex; a combination that is understandably forgotten or ignored by the majority of users. Hardware and software are known pieces required to create digital products, but simply setting them up will not result in a successful product; an ecosystem is needed, which requires considerable configuration and monitoring. Of all the parts required processes and services are surely the least recognized and appreciated. If an ecosystem is where a product lives, process and services are the bridge that provides the access.

There are a variety of processes or services that are required to deploy digital products, but close to all will require automation, testing, setting up code repositories, code review, and continuous integration/continuous deployment (CI/CD); the true unsung heroes of a data engineering team.

Wrap Up

Data visualizations, dashboards, machine learning models, AI, and data science will always get the focal attention of an organization. Education and advocating are important methods for communicating why data engineering, however a product approach may be a more efficient and effective means to bring awareness of the value data engineering brings to an organization.

As any good band knows, they are only as successful as their roadies and management team; so maybe as the field of data analytics matures the understanding of what a data engineering team brings to the success of an organization can be better realized, and maybe just maybe be considered glamourous.