Get your Data system in order

Recently, we had an interesting conversation within our team. To implement certain use case for a client, our SME sought out a series of data sets. The client facing team had difficulty in getting the required data for few weeks. At one point in time, customer simply raised his hands by saying this is what we have and now show us what you got.

Then the internal debate triggered on the topic - what and how much data is a basic necessity in order to achieve an outcome. There were heated to-and-fro arguments between dev and client teams. Both sides were right in their point of views and also wrong with their approach towards customer empathy. Challenges at customer end are numerous and it is important for both dev/client teams to strike the right chord.

In my research, I came across an interesting stats from HeavyReading on barriers to Data Science / ML in Telecom industry. The topper in the list is "Dirty data".

[actually the report cites this data set is from Kaggle]

"Dirty data" in my understanding not only represents partial data spread across the siloed environments but also covers noisy data and non-captured data as well. Especially industries like Telecom, where the underlying rich data sets covering customer usage to network operations supposedly exists, must be a dreamland of sorts for any data scientist worth his salt. Unfortunately rarely is the case.

Few reasons why dirty data exists in Telcos:

• Diverse data streams exists within the Telcos ecosystem - from network flows to KPIs to inventory data models to unstructured data. There's no standardized data structure available.
• Most of these data sets are not streamlined within data infrastructure like data warehouse or data lakes. They exist in independent data stores or databases.
• Since good chunk of data are generated from vendor applications, it often becomes unviable to extract and make sense of the proprietary data. Unless robust API support exists.
• Data ownership and access privilege issues often plague free-flow of rich data sets at enterprise level

And there are many more.

Before even we talk about applying AI/ML to understand new revenue opportunities or rationalizing the costs, it is important to get the data infrastructure in place. Well, there are various data stores available within Telco environment, otherwise the operations would simply get stalled! But what we're talking about is the data infrastructure that can move Telecom operations into different trajectory - Data driven, Automation First and AI / ML enabled.

Though there are big Telcos advancing very well on data infrastructure, AI / ML application and advanced analytics, there is a long tail of operators where this is a total non-starter. For this impressive number of Telcos, is there a cost effective data plumbing & infrastructure solution available, without frills of vendor lock-ins and expensive third party dependencies?

Well, the answer is yes.

With wide adoption of open source software, many vendors started to curate the open source stacks and create "open" platforms or products targeted for specific use cases or industries. This trend is picking up with many Telcos started to show keen interests. Initiatives like AT&T and Linux foundation's ONAP, is completely rooted in this philosophy.

There's a new class of products or more specifically "Telco AI platforms" emerging in the market. These platforms are actually designed & geared to solve the customer issues with data plumbing and self-serve analytics capabilities. More importantly, they also offer stacked use case solutions to intelligently automate well-known time consuming activities like network maintenance, alarm management, network monitoring, CEM, network optimization etc.

We also see traditional application providers going through technology upgrade cycles like 4G / 5G and raiding the wave successfully. Some products thrive through regulatory compliance cycle with periodic upgrades and new features. And many solution focus on point use cases trying to create unique value propositions. Not to talk about biggies where they swamp with list of products & solutions cornering left side of power law curves.

There are impressive list of offerings in the market with different business models. But no matter what business model it is, the availability of rich data has become a key enabler. With advent of technologies like 5G, SDN, IOT, MEC the data is just going to explode north. It is becoming imperative to all those long tail Telcos to have a solid data strategy not only to capture (data) value but also create value by using advanced AI / ML applications.

More often new technologies comes with adoption challenges. There's always a period of intense learning curve before the adoption break-through happens. Until then, it is a messy world of continuous unlearning and learning cycles for all market participants involved in the process.