Azure Data Catalog

Azure Data Catalog is a fully managed cloud service. It lets users discover the data sources they need and understand the data sources they find. At the same time, Data Catalog helps organizations get more value from their existing investments.

With Data Catalog, any user (analyst, data scientist, or developer) can discover, understand, and consume data sources. Data Catalog includes a crowdsourcing model of metadata and annotations. It is a single, central place for all of an organization's users to contribute their knowledge and build a community and culture of data.

Type of Users

Generally, the data users have been categorized into two major categories.

1. Producers
2. Consumers

Both are facing challenges in discovering and understanding of the existing datasets within Enterprises.

Challenges Faced By Consumers

• Lack of knowledge about existance of data sources
• No Central Repository for all data sources
• Unknown data source location
• Lack of understanding about data sources
• Scattered documentation for existing data sources

Challenges Faced By Producers

• Missing descriptive meta-data about data sources
• Out-dated documentation about data sources
• Implementation of Data Governance policies on data sources
• Complexity of creating and maintaining documentation describing data sources

Azure Data Categlog is the Answer

Data Catalog is designed to address these problems and to help enterprises get the most value from their existing information assets. Data Catalog makes data sources easily discoverable and understandable by the users who manage the data.

Data Catalog provides a cloud-based service into which a data source can be registered. The data remains in its existing location, but a copy of its metadata is added to Data Catalog, along with a reference to the data-source location. The metadata is also indexed to make each data source easily discoverable via search and understandable to the users who discover it.

After a data source has been registered, its metadata can then be enriched. The metadata can be added either by the user who registered it or by other users in the enterprise. Any user can annotate a data source by providing descriptions, tags, or other metadata, such as documentation and processes for requesting data source access. This descriptive metadata supplements the structural metadata (such as column names and data types) that's registered from the data source.

Discovering and understanding data sources and their use is the primary purpose of registering the sources. Enterprise users might need data for business intelligence, application development, data science, or any other task where the right data is required. They can use the Data Catalog discovery experience to quickly find data that matches their needs, understand the data to evaluate its fitness for the purpose, and consume the data by opening the data source in their tool of choice.

At the same time, users can contribute to the catalog by tagging, documenting, and annotating data sources that have already been registered. They can also register new data sources, which can then be discovered, understood, and consumed by the community of catalog users.

Reference

This article has been created based on the following Azure Data Catalog documentation.