Workshop - 08/03/24

Building Modern Data Platform for Analytics

I want to thank my friend Ed Pollack for asking me to speak at SQL Saturday Albany. This full day pre-conference workshop is a bargain for those professionals who want to learn about the MDP. You get breakfast, lunch, and my half of a decade experience on creating Modern Data Platforms in Azure for various clients who want to do analytics.

Here are the details that will be covered that day. Please sign-up for the workshop using Eventbrite. Hope to see you this summer!

Many companies are placing their corporate information into data lakes in the cloud. Since storage costs are cheap, the amount of data stored in the lake can easily exceed the amount of data seen in a typical relational database. Regardless of the types of files in the data lake, there is always a need to transform the raw data files into refined data files for analytics, machine learning, and/or AI.

The Delta Lakehouse design uses a medallion (bronze, silver, and gold) architecture for data quality. We can we abstract the read and write actions in Spark to create dynamic notebooks to process data files. Data pipelines can be used to bring remote data into the lake as well as orchestrate data processing. A metadata driven design allows for the inputs to the dynamic notebooks to be stored in a central place.

The most important part of a modern data platform is security. Microsoft Entra, formally known as Azure Active Directory, can be used to secure the files in storage. This security layer is used in both the Apache Spark and Serverless SQL pools. Designers use a variety of tools for reporting. The Serverless SQL Pool turns a data lake files into a read only database tables. While the demos in this course are Azure specific, the concepts can be used with any cloud service.

Lessons:

1. Infrastructure deployment (storage, key vault, Databricks, Synapse)
2. Create a service principle for services
3. Create medallion zones + assign rights
4. Introduction to Data Factory pipelines
5. How to create a hybrid design
6. Working with different sources (database, file shares, rest API's)
7. Hard coding vs meta data design
8. Full vs incremental load patterns
9. Configuring clusters + storage for security
10. Writing data engineering notebooks
11. Orchestrating pipelines with Data Factory
12. Creating a presentation layer with Synapse Serverless Pools
13. Connecting to Synapse with Power BI