From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press

Unlock this course with a free trial

Join today to access over 25,500 courses taught by industry experts.

Using pipelines to ingest data

Using pipelines to ingest data - Microsoft Fabric Tutorial

From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press

Start my 1-month free trial Buy for my team

Using pipelines to ingest data

“

Using pipelines to ingest data. Now with pipelines, we're using it in a no to low code fashion to go and get data. So in terms of data movement strategies, we're loading data from data lakes, we're migrating data. Perhaps we're using pipelines in terms of a medallion ingestion. We can incrementally copy data. we've got a whole bunch of source connectors and destination connectors in pipelines to move data from our source into our destination. This is very useful for batch and bulk orientated data loading and if we use watermark based incremental copy and CDC functionality we can do incremental loading as well and we can use it for orchestration strategies because we have lots of different types of activities within our pipelines, we can orchestrate a data loading and transformation process. Now with our copy data activity specifically, this is responsible for moving data from the source to the destination. We've got something called the intelligent throughput optimization. And we can…

Contents