DPS 2021 TRAINING CLASS – DATA ENGINEERING BEST PRACTICES USING AZURE DATA FACTORY
Abstract: In this workshop, we will cover data engineering best practices while using Azure Data Factory – Performance, Security, and Scalability being the key focus areas. We will build ETL pipelines as part of the workshop for hands-on learning.
1. Data pipeline overview
- What is data factory?
- Data engineering common scenario
- Getting started and environment setup
2. Data Integration: Connectivity and code-free transformation
- Bringing data to data lake on Azure: On-premises and SaaS/PaaS datastore connectivity with a Copy activity
- Transforming data using data flows
3. Best practices for operationalizing data pipelines
- Govern data using Azure Purview Integration
- Scalability and Performance considerations
- External integrations with compute engines (Databricks, SProc)
- Continuous Integration and Continuous Deployment (CICD)