Build End to End Azure Data Engineering Project
Sagar Prajapati
15 modules
English
Certificate of completion
Lifetime access
"Build Your Azure Data Engineering Skills with Hands-On Experience in an End-to-End Project - Develop Real-World Solutions and Earn In-Demand Skills Today!"
Key Highlights
Learn to design and deploy a complete data engineering project on Azure
Explore various data formats, techniques, and tools used for data processing on Azure
Understand the principles of data integration, transformation, and visualization using Azure services
Master the use of Azure Data Factory, Azure Databricks, Azure Stream Analytics, and Power BI
Gain hands-on experience by building an end-to-end data engineering project on Azure
What you will learn
Creating an Azure Data Factory
Learn how to create and configure an Azure Data Factory to manage your data pipeline.
Data Ingestion Techniques
Understand various methods to ingest data into Azure Data Factory like Azure Blob Storage, Azure SQL Database and API.
Designing and Building Data Transformation with Databricks
Learn how to use Azure Databricks to transform the data received from Data Ingestion Techniques into meaningful insights.
Building Data Processing Workflows
Understand how to use ADF pipeline to create data processing workflows to automate the data processing and analysing workloads.
Automating the Data Workflow
Learn how to schedule the ADF workloads and enable Alerting, Monitoring and Audit Trail to generate reports.
Modules
Introduction and Architecture
1 attachment(s) • 15.09 mins
Introduction and Architecture
Let's understand Azure Resources and How to create them
10 attachment(s) • 11.94 mins
What is Azure Data Fcatory and How to setup it?
What is Azure Blob Storage and How to setup it?
What is Azure DataLake GEN2 and How to setup it?
What is AzureKeyvault and How to setup it?
What is SPN and How to register SPN?
What is SQL Databases and How to setup it?
What is Azure DataBricks and How to setup it?
What is Logic App and How to setup it?
What is Azure Automation Runbook and How to setup it?
What is API? and How to Register API_Key ?
Data Preparation and Uploading into the respective sources
8 attachment(s) • -1 hrs 10.63 mins
Architecture
PLANE
UNIQUE_CARRIERS
AIRPORT
Cancellation
flights
Email_Send
Import files,data into Azure Blob, DataLake Gen2, SQL Database
Ingest Data from multiple source
12 attachment(s) • 1 hrs 33.63 mins
Create LinkedServices and Grant Access to Resource
Table_Source_Datalake
Table_Source_SQL
Create Datasets and Ingest CSV files
Unzip files and Ingest them into Data Lake Gen2 with single pipeline
Copy SQL tables from Server to DataLake Gen2 using parameterization
Fetch Data from API using Rest API and Web Activity to fetch credentials from KeyVault
Setup Repository and commit ADF changes to Github Repository
Updates on ADF Pipeline
Create Mount Point of Azure Blob Storage using Databricks
Create Mount Point of Azure DataLake GEN2 and Dumping PDF files using Databricks
Copy PDF files from Azure Blob Storage using parameterization to Azure Datalake using Databricks
Cleansing Process and Data Quality checks
6 attachment(s) • 1 hrs 44.67 mins
Clean the Data using AUTOLOADER in Databricks - Part 1
Clean the Data using AUTOLOADER in Databricks - Part 2
Clean the Data using AUTOLOADER in Databricks - Part 3
Clean the Data using Batch processing in Databricks
Data Quality checks on Cleansed Layer using Delta
Commit Databricks Notebook using Repo feature in GitHub
Create Dimensions and Fact Tables
1 attachment(s) • 55 mins
Create Dimesions and Fact tables
Publish the data into SQL Database using ADF
2 attachment(s) • 52 mins
Publish final tables into SQL Databases using Azure Databricks Delta Lake
publish_table
Setup Complete ADF Pipeline (Source to Publish)
1 attachment(s) • 44 mins
Create Master Pipeline to run the complete project (Source Layer - > Publish Layer)
Send Data Files to End uses using Logic App
3 attachment(s) • 51.28 mins
Email_Send Sample File
Multiple Ways to Send Data files to End Users
Optimize way to send Data Files to End Users
Run Automation Script using Automation Account
1 attachment(s) • 1 hrs 5 mins
Switch to SQL DW and Start/Stop the SQL DW using Automation Account
Introduce Data Quality Check Layer between Mart and Publish Layer in Databricks
1 attachment(s) • 28 mins
Build Data Checks Utility to check Mart logics in Databricks
CI-CD Deployments using Azure Devops and Github Action
4 attachment(s) • 44.43 mins
Azure Data Factory Deployment
Azure Blob Storage and GEN2 files Deployment
Azure Databricks Deployment using Github Action
Add Trigger and Global Parameters and Fix CI-CD for Azure Data Factory
Handle corner cases in ADF
1 attachment(s) • 17 mins
How to use Filter Activity to handle Failure Runs in ADF
Validate your knowlege
Repository and Other Files
3 attachment(s)
Databricks Code Repository
ADF Code Repository
SQL DW Automation Runbook Script
Certification
When you complete this course you receive a ‘Certificate of Completion’ signed and addressed personally by me.
About the creator
Sagar Prajapati
I'm Sagar Prajapati having 4 years of Industry Experience in the field of Data Engineering. I taught more than 14,000 students globally and I teach SQL, Python, PySpark, Azure, Databricks, PowerBI, etc. Subscribe to my YouTube Channel "GeekCoders".
Rate this course
Free
Order ID:
This course is in your library
What are you waiting for? It’s time to start learning!
Wait up!
We see you’re already enrolled in this course till Lifetime. Do you still wish to enroll again?
