/Build End to End Azure Data Engineering Project
Course | Build End to End Azure Data Engineering Project

Build End to End Azure Data Engineering Project

Sagar Prajapati

15 modules


Certificate of completion

Lifetime access

"Build Your Azure Data Engineering Skills with Hands-On Experience in an End-to-End Project - Develop Real-World Solutions and Earn In-Demand Skills Today!"

Key Highlights

Learn to design and deploy a complete data engineering project on Azure

Explore various data formats, techniques, and tools used for data processing on Azure

Understand the principles of data integration, transformation, and visualization using Azure services

Master the use of Azure Data Factory, Azure Databricks, Azure Stream Analytics, and Power BI

Gain hands-on experience by building an end-to-end data engineering project on Azure

What you will learn

Creating an Azure Data Factory

Learn how to create and configure an Azure Data Factory to manage your data pipeline.

Data Ingestion Techniques

Understand various methods to ingest data into Azure Data Factory like Azure Blob Storage, Azure SQL Database and API.

Designing and Building Data Transformation with Databricks

Learn how to use Azure Databricks to transform the data received from Data Ingestion Techniques into meaningful insights.

Building Data Processing Workflows

Understand how to use ADF pipeline to create data processing workflows to automate the data processing and analysing workloads.

Automating the Data Workflow

Learn how to schedule the ADF workloads and enable Alerting, Monitoring and Audit Trail to generate reports.


Introduction and Architecture

1 attachment(s) • 15.09 mins

Introduction and Architecture

Let's understand Azure Resources and How to create them

10 attachment(s) • 11.94 mins

What is Azure Data Fcatory and How to setup it?

What is Azure Blob Storage and How to setup it?

What is Azure DataLake GEN2 and How to setup it?

What is AzureKeyvault and How to setup it?

What is SPN and How to register SPN?

What is SQL Databases and How to setup it?

What is Azure DataBricks and How to setup it?

What is Logic App and How to setup it?

What is Azure Automation Runbook and How to setup it?

What is API? and How to Register API_Key ?

Data Preparation and Uploading into the respective sources

8 attachment(s) • -1 hrs 10.63 mins








Import files,data into Azure Blob, DataLake Gen2, SQL Database

Ingest Data from multiple source

12 attachment(s) • 1 hrs 33.63 mins

Create LinkedServices and Grant Access to Resource



Create Datasets and Ingest CSV files

Unzip files and Ingest them into Data Lake Gen2 with single pipeline

Copy SQL tables from Server to DataLake Gen2 using parameterization

Fetch Data from API using Rest API and Web Activity to fetch credentials from KeyVault

Setup Repository and commit ADF changes to Github Repository

Updates on ADF Pipeline

Create Mount Point of Azure Blob Storage using Databricks

Create Mount Point of Azure DataLake GEN2 and Dumping PDF files using Databricks

Copy PDF files from Azure Blob Storage using parameterization to Azure Datalake using Databricks

Cleansing Process and Data Quality checks

6 attachment(s) • 1 hrs 44.67 mins

Clean the Data using AUTOLOADER in Databricks - Part 1

Clean the Data using AUTOLOADER in Databricks - Part 2

Clean the Data using AUTOLOADER in Databricks - Part 3

Clean the Data using Batch processing in Databricks

Data Quality checks on Cleansed Layer using Delta

Commit Databricks Notebook using Repo feature in GitHub

Create Dimensions and Fact Tables

1 attachment(s) • 55 mins

Create Dimesions and Fact tables

Publish the data into SQL Database using ADF

2 attachment(s) • 52 mins

Publish final tables into SQL Databases using Azure Databricks Delta Lake


Setup Complete ADF Pipeline (Source to Publish)

1 attachment(s) • 44 mins

Create Master Pipeline to run the complete project (Source Layer - > Publish Layer)

Send Data Files to End uses using Logic App

3 attachment(s) • 51.28 mins

Email_Send Sample File

Multiple Ways to Send Data files to End Users

Optimize way to send Data Files to End Users

Run Automation Script using Automation Account

1 attachment(s) • 1 hrs 5 mins

Switch to SQL DW and Start/Stop the SQL DW using Automation Account

Introduce Data Quality Check Layer between Mart and Publish Layer in Databricks

1 attachment(s) • 28 mins

Build Data Checks Utility to check Mart logics in Databricks

CI-CD Deployments using Azure Devops and Github Action

4 attachment(s) • 44.43 mins

Azure Data Factory Deployment

Azure Blob Storage and GEN2 files Deployment

Azure Databricks Deployment using Github Action

Add Trigger and Global Parameters and Fix CI-CD for Azure Data Factory

Handle corner cases in ADF

1 attachment(s) • 17 mins

How to use Filter Activity to handle Failure Runs in ADF

Validate your knowlege

Repository and Other Files

3 attachment(s)

Databricks Code Repository

ADF Code Repository

SQL DW Automation Runbook Script


When you complete this course you receive a ‘Certificate of Completion’ signed and addressed personally by me.

Course Certificate

About the creator

Course Creator | Sagar Prajapati

Sagar Prajapati

I'm Sagar Prajapati having 4 years of Industry Experience in the field of Data Engineering. I taught more than 14,000 students globally and I teach SQL, Python, PySpark, Azure, Databricks, PowerBI, etc. Subscribe to my YouTube Channel "GeekCoders".

Rate this course



Order ID:

This course is in your library

What are you waiting for? It’s time to start learning!

Illustration | Payment success

Wait up!

We see you’re already enrolled in this course till Lifetime. Do you still wish to enroll again?

Illustration | Already enrolled in course