arrow_back
Introduction and Architecture
Introduction and Architecture
Let's understand Azure Resources and How to create them
What is Azure Data Fcatory and How to setup it?
What is Azure Blob Storage and How to setup it?
What is Azure DataLake GEN2 and How to setup it?
What is AzureKeyvault and How to setup it?
What is SPN and How to register SPN?
What is SQL Databases and How to setup it?
What is Azure DataBricks and How to setup it?
What is Logic App and How to setup it?
What is Azure Automation Runbook and How to setup it?
What is API? and How to Register API_Key ?
Data Preparation and Uploading into the respective sources
Architecture
PLANE
UNIQUE_CARRIERS
AIRPORT
Cancellation
flights
Email_Send
Import files,data into Azure Blob, DataLake Gen2, SQL Database
Ingest Data from multiple source
Create LinkedServices and Grant Access to Resource
Table_Source_Datalake
Table_Source_SQL
Create Datasets and Ingest CSV files
Unzip files and Ingest them into Data Lake Gen2 with single pipeline
Copy SQL tables from Server to DataLake Gen2 using parameterization
Fetch Data from API using Rest API and Web Activity to fetch credentials from KeyVault
Setup Repository and commit ADF changes to Github Repository
Updates on ADF Pipeline
Create Mount Point of Azure Blob Storage using Databricks
Create Mount Point of Azure DataLake GEN2 and Dumping PDF files using Databricks
Copy PDF files from Azure Blob Storage using parameterization to Azure Datalake using Databricks
Cleansing Process and Data Quality checks
Clean the Data using AUTOLOADER in Databricks - Part 1
Clean the Data using AUTOLOADER in Databricks - Part 2
Clean the Data using AUTOLOADER in Databricks - Part 3
Clean the Data using Batch processing in Databricks
Data Quality checks on Cleansed Layer using Delta
Commit Databricks Notebook using Repo feature in GitHub
Create Dimensions and Fact Tables
Create Dimesions and Fact tables
Publish the data into SQL Database using ADF
Publish final tables into SQL Databases using Azure Databricks Delta Lake
publish_table
Setup Complete ADF Pipeline (Source to Publish)
Create Master Pipeline to run the complete project (Source Layer - > Publish Layer)
Send Data Files to End uses using Logic App
Email_Send Sample File
Multiple Ways to Send Data files to End Users
Optimize way to send Data Files to End Users
Run Automation Script using Automation Account
Switch to SQL DW and Start/Stop the SQL DW using Automation Account
Introduce Data Quality Check Layer between Mart and Publish Layer in Databricks
Build Data Checks Utility to check Mart logics in Databricks
CI-CD Deployments using Azure Devops and Github Action
Azure Data Factory Deployment
Azure Blob Storage and GEN2 files Deployment
Azure Databricks Deployment using Github Action
Add Trigger and Global Parameters and Fix CI-CD for Azure Data Factory
Handle corner cases in ADF
How to use Filter Activity to handle Failure Runs in ADF
Build Report and Dashboard using PowerBI
Import Data and Create Report
PowerBI Report File
Validate your knowlege
Repository and Other Files
Databricks Code Repository
SQL DW Automation Runbook Script
ADF Code Repository