Preview Databricks End - End Project with Delta Lake and Gen AI

Project Architecture

Architecture and Problem Statement

Procurement_Data- Download

GeekCoders_P2P -Code Base

Setup Databricks

Setup of Databricks Account Free Edition

Ingestion of Data

Create Catalogs and Assign Permission

Link Github Repo in Databricks

Read Data from Google Drive - Part 1

Read Csv files and Parameterize Notebook

Incremental Load - Ingestion

Ingestion from Staging to Bronze

Ingest CSV files from Staging to Bronze - Part 1

Ingest CSV files from Staging to Bronze| Parameterize - Part 2

Ingest Pdf from Staging to Bronze

Ingest Pdf from Staging to Bronze using SCD 1

Silver Layer

Extract Invoice information from invoice bronze table

Extract and Clean Po,GR , Materials and Vendor Tables

Gold Layer

Vector Search Endpoint and Index

Use LLM to match PO vs Invoice

3 Way Match PO vs Invoice vs GR

Optimising the Notebook

End to End Pipeline - Staging to Gold

Dashboard for Analytics