Preview Build AI Application using Databricks and LLM

Architecture and Prerequiste

Architecture and Problem Statement

Understanding of Dataset

Databricks Workspace Setup in AWS

Create S3 Bucket

Prepare Source System - Upload Datasets in S3 Bucket

Download Datasets files

Project Code Base - Please download the zip

Ingest Data from S3 to Unity Catalog

Create Schema

Understand Groups and Permission

How to get access on S3 bucket and Create Scopes in AWS Databricks

Ingest Data from S3 to Bronze Layer

Create Repo and Link in Databricks

Ingest other data into Bronze Layer and Setup Workflow

Cleaning the bronze data

Build Lakehouse Quality Dashboard Manually

Create Monitor for all the Bronze Tables - Automation

Create Clean Notebook for Patient Data

Create Clean Notebook for Other Tables - Optimise

Create Clean Notebook for Other Tables

Update Workflow for Silver Notebooks

Ingest data into Gold Layer

Create Dimension Master Table Gold Layer and Add in Workflows

Vector Search and Embedding Models

Create Vector Search Endpoint

Create Embedding Model and Serving EndPoint

Create Embeddings and Create VS Index table

Update Workflow - Add VS Notebooks

Setup Model Serving Endpoint

Create Serving Endpoint with RAG

Query serving endpooint

Run Workflows using SPN

Build Streamlit Chatbot App

Create Streamlit ChatBot Application