End-to-End Medallion Architecture Implementation for GlobalMart
5 Scenarios
4 Hours 30 Minutes
Beginner
Azure-databricks-api-2
Popular
7 credits + 7 credits(for lab)
Industry
e-commerce
general
Skills
data-quality
data-wrangling
batch-etl
data-storage
approach
Tools
databricks
spark
azure
Learning Objectives
Understand the step-by-step workflow of the Bronze, Silver, and Gold layers in Databricks.
Perform data ingestion and create datasets in ADLS with proper configuration.
Build Delta tables in the Bronze and Silver layers, applying schema constraints and validations.
Generate aggregated insights in the Gold layer optimized for analytics.
Overview
This project guides you through building a complete Medallion Architecture (Bronze–Silver–Gold) pipeline in Databricks.
You’ll start by ingesting data into ADLS, organize it in the Bronze layer, perform cleaning and transformation in the Silver layer, and finally create aggregated outputs in the Gold layer.
By the end of this 3-hour hands-on session, you’ll gain practical experience in managing each layer efficiently using PySpark and SQL.
Prerequisites
- Basic knowledge of PySpark and SQL in Databricks
- Understanding of storage accounts and data upload process
- Familiarity with Delta tables and Medallion architecture concepts
- Understanding of Databricks workspace and clusters