README.md

Data Engineering with Notebooks

This repository contains the code for the Data Engineering with Notebooks Snowflake Quickstart.

➡️ For overview, prerequisites, and to learn more, complete this end-to-end tutorial Data Engineering with Notebooks on quickstarts.snowflake.com.

---

Here is an overview of what we'll build in this tutorial:

Lab 1

Step 1 : Set Up Snowflake

Initially setup the base repo by forking and creating the snowflake account's secret keys. Uploading the 00_start_here.ipynb file in the Snowflake account.

Running the "03 : Setting up Snowflake" step for the creation of account level,database objects and event tables.

Step 2 : Deploy Notebooks

With the prerequistes setup observe the Github Actions for the deployment of the notebooks - load_excel_files and load_daily_city_metrics with the provided suffix.

After the successful deployments observe the notesbooks in the configured snowflake account.

Step 3 : Accessing Weather data from FrostByte in snowflake market place

Fetching the FrostByte database from Marketplace

Step 4 : Loading data from two Excel files in S3 into the LOCATION and ORDER_DETAIL tables

Executing the data load from the deployed PROD_06_load_excel_files in the snowflake by executing all the provided commands.

Step 5 : Load data into the DAILY_CITY_METRICS table with support for incremental processing

Executing the provided commands for incremental data load from the deployed PROD_07_load_daily_city_metrics

Step 6 : Python Task DAG API & Dag Deployment

Back in the 00_start_here Notebook, the "Step 08 Orchestrate Pipelines" section needs to be executed for creation of dags and they can be observed in the tasks section of the schema.

Step 7 : Executing the Production DAG

The executed dags runs can be observed in the below mentioned runs.