Merge pull request #22 from techindicium/feature/testing-configs

Feature: testing README and configurations

Author: laysabit

.github/workflows/release.yml

@@ -49,7 +49,7 @@ jobs:
           elif [[ $BRANCH_NAME =~ ^(feature|minor|Feature|Minor)/ ]]; then
             VERSION_PARTS[1]=$((VERSION_PARTS[1] + 1))
             VERSION_PARTS[2]=0
-          elif [[ $BRANCH_NAME =~ ^(patch|fix|hotfix|Patch|Fix|Hotfix)/ ]]; then
+          elif [[ $BRANCH_NAME =~ ^(patch|fix|hotfix|bugfix|Patch|Fix|Hotfix|Bugfix)/ ]]; then
             VERSION_PARTS[2]=$((VERSION_PARTS[2] + 1))
           fi
           NEXT_VERSION="${VERSION_PARTS[0]}.${VERSION_PARTS[1]}.${VERSION_PARTS[2]}"

.gitignore

@@ -2,4 +2,6 @@ integration_tests/logs/
 integration_tests/target/
 integration_tests/.env
 integration_tests/.user.yml
-integration_tests/dbt_packages
+integration_tests/dbt_packages
+integration_tests/env
+integration_tests/package-lock.yml

README.md

@@ -4,13 +4,26 @@
 
 This package allows you to easily monitor and manage your BigQuery query costs, identifying costly queries in order to enhance optimizing and resource saving.
 
+# Table of Contents
+
+- [Before creating a branch](#Before-creating-a-branch)
+- [Quickstart](#:running:-Quickstart)
+  - [Requirements](#Requirements)
+  - [Installation](#Installation)
+  - [Configuring package](#Configuring-models-package)
+    - [Models](#Models)
+    - [Vars](#Vars)
+    - [Source](#Source)
+  - [Running the models](#Running-the-models)
+  - [Testing modifications in the project](#Testing-modifications-in-the-project)
+
 ## Before creating a branch
 
 Pay attention, it is very important to know if your modification to this repository is a release (breaking changes), a feature (functionalities) or a patch(to fix bugs). With that information, create your branch name like this:
 
 - `release/<branch-name>` or `major/<branch-name>` or `Release/<branch-name>` or `Major/<branch-name>`
 - `feature/<branch-name>` or `minor/<branch-name>` with capitalised letters work as well
-- `patch/<branch-name>` or `fix/<branch-name>` or `hotfix/<branch-name>` with capitalised letters work as well
+- `patch/<branch-name>` or `fix/<branch-name>` or `hotfix/<branch-name>` or `bugfix/<branch-name>` with capitalised letters work as well
 
 If branch is already made, just rename it _before passing the pull request_.
 
@@ -37,7 +50,7 @@ packages:
 
 
 
-## Configuring models package
+## Configuring package
 
 The package's models can be configured in your `dbt_project.yml` by specifying the package under `models` node. 
 
@@ -81,12 +94,10 @@ vars:
   dbt_prod_monitoring_dataset: 'public_monitoring' # if you have one, add the destination dataset of your monitoring transformed data
 ```
 
-## Configuring project sources
+### Source
 
 The project's sources can be configured in your `source.yml`, normally on your staging folder. Attention is needed while naming the source' schema and tables names to ensure you are matching the name BigQuery sets on the and schema and table with your source. If the names match, your package will work as expected.
 
-### Source configuration
-
 ```yaml
 version: 2
 
@@ -102,3 +113,29 @@ sources:
 ## Running the models
 
 After setting up the package in `dbt_project.yml` and `source.yml` as the previous steps, you can now run the package with the following command line: `dbt run -s bigquery_analytics`. After running it, the 5 models of the package will materialize in your target schema as they have been configured.
+
+## Testing modifications in the project
+
+Inside the integration_tests folder, you can use the setup.sh to auxiliate the creation of an env and installation of requirements.txt for you.
+
+The only commands that you need to do is:
+
+```chmod +x setup.sh```
+and
+```source setup.sh```
+
+Inside the integration_tests folder, you have the dbt_project already configured, you have profiles.yml already set to BigQuery. 
+To make a connection to BigQuery, so you can start testing, you should have credentials in indicium-sandbox project.
+
+To set your credentials in your environment, you should rename the example.env:
+```example.env``` --> ```.env```.
+
+You should put the name of your dataset 
+
+```dev_firstName_lastName```
+
+and then do the command
+
+```source .env```
+
+Now you can use your credentials in your environment. Dbt debug it!

dbt_project.yml

@@ -10,5 +10,29 @@ clean-targets: ["target", "dbt_modules", "dbt_packages"]
 macro-paths: ["macros"]
 log-path: "logs"
 
-vars:
-  region: "region-us" ## vars to fit your use case
+# When running dbt-bigquery-analytics in your project, you must add the following configurations:
+
+# models:
+#     bigquery_analytics:
+#         staging:
+#             materialized: ephemeral
+#         marts:
+#             materialized: table
+
+# vars:
+#   dbt_bigquery_analytics:
+#       bigquery_analytics_start_date: cast('2023-01-01' as date) # inside the double quotes, add the start date of the project
+#   region: "region-us"
+#   usd_dollars_real: 5.58 # here you can add the price of the dollar in reais
+#   price_per_tib: 6.25 # here you can add the price per terabyte processed
+#   price_per_tib_brl: 40.167405531
+#   dbt_sources: [
+#       'adw'
+#       , 'adf'
+#       , 'databricks_workflow'
+#   ] # you can configure the sources of your procject here
+#   ci_dbt_target: 'ci' # add the CI/CD target of your dbt project
+#   prod_dbt_target: 'prod' # add the development target of your dbt project
+#   dev_dbt_target: 'dev' # add the production target of your dbt project
+#   dbt_source_monitoring_dataset: 'raw_monitoring' # if you have one, add a source dataset of your monitoring data
+#   dbt_prod_monitoring_dataset: 'public_monitoring' # if you have one, add the destination dataset of your monitoring transformed data

integration_tests/example.env

@@ -0,0 +1,6 @@
+export DBT_DEFAULT_TARGET="bigquery";
+export BIGQUERY_DATASET=""; 
+export BIGQUERY_PROJECT="indicium-sandbox";
+export DBT_JOB_TIMEOUT="300";
+export DBT_THREADS="16"; 
+export DBT_JOB_RETRIES="1";

integration_tests/setup.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+#if something goes wrong, stop the script
+set -e
+
+# exports all variables in .env file. Any variable defined in the script will be available in the environment.
+set -a
+
+# Install virtualenv if not installed
+pip install virtualenv
+
+# Create a new virtual environment
+virtualenv env
+
+# It gives permission to activate the virtual environment
+chmod +x env/bin/activate
+
+# Activate the virtual environment
+source env/bin/activate  # On Windows, use `env\Scripts\activate`
+
+# # Load the environment variables
+source .env
+
+# Install requirements
+pip install -r ../requirements.txt
+
+dbt deps