|
1 | | -# PROJECT NAME |
2 | | -Standard Project Template for Databricks Labs Projects |
| 1 | +# Rules Engine |
| 2 | +Simplified Validation for Production Workloads |
3 | 3 |
|
4 | 4 | ## Project Description |
5 | | -Short description of project's purpose |
| 5 | +As pipelines move from bronze to gold, it's very common that some level of governance be performed in |
| 6 | +Silver or at various places in the pipeline. The need for business rule validation is very common. |
| 7 | +Databricks recognizes this and, as such, is building Delta Pipelines with Expectations. |
| 8 | +Upon release of Delta Pipelines, the need for this package will be re-evaluated and the code base will |
| 9 | +be adjusted appropriately. This is serves an immediate need and Delta Expectations is expected to be a more, |
| 10 | +full-fledged and robust example of this functionality. |
6 | 11 |
|
7 | | -## Project Support |
8 | | -Please note that all projects in the /databrickslabs github account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects. |
| 12 | +Introducing Databricks Labs - dataframe-rules-engine, a simple solution for validating data in dataframes before you |
| 13 | +move the data to production and/or in-line (coming soon). |
9 | 14 |
|
10 | | -Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support. |
| 15 | + |
| 16 | +## Using The Rules Engine In Your Project |
| 17 | +* Pull the latest release from the releases |
| 18 | +* Add it as a dependency (will be in Maven eventually) |
| 19 | +* Reference it in your imports |
11 | 20 |
|
| 21 | +## Getting Started |
| 22 | +A list of usage examples is available in the `demo` folder of this repo in [html](demo/Rules_Engine_Examples.html) |
| 23 | +and as a [Databricks Notebook DBC](demo/Rules_Engine_Examples.dbc). |
12 | 24 |
|
13 | | -## Building the Project |
14 | | -Instructions for how to build the project |
| 25 | +The process simple: |
| 26 | +* Define Rules |
| 27 | +* Build a RuleSet from your Dataframe using your Rules you built |
| 28 | +```scala |
| 29 | +import com.databricks.labs.validation.utils.Structures._ |
| 30 | +import com.databricks.labs.validation._ |
| 31 | +``` |
| 32 | + |
| 33 | +As of version 0.1 There are three primary rule types |
| 34 | +* Boundary Rules |
| 35 | +* Categorical Rules (Strings and Numerical) |
| 36 | +* Date Rules (in progress) |
| 37 | + |
| 38 | +Rules can be composed of: |
| 39 | +* simple column references `col("my_column_name")` |
| 40 | +* complex columns `col("Revenue") - col("Cost")` |
| 41 | +* aggregate columns `min("ColumnName")` |
| 42 | + |
| 43 | +Rules can be applied to simple DataFrames or grouped Dataframes. To use a grouped dataframe simply pass |
| 44 | +your dataframe into the RuleSet and pass one or more columns in as `by` columns. This will apply the rule |
| 45 | +at the group level which can be helpful at times. |
| 46 | + |
| 47 | +### Simple Rule |
| 48 | +`val validateRetailPrice = Rule("Retail_Price_Validation", col("retail_price"), Bounds(0.0, 6.99))` |
| 49 | + |
| 50 | +### List of Rules |
| 51 | +NOTE: While validations can be performed on aggregate cols (whether the DF is grouped or not) aggregate columns |
| 52 | +only return a single value - as such the failed count will be set to 1 for failures so for aggregate columns |
| 53 | +the `Invalid_Count` is rendered somewhat useless. Better granularity can be seen in the report when not using |
| 54 | +aggregates. |
| 55 | +```scala |
| 56 | +val specializedRules = Array( |
| 57 | + // Example of aggregate column |
| 58 | + Rule("Reasonable_sku_counts", count(col("sku")), Bounds(lower = 20.0, upper = 200.0)), |
| 59 | + // Example of calculated column from optimized UDF |
| 60 | + Rule("Max_allowed_discount", |
| 61 | + max(getDiscountPercentage(col("retail_price"), col("scan_price"))), |
| 62 | + Bounds(upper = 90.0)), |
| 63 | + // Example distinct values rule |
| 64 | + Rule("Unique_Skus", countDistinct("sku"), Bounds(upper = 1.0)) |
| 65 | +) |
| 66 | +``` |
| 67 | + |
| 68 | +### MinMax Rules |
| 69 | +It's very common to build rules to validate min and max allowable values so there's a helper function |
| 70 | +to speed up this process. It really only makes sense to use minmax when specifying both an upper and a lower bound |
| 71 | +in the Bounds object. Using this method in the example below will only require three lines of code instead of the 6 |
| 72 | +if each rule was built manually |
| 73 | +```scala |
| 74 | +val minMaxPriceDefs = Array( |
| 75 | + MinMaxRuleDef("MinMax_Sku_Price", col("retail_price"), Bounds(0.0, 29.99)), |
| 76 | + MinMaxRuleDef("MinMax_Scan_Price", col("scan_price"), Bounds(0.0, 29.99)), |
| 77 | + MinMaxRuleDef("MinMax_Cost", col("cost"), Bounds(0.0, 12.0)) |
| 78 | +) |
| 79 | + |
| 80 | +// Generate the array of Rules from the minmax generator |
| 81 | +val minMaxPriceRules = RuleSet.generateMinMaxRules(minMaxPriceDefs: _*) |
| 82 | +``` |
| 83 | +OR -- simply add the list of minmax rules or simple individual rule definitions |
| 84 | +to an existing RuleSet (if not using builder pattern) |
| 85 | +```scala |
| 86 | +val someRuleSet = RuleSet(df) |
| 87 | +someRuleSet.addMinMaxRules(minMaxPriceDefs: _*) |
| 88 | +someRuleSet.addMinMaxRules("Retail_Price_Validation", col("retail_price"), Bounds(0.0, 6.99)) |
| 89 | +``` |
| 90 | + |
| 91 | +### Categorical Rules |
| 92 | +There are two types of categorical rules which are used to validate against a pre-defined list of valid |
| 93 | +values. Currently (as of 0.1) accepted categorical types are String, Double, Int, Long |
| 94 | +```scala |
| 95 | +val catNumerics = Array( |
| 96 | +Rule("Valid_Stores", col("store_id"), Lookups.validStoreIDs), |
| 97 | +Rule("Valid_Skus", col("sku"), Lookups.validSkus) |
| 98 | +) |
| 99 | + |
| 100 | +val catStrings = Array( |
| 101 | +Rule("Valid_Regions", col("region"), Lookups.validRegions) |
| 102 | +) |
| 103 | +``` |
15 | 104 |
|
16 | | -## Deploying / Installing the Project |
17 | | -Instructions for how to deploy the project, or install it |
| 105 | +### Validation |
| 106 | +Now that you have some rules built up... it's time to build the ruleset and validate it. As mentioned above, |
| 107 | +the dataframe can be simple or groupBy column[s] can be passed in (as string) to perform validation at the |
| 108 | +grouped level. |
| 109 | +```scala |
| 110 | +val (rulesReport, passed) = RuleSet(df) |
| 111 | +.add(specializedRules) |
| 112 | +.add(minMaxPriceRules) |
| 113 | +.add(catNumerics) |
| 114 | +.add(catStrings) |
| 115 | +.validate() |
18 | 116 |
|
19 | | -## Releasing the Project |
20 | | -Instructions for how to release a version of the project |
| 117 | +val (rulesReport, passed) = RuleSet(df, Array("store_id")) |
| 118 | +.add(specializedRules) |
| 119 | +.add(minMaxPriceRules) |
| 120 | +.add(catNumerics) |
| 121 | +.add(catStrings) |
| 122 | +.validate() |
| 123 | +``` |
| 124 | +The validation returns two items, a boolean (true/false) as to whether all rules passed or not. If a single rule |
| 125 | +fails the `passed` value above will return false. The `rulesReport` is a summary of which rules failed and, |
| 126 | +if the input column was not an aggregate column, the number of failed records. An image of the report is below. |
| 127 | + |
21 | 128 |
|
22 | | -## Using the Project |
23 | | -Simple examples on how to use the project |
| 129 | +## Next Steps |
| 130 | +Clearly, this is just a start. This is a small package and, as such, a GREAT place to start if you've never |
| 131 | +contributed to a project before. Please feel free to fork the repo and/or submit PRs. I'd love to see what |
| 132 | +you come up with. If you're not much of a developer or don't have the time you can still contribute! Please |
| 133 | +post your ideas in the issues and label them appropriately (i.e. bug/enhancement) and someone will review it |
| 134 | +and add it as soon as possible. |
| 135 | + |
| 136 | +Some ideas of great adds are: |
| 137 | +* Add a Python wrapper |
| 138 | +* Refactor Rule and/or Validator to implement an Abstract class or trait |
| 139 | + * There's a clear opportunity to abstract away some of the redundancy between rule types. |
| 140 | +* Implement a fast runner |
| 141 | + * Optimize performance by failing fast for big data. Smart sampling could be implemented to review subsets |
| 142 | + of columns/records and look for failures to enable a faster failure. |
| 143 | +* Implement tests |
| 144 | + * Yeah, I know...I should have done this on day 0...but...time is always an issue. I plan to come back and add |
| 145 | + tests but if you'd like to add tests, that's a great way to learn code base (especially one this small) |
| 146 | +* Implement the date time rule (or somet other custom rule) |
| 147 | + * The date time rule has already been scaffolded, it just needs to be built out |
| 148 | + * What kind of complex rules does your business require that isn't possible here |
| 149 | +* Add a quarantine pattern |
| 150 | + * Enable a configuration to a Ruleset to identify records that didn't pass the validations and add |
| 151 | + them to a predefined quarantine zone. |
| 152 | +* Add logic to attempt to auto-handle certain types of failures based on common business patterns |
| 153 | +* When Delta Pipelines feature is release, simplify this package by wrapping the logic with pipelines. |
| 154 | + |
| 155 | + |
| 156 | +## Legal Information |
| 157 | +This software is provided as-is and is not officially supported by Databricks through customer technical support channels. |
| 158 | +Support, questions, and feature requests can be submitted through the Issues page of this repo. |
| 159 | +Please see the [legal agreement](LICENSE.txt) and understand that issues with the use of this code will |
| 160 | +not be answered or investigated by Databricks Support. |
| 161 | + |
| 162 | +## Core Contribution team |
| 163 | +* Lead Developer: [Daniel Tomes](https://www.linkedin.com/in/tomes/), Practice Leader, Databricks |
| 164 | +* Developer: <b> your name here </b> Contribute to the project |
| 165 | + |
| 166 | + |
| 167 | +## Project Support |
| 168 | +Please note that all projects in the /databrickslabs github account are provided for your exploration only, |
| 169 | +and are not formally supported by Databricks with Service Level Agreements (SLAs). |
| 170 | +They are provided AS-IS and we do not make any guarantees of any kind. |
| 171 | +Please do not submit a support ticket relating to any issues arising from the use of these projects. |
| 172 | + |
| 173 | +Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. |
| 174 | +They will be reviewed as time permits, but there are no formal SLAs for support. |
| 175 | + |
| 176 | + |
| 177 | +## Building the Project |
| 178 | +To build the project: <br> |
| 179 | +``` |
| 180 | +cd Downloads |
| 181 | +git pull repo |
| 182 | +sbt clean package |
| 183 | +``` |
0 commit comments