|
| 1 | +<p align="center"> |
| 2 | + <img src="docs/readme/sqlmesh.png" alt="SQLMesh logo" width="50%" height="50%"> |
| 3 | +</p> |
| 4 | +<p align="center">SQLMesh 是 <a href="https://www.linuxfoundation.org/">Linux 基金会</a>的一个项目</p> |
| 5 | + |
| 6 | +SQLMesh 是一款下一代数据转换框架,旨在快速、高效且无错误地传输数据。数据团队可以运行和部署用SQL或Python编写的数据转换,且可在任何规模下实现可视化和控制。 |
| 7 | + |
| 8 | +它不仅仅是[DBT的替代方案](https://tobikodata.com/reduce_costs_with_cron_and_partitions.html)。 |
| 9 | + |
| 10 | +<p align="center"> |
| 11 | + <img src="docs/readme/architecture_diagram.png" alt="Architecture Diagram" width="100%" height="100%"> |
| 12 | +</p> |
| 13 | + |
| 14 | +## 核心功能 |
| 15 | + |
| 16 | +<img src="https://github.com/SQLMesh/sqlmesh-public-assets/blob/main/vscode.gif?raw=true" alt="SQLMesh Plan Mode"> |
| 17 | + |
| 18 | +> 在CLI和[SQLMesh VSCode扩展](https://sqlmesh.readthedocs.io/en/latest/guides/vscode/?h=vs+cod)中,都能即时获得SQL的影响和更改的上下文 |
| 19 | +
|
| 20 | + <details> |
| 21 | + <summary><b>虚拟数据环境</b></summary> |
| 22 | + |
| 23 | + * 查看[虚拟数据环境](https://whimsical.com/virtual-data-environments-MCT8ngSxFHict4wiL48ymz)工作的完整示意图 |
| 24 | + * [观看此视频以了解更多](https://www.youtube.com/watch?v=weJH3eM0rzc) |
| 25 | + |
| 26 | + </details> |
| 27 | + |
| 28 | + * 创建无数据仓库成本的独立开发环境 |
| 29 | + * 规划/应用像[Terraform](https://www.terraform.io/)这样的工作流程,以了解变更可能带来的影响 |
| 30 | + * 易于使用的[CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/),适合真正的蓝绿部署 |
| 31 | + |
| 32 | +<details> |
| 33 | +<summary><b>效率与测试</b></summary> |
| 34 | + |
| 35 | +执行此命令将会生成一个单元测试文件 `tests/` : `test_stg_payments.yaml` |
| 36 | + |
| 37 | +运行实时查询以生成模型的预期输出 |
| 38 | + |
| 39 | +```bash |
| 40 | +sqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments "select * from tcloud_demo.seed_raw_payments limit 5" |
| 41 | + |
| 42 | +# 运行单元测试 |
| 43 | +sqlmesh test |
| 44 | +``` |
| 45 | + |
| 46 | +```sql |
| 47 | +MODEL ( |
| 48 | + name tcloud_demo.stg_payments, |
| 49 | + cron '@daily', |
| 50 | + grain payment_id, |
| 51 | + audits (UNIQUE_VALUES(columns = ( |
| 52 | + payment_id |
| 53 | + )), NOT_NULL(columns = ( |
| 54 | + payment_id |
| 55 | + ))) |
| 56 | +); |
| 57 | + |
| 58 | +SELECT |
| 59 | + id AS payment_id, |
| 60 | + order_id, |
| 61 | + payment_method, |
| 62 | + amount / 100 AS amount, /* `amount 字段当前以美分为单位存储,所有我们将其转换为美元 */ |
| 63 | + 'new_column' AS new_column, /* 非破坏性变更的实例 */ |
| 64 | +FROM tcloud_demo.seed_raw_payments |
| 65 | +``` |
| 66 | + |
| 67 | +```yaml |
| 68 | +test_stg_payments: |
| 69 | +model: tcloud_demo.stg_payments |
| 70 | +inputs: |
| 71 | + tcloud_demo.seed_raw_payments: |
| 72 | + - id: 66 |
| 73 | + order_id: 58 |
| 74 | + payment_method: coupon |
| 75 | + amount: 1800 |
| 76 | + - id: 27 |
| 77 | + order_id: 24 |
| 78 | + payment_method: coupon |
| 79 | + amount: 2600 |
| 80 | + - id: 30 |
| 81 | + order_id: 25 |
| 82 | + payment_method: coupon |
| 83 | + amount: 1600 |
| 84 | + - id: 109 |
| 85 | + order_id: 95 |
| 86 | + payment_method: coupon |
| 87 | + amount: 2400 |
| 88 | + - id: 3 |
| 89 | + order_id: 3 |
| 90 | + payment_method: coupon |
| 91 | + amount: 100 |
| 92 | +outputs: |
| 93 | + query: |
| 94 | + - payment_id: 66 |
| 95 | + order_id: 58 |
| 96 | + payment_method: coupon |
| 97 | + amount: 18.0 |
| 98 | + new_column: new_column |
| 99 | + - payment_id: 27 |
| 100 | + order_id: 24 |
| 101 | + payment_method: coupon |
| 102 | + amount: 26.0 |
| 103 | + new_column: new_column |
| 104 | + - payment_id: 30 |
| 105 | + order_id: 25 |
| 106 | + payment_method: coupon |
| 107 | + amount: 16.0 |
| 108 | + new_column: new_column |
| 109 | + - payment_id: 109 |
| 110 | + order_id: 95 |
| 111 | + payment_method: coupon |
| 112 | + amount: 24.0 |
| 113 | + new_column: new_column |
| 114 | + - payment_id: 3 |
| 115 | + order_id: 3 |
| 116 | + payment_method: coupon |
| 117 | + amount: 1.0 |
| 118 | + new_column: new_column |
| 119 | +``` |
| 120 | +</details> |
| 121 | +
|
| 122 | +* 永远不要[重复](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html)建立一个表 |
| 123 | +* 跟踪被修改的数据,并只运行[增量模型](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html)所需的转换 |
| 124 | +* 免费运行[单元测试](https://tobikodata.com/we-need-even-greater-expectations.html)并配置自动审计 |
| 125 | +* 根据受变更影响的表/视图,运行生产和开发之间的[表差异](https://sqlmesh.readthedocs.io/en/stable/examples/sqlmesh_cli_crash_course/?h=crash#run-data-diff-against-prod) |
| 126 | +
|
| 127 | +<details> |
| 128 | +<summary><b>提升你的SQL水平</b></summary> |
| 129 | +用任何语言写 SQL,SQLMesh 都会在发送到仓库前实时转译成目标 SQL 语言。 |
| 130 | +<img src="https://github.com/SQLMesh/sqlmesh/blob/main/docs/readme/transpile_example.png?raw=true" alt="Transpile Example"> |
| 131 | +</details> |
| 132 | +
|
| 133 | +* 在仓库运行转换错误 *之前* ,先调试 [10+种不同的SQL语言](https://sqlmesh.readthedocs.io/en/stable/integrations/overview/#execution-engines) |
| 134 | +* 仅使用 [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) 定义 (无需冗余和难懂的 `Jinja` + `YAML`) |
| 135 | +* 在数据仓库中执行变更操作前,通过列级血缘关系查看变更的影响范围 |
| 136 | + |
| 137 | +对于更多信息, 请查看 [技术文档](https://sqlmesh.readthedocs.io/en/stable/). |
| 138 | + |
| 139 | +## 入门指南 |
| 140 | +通过 [pypi](https://pypi.org/project/sqlmesh/) 安装 SQLMesh, |
| 141 | +执行: |
| 142 | + |
| 143 | +```bash |
| 144 | +mkdir sqlmesh-example |
| 145 | +cd sqlmesh-example |
| 146 | +python -m venv .venv |
| 147 | +source .venv/bin/activate |
| 148 | +pip install 'sqlmesh[lsp]' # 安装带VSCode扩展功能的sqlmesh包 |
| 149 | +source .venv/bin/activate # 重新激活虚拟环境,确保使用的是正确的安装版本 |
| 150 | +sqlmesh init # 按照提示开始操作(选择DuckDB) |
| 151 | +``` |
| 152 | + |
| 153 | +</details> |
| 154 | + |
| 155 | +> 注意:根据你的 Python 安装情况,可能需要运行 `python3` 或 `pip3` 来代替 `python` or `pip` |
| 156 | + |
| 157 | +<details> |
| 158 | +<summary><b>Windows 安装</b></summary> |
| 159 | + |
| 160 | +```bash |
| 161 | +mkdir sqlmesh-example |
| 162 | +cd sqlmesh-example |
| 163 | +python -m venv .venv |
| 164 | +.\.venv\Scripts\Activate.ps1 |
| 165 | +pip install 'sqlmesh[lsp]' # 安装带VSCode扩展功能的sqlmesh包 |
| 166 | +.\.venv\Scripts\Activate.ps1 # 重新激活虚拟环境,确保使用的是正确的安装版本 |
| 167 | +sqlmesh init # 按照提示开始操作(选择DuckDB) |
| 168 | +``` |
| 169 | +</details> |
| 170 | + |
| 171 | + |
| 172 | +跟着 [快速入门指南](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/) 学习如何使用SQLMesh,你已经有了一个好的开始! |
| 173 | + |
| 174 | +跟着 [速成课程](https://sqlmesh.readthedocs.io/en/stable/examples/sqlmesh_cli_crash_course/) 学习核心招式,并使用易于查阅的速查表。 |
| 175 | + |
| 176 | +跟着这个 [例子](https://sqlmesh.readthedocs.io/en/stable/examples/incremental_time_full_walkthrough/) 学习如何完整操作SQLMesh。 |
| 177 | + |
| 178 | +## 加入我们的社区 |
| 179 | +通过以下方式与我们联系: |
| 180 | + |
| 181 | +* 加入 [Tobiko Slack社区](https://tobikodata.com/slack) ,提问或打个招呼吧! |
| 182 | +* 请在我们的 [GitHub](https://github.com/SQLMesh/sqlmesh/issues/new) 提交问题 |
| 183 | +* 请通过 [hello@tobikodata.com](mailto:hello@tobikodata.com) 发送电子邮件,提出您的问题或反馈 |
| 184 | +* 阅读我们的 [blog](https://tobikodata.com/blog) |
| 185 | + |
| 186 | +## 贡献 |
| 187 | +我们欢迎大家的贡献!请参阅 [CONTRIBUTING.md](CONTRIBUTING.md) 有关如何参与的指导方针,包括我们的DCO签字要求。 |
| 188 | + |
| 189 | +请查阅我们的 [行为准则](CODE_OF_CONDUCT.md) 和 [管理文件](GOVERNANCE.md) . |
| 190 | + |
| 191 | +[阅读更多](https://sqlmesh.readthedocs.io/en/stable/development/) 关于如何搭建你的开发环境的信息。 |
| 192 | + |
| 193 | +## 许可说明 |
| 194 | +本项目采用 [Apache License 2.0](LICENSE) 授权。文档采用 [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) 授权. |
0 commit comments