Skip to content

Commit fffac81

Browse files
max-ostapenkodependabot[bot]google-labs-jules[bot]
authored
Add detection for outer DECLARE statements and update documentation (#40)
* Add detection for outer DECLARE statements and update documentation - Implemented `hasOuterDeclare` function to check for outer level DECLARE statements in SQL. - Updated `autoAssignActions` to skip operations with outer DECLARE. - Enhanced README and Copilot instructions to clarify limitations of automated assignment. - Added tests for various scenarios involving outer DECLARE statements. * feat: enhance reservation handling with native support detection * Bump eslint from 10.0.0 to 10.0.1 (#41) Bumps [eslint](https://github.com/eslint/eslint) from 10.0.0 to 10.0.1. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v10.0.0...v10.0.1) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.0.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump @dataform/core from 3.0.46 to 3.0.47 in /test-project (#42) Bumps [@dataform/core](https://github.com/dataform-co/dataform) from 3.0.46 to 3.0.47. - [Release notes](https://github.com/dataform-co/dataform/releases) - [Commits](dataform-co/dataform@3.0.46...3.0.47) --- updated-dependencies: - dependency-name: "@dataform/core" dependency-version: 3.0.47 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump @dataform/cli from 3.0.46 to 3.0.47 in /test-project (#43) Bumps [@dataform/cli](https://github.com/dataform-co/dataform) from 3.0.46 to 3.0.47. - [Release notes](https://github.com/dataform-co/dataform/releases) - [Commits](dataform-co/dataform@3.0.46...3.0.47) --- updated-dependencies: - dependency-name: "@dataform/cli" dependency-version: 3.0.47 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> * Bump eslint from 10.0.1 to 10.0.2 (#48) Bumps [eslint](https://github.com/eslint/eslint) from 10.0.1 to 10.0.2. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v10.0.1...v10.0.2) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.0.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump globals from 17.3.0 to 17.4.0 (#49) Bumps [globals](https://github.com/sindresorhus/globals) from 17.3.0 to 17.4.0. - [Release notes](https://github.com/sindresorhus/globals/releases) - [Commits](sindresorhus/globals@v17.3.0...v17.4.0) --- updated-dependencies: - dependency-name: globals dependency-version: 17.4.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump the npm_and_yarn group across 2 directories with 1 update (#44) Bumps the npm_and_yarn group with 1 update in the / directory: [minimatch](https://github.com/isaacs/minimatch). Bumps the npm_and_yarn group with 1 update in the /test-project directory: [minimatch](https://github.com/isaacs/minimatch). Updates `minimatch` from 3.1.2 to 3.1.5 - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v3.1.2...v3.1.5) Updates `minimatch` from 9.0.5 to 9.0.9 - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v3.1.2...v3.1.5) --- updated-dependencies: - dependency-name: minimatch dependency-version: 3.1.5 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: minimatch dependency-version: 9.0.9 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update Dataform version in CI matrix and changelog to 3.0.48 * Refactor isNativeReservationSupported to always return false * Bump eslint from 10.0.0 to 10.0.1 (#41) Bumps [eslint](https://github.com/eslint/eslint) from 10.0.0 to 10.0.1. - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v10.0.0...v10.0.1) --- updated-dependencies: - dependency-name: eslint dependency-version: 10.0.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * 🧹 Refactor duplicated logic in applyReservationToAction (#45) * refactor: deduplicate logic in applyReservationToAction - Extracted duplicated logic for prepending statements into `prependStatement` and `isArrayOrString` helpers. - Refactored `applyReservationToAction` to use these helpers for `contextablePreOps`, `contextableQueries`, `proto.preOps`, and `proto.queries`. - Simplified the `action.queries` monkeypatch using `prependStatement`. - Ensured non-mutating behavior for array prepending. - Restored `test-project/workflow_settings.yaml` that was accidentally renamed during matrix tests. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * refactor: update exports and enhance test coverage with new utility functions --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * ⚡ Optimize reservation lookup using Map (#47) * ⚡ Optimize reservation lookup using Map - Refactored `preprocessConfig` to build an optimized `Map` for reservation lookups. - Updated `findReservation` to perform O(1) lookups using the `Map`. - Preserved "first match wins" logic and existing validation. - Maintained backward compatibility by preserving original data structures in preprocessed config. - Improved lookup performance by ~24x in benchmarks. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * Refactor findReservation tests to use Map for action-reservation mapping --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * fix badge link * Update changelog for version 0.2.1 * 🧪 [testing improvement] Missing test for compiled objects (proto.preOps) (#46) * 🧪 test: add coverage for compiled objects and proto.preOps - Added comprehensive tests for compiled objects in `test/compiled_objects.test.js`. - Covered `proto.preOps` and `proto.queries` handling for both array and string types. - Verified fallback mechanisms and monkeypatching logic. - Increased line coverage of `index.js` to 100%. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * fix: standardize string quotes in compiled objects tests * refactor: simplify monkeypatched queries tests with parameterized cases --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> * Remove local integration testing instructions from CONTRIBUTING.md and update isNativeSupported logic in verify_compilation.js * fix createReservationSetter to directly use actionToReservation from preprocessConfig * refactor createReservationSetter * Update CONTRIBUTING.md and README.md for clarity and additional testing instructions * Remove duplicated isArrayOrString and prependStatement functions from index.js * lint --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
1 parent 33d3cee commit fffac81

9 files changed

Lines changed: 736 additions & 232 deletions

File tree

.github/copilot-instructions.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,9 @@ For `operations`, the SQL is often set via `.queries(["SQL"])`. This method can
4949
### 4. Assertions
5050
Assertions in Dataform are strict. They expect a single `SELECT` statement. Prepending a `SET` statement will cause a syntax error in BigQuery because assertions are often wrapped in subqueries or views by Dataform. We explicitly skip assertions in this package.
5151

52+
### 5. Outer DECLARE Detection
53+
Operations where `DECLARE` is the first statement at the outer level are automatically skipped. BigQuery requires `DECLARE` before any other statements in a script, so prepending `SET @@reservation` would fail. The package strips leading whitespace and SQL comments (`--`, `#`, `/* */`) to reliably detect this case. `DECLARE` inside `BEGIN...END` or `EXECUTE IMMEDIATE` is not flagged — reservation is applied normally in those cases.
54+
5255
## Release Process
5356

5457
See [CONTRIBUTING.md](../CONTRIBUTING.md#release-process) for the full release workflow steps.

.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ jobs:
1010
runs-on: ubuntu-latest
1111
strategy:
1212
matrix:
13-
dataform-version: ['2.4.2', '3.0.43']
13+
dataform-version: ['2.4.2', '3.0.48']
1414

1515
steps:
1616
- name: Checkout code

CHANGELOG.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2727

2828
- **SQL statement prepending logic refactored**
2929
- **Reservation lookup performance improved** - Refactored to use a Set for O(1) lookups instead of array iteration, significantly improving performance for large reservation lists
30+
- **Updated test version for v3.0 from 3.0.43 to 3.0.48** - Updated the Dataform version used in matrix testing to ensure compatibility with the latest stable release.
31+
3032

3133
## [0.2.0] - 2026-01-20
3234

3335
### Added
3436

3537
- **`autoAssignActions()` method** - Primary integration approach that automatically assigns actions to reservations to all Dataform actions globally without requiring manual code in each action file
36-
- **Matrix testing infrastructure** - Automated testing across multiple Dataform versions (currently - v2.4.2 and v3.0.43)
38+
- **Matrix testing** - Automated testing across multiple Dataform versions (currently - v2.4.2 and v3.0.43)
3739
- **API Reference section** in README with comprehensive documentation of all exported methods
3840

3941
## [0.1.0] - 2025-10-27

CONTRIBUTING.md

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,57 +4,68 @@ We welcome contributions to the Dataform package! This document provides guideli
44

55
## Development Setup
66

7-
1. **Clone the repository:**
7+
### 1. **Clone the repository:**
88

99
```bash
1010
git clone https://github.com/masthead-data/dataform-package.git
1111
cd dataform-package
1212
```
1313

14-
2. **Install dependencies:**
14+
### 2. **Install dependencies:**
1515

1616
```bash
1717
npm install
1818
```
1919

20-
3. **Run tests:**
20+
### 3. **Run tests:**
21+
22+
#### 3.1 Matrix Testing (Default)
2123

22-
#### Matrix Testing (Default)
2324
Run from the root to test all supported versions:
25+
2426
```bash
2527
npm test
2628
```
29+
2730
This command iterates through all supported Dataform versions (currently v2.4.2 and v3.X.X), managing configuration file conflicts automatically.
2831

29-
#### Single Version (Fast Iteration)
32+
#### 3.2 Single Version (Fast Iteration)
33+
3034
For rapid development on the version currently installed in `test-project`:
35+
3136
```bash
3237
npm run test:single
3338
```
39+
3440
This runs:
3541
1. `jest`: Unit tests for helper functions.
3642
2. `dataform compile`: Generates the actual project graph.
3743
3. `verify_compilation.js`: In-depth JSON inspection.
3844

39-
#### Specific Version
45+
#### 3.3 Specific Version
46+
4047
Test a single Dataform version:
48+
4149
```bash
4250
npm test -- 2.4.2
4351
```
4452

45-
4. **Run linting:**
53+
### 4. **Run linting:**
4654

4755
```bash
4856
npm run lint
4957
```
5058

5159
### Local Integration Testing
60+
5261
The `test-project` is configured to use the local version of the package. In `test-project/package.json`:
62+
5363
```json
5464
"dependencies": {
5565
"@masthead-data/dataform-package": "file:../"
5666
}
5767
```
68+
5869
**Note:** `npm ci` or `npm install` in the `test-project` caches the local package. If you make changes to `index.js` and don't see them reflected, you may need to force an update or avoid `npm ci` during rapid iteration.
5970

6071
## Project Structure
@@ -66,8 +77,7 @@ dataform-package/
6677
│ └── index.test.js # Test suite
6778
├── package.json # Package configuration
6879
├── README.md # Main documentation
69-
├── CHANGELOG.md # Version history
70-
└── .eslintrc.js # ESLint configuration
80+
└── CHANGELOG.md # Version history
7181
```
7282

7383
## Making Changes
@@ -80,6 +90,7 @@ This project uses `npm ci` in CI/CD pipelines, which requires `package-lock.json
8090
The project includes optional platform-specific dependencies (e.g., `@unrs/resolver-binding-*`). If you update dependencies on macOS, `npm` might "clean" other platform bindings from the lockfile, causing CI to fail on Linux.
8191

8292
If CI fails with `npm error EUSAGE` related to missing platform bindings:
93+
8394
1. Restore the `package-lock.json` to a known good state.
8495
2. Run `npm install` to update the version/dependencies without removing optional bindings.
8596
3. Verify that the lockfile still contains entries for `@unrs/resolver-binding-linux-*` before committing.

README.md

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,13 @@ autoAssignActions(RESERVATION_CONFIG);
6161

6262
With automated assignement, you don't need to edit your individual action files — the package handles everything globally.
6363

64+
#### Limitations of Automated Assignment
65+
66+
* `DECLARE` at the top level of the SQL (the first real statement after whitespace/comments).
67+
The automation skips operations where `DECLARE` is the first statement at the outer level. BigQuery requires `DECLARE` to appear before any other statements in a script, so prepending `SET @@reservation` would cause a syntax error. This detection works automatically without any configuration needed.
68+
69+
Use manual assignment for any actions that require top-level `DECLARE` statements.
70+
6471
### Manual Assignment (Optional)
6572

6673
For more granular control, you can manually apply reservations per file. Create a setter function in your global scope under `/includes` directory:
@@ -218,8 +225,8 @@ Extracts the action name from a Dataform context object.
218225

219226
This package is tested and compatible with:
220227

221-
* **Dataform v2.4.2**
222-
* **Dataform v3 - latest version**
228+
* **Dataform v2.x** (e.g., v2.4.2)
229+
* **Dataform v3.x**
223230

224231
## Under the Hood
225232

@@ -234,13 +241,15 @@ The package supports various Dataform contexts for action name detection:
234241

235242
Actions are matched against the `RESERVATION_CONFIG` using exact string matching. The action is assigned to the first matching reservation. If no match is found, the actions is assigned to the default reservation (first entry with `null` reservation). If no default is defined, no reservation override is applied.
236243

237-
### SQL Generation
244+
### Reservation Assignment Implementation
245+
246+
Based on the matched reservation, the package automatically prepends the `SET @@reservation` SQL statement to your queries or pre-operations.
238247

239-
Based on the matched reservation, the system generates appropriate SQL:
248+
The specific reservation value applied follows this logic:
240249

241-
* **Specific Reservation**: `SET @@reservation='projects/{project}/locations/{location}/reservations/{name}';`
242-
* **On-demand**: `SET @@reservation='none';`
243-
* **Default/Null**: Empty string (no reservation override)
250+
* **Specific Reservation**: `projects/{project}/locations/{location}/reservations/{name}`
251+
* **On-demand**: `none`
252+
* **Default/Null**: No reservation override applied
244253

245254
### Limitations
246255

0 commit comments

Comments
 (0)