Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
058ecc1
Add detection for outer DECLARE statements and update documentation
max-ostapenko Feb 17, 2026
408bb7a
feat: enhance reservation handling with native support detection
max-ostapenko Mar 5, 2026
719db94
Bump eslint from 10.0.0 to 10.0.1 (#41)
dependabot[bot] Feb 23, 2026
09840a4
Bump @dataform/core from 3.0.46 to 3.0.47 in /test-project (#42)
dependabot[bot] Feb 23, 2026
aa8a15a
Bump @dataform/cli from 3.0.46 to 3.0.47 in /test-project (#43)
dependabot[bot] Feb 23, 2026
02ff9af
Bump eslint from 10.0.1 to 10.0.2 (#48)
dependabot[bot] Mar 2, 2026
909747e
Bump globals from 17.3.0 to 17.4.0 (#49)
dependabot[bot] Mar 2, 2026
75897fc
Bump the npm_and_yarn group across 2 directories with 1 update (#44)
dependabot[bot] Mar 5, 2026
4a82b89
Update Dataform version in CI matrix and changelog to 3.0.48
max-ostapenko Mar 8, 2026
533850d
Refactor isNativeReservationSupported to always return false
max-ostapenko Mar 8, 2026
50f3235
Bump eslint from 10.0.0 to 10.0.1 (#41)
dependabot[bot] Feb 23, 2026
3d3e7ca
🧹 Refactor duplicated logic in applyReservationToAction (#45)
max-ostapenko Mar 5, 2026
811cb5f
⚡ Optimize reservation lookup using Map (#47)
max-ostapenko Mar 6, 2026
7f96fb5
fix badge link
max-ostapenko Mar 6, 2026
804e997
Update changelog for version 0.2.1
max-ostapenko Mar 6, 2026
eac4e2d
🧪 [testing improvement] Missing test for compiled objects (proto.preO…
max-ostapenko Mar 6, 2026
40caf9c
Remove local integration testing instructions from CONTRIBUTING.md an…
max-ostapenko Mar 9, 2026
27006ef
fix createReservationSetter to directly use actionToReservation from …
max-ostapenko Mar 9, 2026
0a5377c
refactor createReservationSetter
max-ostapenko Mar 9, 2026
61b7421
Merge branch 'main' into friendly-peafowl
max-ostapenko Mar 9, 2026
8ab74e9
Update CONTRIBUTING.md and README.md for clarity and additional testi…
max-ostapenko Mar 9, 2026
654c8f9
Remove duplicated isArrayOrString and prependStatement functions from…
max-ostapenko Mar 9, 2026
9d2ded6
lint
max-ostapenko Mar 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@ For `operations`, the SQL is often set via `.queries(["SQL"])`. This method can
### 4. Assertions
Assertions in Dataform are strict. They expect a single `SELECT` statement. Prepending a `SET` statement will cause a syntax error in BigQuery because assertions are often wrapped in subqueries or views by Dataform. We explicitly skip assertions in this package.

### 5. Outer DECLARE Detection
Operations where `DECLARE` is the first statement at the outer level are automatically skipped. BigQuery requires `DECLARE` before any other statements in a script, so prepending `SET @@reservation` would fail. The package strips leading whitespace and SQL comments (`--`, `#`, `/* */`) to reliably detect this case. `DECLARE` inside `BEGIN...END` or `EXECUTE IMMEDIATE` is not flagged — reservation is applied normally in those cases.

## Release Process

See [CONTRIBUTING.md](../CONTRIBUTING.md#release-process) for the full release workflow steps.
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
dataform-version: ['2.4.2', '3.0.43']
dataform-version: ['2.4.2', '3.0.48']

steps:
- name: Checkout code
Expand Down
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- **SQL statement prepending logic refactored**
- **Reservation lookup performance improved** - Refactored to use a Set for O(1) lookups instead of array iteration, significantly improving performance for large reservation lists
- **Updated test version for v3.0 from 3.0.43 to 3.0.48** - Updated the Dataform version used in matrix testing to ensure compatibility with the latest stable release.


## [0.2.0] - 2026-01-20

### Added

- **`autoAssignActions()` method** - Primary integration approach that automatically assigns actions to reservations to all Dataform actions globally without requiring manual code in each action file
- **Matrix testing infrastructure** - Automated testing across multiple Dataform versions (currently - v2.4.2 and v3.0.43)
- **Matrix testing** - Automated testing across multiple Dataform versions (currently - v2.4.2 and v3.0.43)
- **API Reference section** in README with comprehensive documentation of all exported methods

## [0.1.0] - 2025-10-27
Expand Down
29 changes: 20 additions & 9 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,57 +4,68 @@ We welcome contributions to the Dataform package! This document provides guideli

## Development Setup

1. **Clone the repository:**
### 1. **Clone the repository:**

```bash
git clone https://github.com/masthead-data/dataform-package.git
cd dataform-package
```

2. **Install dependencies:**
### 2. **Install dependencies:**

```bash
npm install
```

3. **Run tests:**
### 3. **Run tests:**

#### 3.1 Matrix Testing (Default)

#### Matrix Testing (Default)
Run from the root to test all supported versions:

```bash
npm test
```

This command iterates through all supported Dataform versions (currently v2.4.2 and v3.X.X), managing configuration file conflicts automatically.

#### Single Version (Fast Iteration)
#### 3.2 Single Version (Fast Iteration)

For rapid development on the version currently installed in `test-project`:

```bash
npm run test:single
```

This runs:
1. `jest`: Unit tests for helper functions.
2. `dataform compile`: Generates the actual project graph.
3. `verify_compilation.js`: In-depth JSON inspection.

#### Specific Version
#### 3.3 Specific Version

Test a single Dataform version:

```bash
npm test -- 2.4.2
```

4. **Run linting:**
### 4. **Run linting:**

```bash
npm run lint
```

### Local Integration Testing

The `test-project` is configured to use the local version of the package. In `test-project/package.json`:

```json
"dependencies": {
"@masthead-data/dataform-package": "file:../"
}
```

**Note:** `npm ci` or `npm install` in the `test-project` caches the local package. If you make changes to `index.js` and don't see them reflected, you may need to force an update or avoid `npm ci` during rapid iteration.

## Project Structure
Expand All @@ -66,8 +77,7 @@ dataform-package/
│ └── index.test.js # Test suite
├── package.json # Package configuration
├── README.md # Main documentation
├── CHANGELOG.md # Version history
└── .eslintrc.js # ESLint configuration
└── CHANGELOG.md # Version history
```

## Making Changes
Expand All @@ -80,6 +90,7 @@ This project uses `npm ci` in CI/CD pipelines, which requires `package-lock.json
The project includes optional platform-specific dependencies (e.g., `@unrs/resolver-binding-*`). If you update dependencies on macOS, `npm` might "clean" other platform bindings from the lockfile, causing CI to fail on Linux.

If CI fails with `npm error EUSAGE` related to missing platform bindings:

1. Restore the `package-lock.json` to a known good state.
2. Run `npm install` to update the version/dependencies without removing optional bindings.
3. Verify that the lockfile still contains entries for `@unrs/resolver-binding-linux-*` before committing.
Expand Down
23 changes: 16 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,13 @@ autoAssignActions(RESERVATION_CONFIG);

With automated assignement, you don't need to edit your individual action files — the package handles everything globally.

#### Limitations of Automated Assignment

* `DECLARE` at the top level of the SQL (the first real statement after whitespace/comments).
The automation skips operations where `DECLARE` is the first statement at the outer level. BigQuery requires `DECLARE` to appear before any other statements in a script, so prepending `SET @@reservation` would cause a syntax error. This detection works automatically without any configuration needed.

Use manual assignment for any actions that require top-level `DECLARE` statements.

### Manual Assignment (Optional)

For more granular control, you can manually apply reservations per file. Create a setter function in your global scope under `/includes` directory:
Expand Down Expand Up @@ -218,8 +225,8 @@ Extracts the action name from a Dataform context object.

This package is tested and compatible with:

* **Dataform v2.4.2**
* **Dataform v3 - latest version**
* **Dataform v2.x** (e.g., v2.4.2)
* **Dataform v3.x**

## Under the Hood

Expand All @@ -234,13 +241,15 @@ The package supports various Dataform contexts for action name detection:

Actions are matched against the `RESERVATION_CONFIG` using exact string matching. The action is assigned to the first matching reservation. If no match is found, the actions is assigned to the default reservation (first entry with `null` reservation). If no default is defined, no reservation override is applied.

### SQL Generation
### Reservation Assignment Implementation

Based on the matched reservation, the package automatically prepends the `SET @@reservation` SQL statement to your queries or pre-operations.

Based on the matched reservation, the system generates appropriate SQL:
The specific reservation value applied follows this logic:

* **Specific Reservation**: `SET @@reservation='projects/{project}/locations/{location}/reservations/{name}';`
* **On-demand**: `SET @@reservation='none';`
* **Default/Null**: Empty string (no reservation override)
* **Specific Reservation**: `projects/{project}/locations/{location}/reservations/{name}`
* **On-demand**: `none`
* **Default/Null**: No reservation override applied

### Limitations

Expand Down
Loading