Skip to content

Commit 62050fb

Browse files
wagjaminwagjamin
authored
Add Support for Firebolt Core (#152)
1 parent f71dc5d commit 62050fb

49 files changed

Lines changed: 2548 additions & 385 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.cursor/rules/architecture/RULE.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
alwaysApply: false
3+
---
4+
# Firebolt Node.js SDK Architecture Rules
5+
6+
## Version Architecture (Critical)
7+
8+
**V1 (Legacy)**: Username/password auth → `ConnectionV1`, `QueryFormatterV1`, `DatabaseServiceV1`, `EngineServiceV1`
9+
**V2 (Current)**: Service account auth (client_id/secret) → `ConnectionV2`, `QueryFormatterV2`, `DatabaseServiceV2`, `EngineServiceV2`
10+
**Core (Self-Hosted)**: `FireboltCore()` auth → `ConnectionCore`, `QueryFormatterV2`, no ResourceManager, no async queries, no transactions
11+
12+
Version selected in `makeConnection()` based on auth type. **Always support all versions** unless explicitly version-specific feature.
13+
14+
## Core Patterns
15+
16+
**Dependency Injection**: Factory functions (`FireboltClient()`, `ResourceClient()`) accept `logger` and `httpClient`, return configured instances. Context object passes dependencies.
17+
18+
**Abstract Base Classes**: `Connection` (base) → `ConnectionV1`/`ConnectionV2`. `QueryFormatter` (base) → `QueryFormatterV1`/`QueryFormatterV2`. Base classes contain shared logic.
19+
20+
**Statement Types**:
21+
- `Statement`: `execute()` → `fetchResult()` (in-memory) or `streamResult()` (in-memory stream, not true streaming)
22+
- `StreamStatement`: `executeStream()` → true server-side streaming
23+
- `AsyncStatement`: `executeAsync()` → returns token, no immediate data
24+
25+
## Query Execution Flow
26+
27+
1. `prepareQuery()`: Format with `QueryFormatter` (handles `?`, `:name`, or `$1`/$2` for server-side)
28+
2. `getRequestUrl()`: Add query params and settings
29+
3. `executeQuery()`: POST to engine endpoint
30+
4. `processHeaders()`: Update session parameters from response headers
31+
5. Parse JSON, handle errors via `throwErrorIfErrorBody()`
32+
6. Return appropriate Statement type
33+
34+
## Parameter Management
35+
36+
Connection maintains session parameters:
37+
- **Immutable**: `database`, `account_id`, `output_format` (cannot be removed)
38+
- **Mutable**: Updated via `SET` statements or response headers
39+
- **Server headers**: `Firebolt-Update-Parameters`, `Firebolt-Update-Endpoint`, `Firebolt-Reset-Session`, `Firebolt-Remove-Parameters`
40+
41+
## Authentication & Caching
42+
43+
**Managed Firebolt**: `Authenticator` handles OAuth tokens with thread-safe caching (read/write locks via `rwlock`). Cache key: `{clientId, secret, apiEndpoint}`. Tokens expire at 50% of actual expiry for safety. Disable with `useCache: false`.
44+
45+
**Firebolt Core**: `CoreAuthenticator` provides no-op authentication (no tokens, no caching). Core connections don't require authentication.
46+
47+
## Engine Endpoint Resolution
48+
49+
**V2**: Get system engine URL → connect → `USE DATABASE` → `USE ENGINE` (if specified)
50+
**V1**: Resolve account ID → get engine URL by database/engine → direct connection
51+
**Core**: `engineEndpoint` must be provided explicitly in connection options (no resolution needed)
52+
53+
## Error Handling
54+
55+
Use `CompositeError` for multiple errors. Custom errors: `AccountNotFoundError`, `AuthenticationError`, `ApiError` (from `src/common/errors.ts`).
56+
57+
## Important Details
58+
59+
- **Prepared statements**: `native` (default, client-side `?`/`:name`) vs `fb_numeric` (server-side `$1`/`$2`)
60+
- **Transactions**: `begin()`, `commit()`, `rollback()` on Connection (state per connection).
61+
- **Async queries**: `async: true` setting → token for `isAsyncQueryRunning()`, `isAsyncQuerySuccessful()`, `cancelAsyncQuery()`. **Not supported in Core** - methods throw errors.
62+
- **ResourceManager**: Available for V1/V2 managed connections. **Not available in Core** - accessing `resourceManager` throws error.
63+
- **Result hydration**: SQL types → JS types (dates, BigNumber for large ints, normalization)
64+
- **Connection cleanup**: `destroy()` aborts all active requests
65+
- **Response formats**: `JSON_COMPACT` (default), `JSON`, `JSON_LINES`
66+
67+
## Development Rules
68+
69+
1. Support V1, V2, and Core unless version-specific feature
70+
2. Use abstract base classes for shared functionality (`Connection`, `QueryFormatter`, `Authenticator`)
71+
3. Keep DI pattern for testability
72+
4. Use custom error types from `src/common/errors.ts`
73+
5. Maintain separate V1/V2/Core test suites
74+
6. V1 is legacy; prioritize V2 for new features
75+
7. **Core limitations**: No ResourceManager, no async queries, no transactions. Use `CoreAuthenticator` for no-op auth. `engineEndpoint` required.
76+
8. **Type guards**: Use `"type" in auth && auth.type === "firebolt-core"` to detect Core connections
77+

‎.env.example‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
FIREBOLT_USERNAME=
2-
FIREBOLT_PASSWORD=
1+
FIREBOLT_CLIENT_ID=
2+
FIREBOLT_CLIENT_SECRET=
3+
FIREBOLT_ACCOUNT=
34
FIREBOLT_DATABASE=
45
FIREBOLT_ENGINE_NAME=
5-
FIREBOLT_ENGINE_ENDPOINT=
6-
FIREBOLT_API_ENDPOINT=
6+
FIREBOLT_CORE_ENDPOINT="http://localhost:3473"

‎.github/actions/run-core-integration-tests/action.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: 'Run Core Integration Tests'
2+
description: 'Setup, run, and teardown Firebolt Core integration tests'
3+
inputs:
4+
database:
5+
description: 'Database name for Firebolt Core'
6+
required: false
7+
default: 'firebolt'
8+
endpoint:
9+
description: 'Firebolt Core endpoint'
10+
required: false
11+
default: 'http://127.0.0.1:3473'
12+
runs:
13+
using: 'composite'
14+
steps:
15+
- name: Setup Firebolt Core
16+
id: setup-core
17+
uses: ./.github/actions/setup-firebolt-core
18+
with:
19+
database: ${{ inputs.database }}
20+
endpoint: ${{ inputs.endpoint }}
21+
22+
- name: Run Core integration tests
23+
shell: bash
24+
env:
25+
FIREBOLT_DATABASE: ${{ steps.setup-core.outputs.database }}
26+
FIREBOLT_CORE_ENDPOINT: ${{ steps.setup-core.outputs.endpoint }}
27+
run: |
28+
npm run test:ci integration/core
29+
30+
- name: Teardown Firebolt Core
31+
if: always()
32+
uses: ./.github/actions/teardown-firebolt-core
33+

‎.github/actions/setup-firebolt-core/action.yml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
name: 'Setup Firebolt Core'
2+
description: 'Start Firebolt Core in Docker and wait for it to be ready'
3+
inputs:
4+
database:
5+
description: 'Database name for Firebolt Core'
6+
required: false
7+
default: 'firebolt'
8+
endpoint:
9+
description: 'Firebolt Core endpoint'
10+
required: false
11+
default: 'http://127.0.0.1:3473'
12+
outputs:
13+
database:
14+
description: 'Database name'
15+
value: ${{ inputs.database }}
16+
endpoint:
17+
description: 'Firebolt Core endpoint'
18+
value: ${{ inputs.endpoint }}
19+
runs:
20+
using: 'composite'
21+
steps:
22+
- name: Create firebolt-core-data directory
23+
shell: bash
24+
run: |
25+
mkdir -p ./firebolt-core-data
26+
# Firebolt Core runs as user 1111:1111, so we need to set ownership
27+
sudo chown -R 1111:1111 ./firebolt-core-data
28+
sudo chmod 755 ./firebolt-core-data
29+
30+
- name: Start Firebolt Core
31+
shell: bash
32+
run: |
33+
docker run -d --rm --name firebolt-core \
34+
--ulimit memlock=8589934592:8589934592 \
35+
--security-opt seccomp=unconfined \
36+
-p 127.0.0.1:3473:3473 \
37+
-v $(pwd)/firebolt-core-data:/firebolt-core/volume \
38+
ghcr.io/firebolt-db/firebolt-core:preview-rc
39+
40+
- name: Wait for Firebolt Core to be ready
41+
shell: bash
42+
run: |
43+
echo "Waiting for Firebolt Core to be ready..."
44+
timeout=30
45+
elapsed=0
46+
while [ $elapsed -lt $timeout ]; do
47+
# Try to connect and execute a simple query
48+
response=$(curl -s -w "\n%{http_code}" -X POST "${{ inputs.endpoint }}/" \
49+
--data-binary "SELECT 1" 2>&1) || response=""
50+
http_code=$(echo "$response" | tail -n1)
51+
52+
if [ "$http_code" = "200" ]; then
53+
echo "Firebolt Core is ready!"
54+
break
55+
fi
56+
echo "Waiting... ($elapsed/$timeout seconds) - HTTP $http_code"
57+
sleep 3
58+
elapsed=$((elapsed + 3))
59+
done
60+
61+
if [ $elapsed -ge $timeout ]; then
62+
echo "Firebolt Core failed to start within $timeout seconds"
63+
echo "Container logs:"
64+
docker logs firebolt-core || true
65+
echo "Container status:"
66+
docker ps -a | grep firebolt-core || true
67+
exit 1
68+
fi
69+
70+
# Give Core a bit more time to fully initialize
71+
echo "Core is responding, waiting additional 5 seconds for full initialization..."
72+
sleep 5
73+
74+
- name: Create database if it doesn't exist
75+
shell: bash
76+
run: |
77+
echo "Creating database '${{ inputs.database }}' ..."
78+
# Create the database using a SQL command
79+
response=$(curl -s -w "\n%{http_code}" -X POST "${{ inputs.endpoint }}/?database=default" \
80+
--data-binary "CREATE DATABASE IF NOT EXISTS \"${{ inputs.database }}\"" 2>&1)
81+
http_code=$(echo "$response" | tail -n1)
82+
83+
if [ "$http_code" = "200" ]; then
84+
echo "Database '${{ inputs.database }}' created or already exists"
85+
else
86+
echo "Warning: Database creation returned HTTP $http_code"
87+
echo "Response: $response"
88+
exit 1
89+
fi
90+
91+
- name: Set outputs
92+
shell: bash
93+
run: |
94+
echo "database=${{ inputs.database }}" >> $GITHUB_OUTPUT
95+
echo "endpoint=${{ inputs.endpoint }}" >> $GITHUB_OUTPUT
96+

‎.github/actions/teardown-firebolt-core/action.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
name: 'Teardown Firebolt Core'
2+
description: 'Stop Firebolt Core Docker container'
3+
runs:
4+
using: 'composite'
5+
steps:
6+
- name: Stop Firebolt Core
7+
shell: bash
8+
run: docker stop firebolt-core || true
9+

‎.github/workflows/integration-tests-core.yaml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: Run integration tests for Core
2+
on:
3+
workflow_dispatch:
4+
workflow_call:
5+
inputs:
6+
token:
7+
description: 'GitHub token if called from another workflow'
8+
required: false
9+
type: string
10+
11+
jobs:
12+
core-tests:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- name: Check out code
16+
uses: actions/checkout@v4
17+
18+
- name: Set up node.js
19+
uses: actions/setup-node@v6
20+
with:
21+
node-version: '24'
22+
23+
- name: Install dependencies
24+
run: npm install
25+
26+
- name: Run Core integration tests
27+
uses: ./.github/actions/run-core-integration-tests
28+
29+
allure-report:
30+
needs: core-tests
31+
runs-on: ubuntu-latest
32+
if: always() && github.repository == 'firebolt-db/firebolt-node-sdk'
33+
steps:
34+
# Need to pull the pages branch in order to fetch the previous runs
35+
# Only run Allure reporting on the main repository, not forks
36+
- name: Get Allure history
37+
uses: actions/checkout@v4
38+
continue-on-error: true
39+
with:
40+
ref: gh-pages
41+
path: gh-pages
42+
43+
- name: Allure Report
44+
uses: firebolt-db/action-allure-report@v1
45+
with:
46+
github-key: ${{ inputs.token || secrets.GITHUB_TOKEN }}
47+
test-type: integration-core
48+
allure-dir: allure-results
49+
pages-branch: gh-pages
50+
repository-name: firebolt-node-sdk
51+

‎.github/workflows/integration-tests-v2.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,17 +57,18 @@ jobs:
5757
npm run test:ci integration/v2
5858
5959
# Need to pull the pages branch in order to fetch the previous runs
60+
# Only run Allure reporting on the main repository, not forks
6061
- name: Get Allure history
6162
uses: actions/checkout@v6
62-
if: always()
63+
if: always() && github.repository == 'firebolt-db/firebolt-node-sdk'
6364
continue-on-error: true
6465
with:
6566
ref: gh-pages
6667
path: gh-pages
6768

6869
- name: Allure Report
6970
uses: firebolt-db/action-allure-report@v1
70-
if: always()
71+
if: always() && github.repository == 'firebolt-db/firebolt-node-sdk'
7172
with:
7273
github-key: ${{ inputs.token || secrets.GITHUB_TOKEN }}
7374
test-type: integration

‎.github/workflows/integration-tests.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,6 @@ jobs:
2929
secrets:
3030
FIREBOLT_CLIENT_ID_STG_NEW_IDN: ${{ secrets.FIREBOLT_CLIENT_ID_STG_NEW_IDN }}
3131
FIREBOLT_CLIENT_SECRET_STG_NEW_IDN: ${{ secrets.FIREBOLT_CLIENT_SECRET_STG_NEW_IDN }}
32+
integration-test-core:
33+
uses: ./.github/workflows/integration-tests-core.yaml
3234

‎.github/workflows/nightly.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,12 @@ jobs:
4545
run: |
4646
npm run test:ci integration/v2
4747
48+
- name: Run Core integration tests
49+
if: matrix.os == 'ubuntu-latest'
50+
uses: ./.github/actions/run-core-integration-tests
51+
4852
- name: Slack Notify of failure
4953
if: failure()
50-
id: slack
5154
uses: firebolt-db/action-slack-nightly-notify@v1
5255
with:
5356
os: ${{ matrix.os }}

0 commit comments

Comments
 (0)