Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ wss-*-agent.config
wss-unified-agent.jar
whitesource/
.nyc_output
/.claude/*.local.*

rsa_*.p8

# SSH private key for WIF tests
Expand Down
1 change: 1 addition & 0 deletions .npmignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
*
!dist/**/*
dist/test/**/*
dist/**/*.map
!LICENSE
!README.md
!SECURITY.md
119 changes: 119 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
## About this fork

This is a fork of [`snowflake-connector-nodejs`](https://github.com/snowflakedb/snowflake-connector-nodejs) published as **`@naturalcycles/snowflake-sdk`**.

The motivating change vs. upstream is that heavy cloud SDKs are declared as **optional `peerDependencies`** rather than hard `dependencies`. Consumers who don't use a particular cloud's stage or workload-identity features don't have to install its SDK. The current peer-dep set:

- AWS: `@aws-sdk/client-s3`, `@aws-sdk/client-sts`, `@aws-sdk/credential-provider-node`, `@aws-sdk/ec2-metadata-service`, `@aws-crypto/sha256-js`, `@smithy/node-http-handler`, `@smithy/protocol-http`, `@smithy/signature-v4`
- Azure: `@azure/storage-blob`, `@azure/identity`
- GCP: `google-auth-library`
- Plus `asn1.js` (inherited from upstream's own optional peer)

These same packages are **also listed in `devDependencies`** so local dev/CI installs them and the TypeScript build (`npm run prepack`) can resolve their types. They are only optional for downstream consumers.

Two long-lived branches:

- `master` tracks upstream Snowflake releases. Sync via the `upstream` remote (`https://github.com/snowflakedb/snowflake-connector-nodejs.git`). Merge `upstream/master` → local `master`, push, then merge `master` → `next`.
- `next` is the active fork branch and the default target for PRs in this repo.

Re-sync risks to watch for when merging upstream into `next`:

- Upstream's `package.json` keeps adding hard cloud-SDK deps over time. Each sync must move new ones into `peerDependencies` (+ optional `peerDependenciesMeta` + duplicate in `devDependencies`).
- Upstream writes new code with **static `import`/`require`** from cloud SDKs. The static imports compile cleanly because we have the SDKs in `devDependencies`, but at runtime a consumer who hasn't installed them will throw on first `require` of the module. **The lazy-require discipline is the load-bearing invariant of this fork.** Files that currently follow it (don't break them, and apply the same pattern to any new peer-SDK callsite):
- `lib/telemetry/platform_detection.ts` — uses `import type` only; requires `@aws-sdk/client-sts` inside `hasAwsIdentity()`. **Critical** because `lib/services/sf.js` requires this module on every connection.
- `lib/authentication/auth_workload_identity/attestation_aws.ts` — `import type` only; the AWS / `@smithy/*` / `@aws-crypto/sha256-js` bundle is loaded by an internal `awsSdk()` helper on first use. **Critical** because `lib/authentication/authentication.js` requires `auth_workload_identity` eagerly.
- `lib/authentication/auth_workload_identity/attestation_azure.ts` and `attestation_gcp.ts` — same pattern with `@azure/identity` and `google-auth-library`.
- `lib/file_transfer_agent/s3_util.js` — `@aws-sdk/client-s3` is required inside the `S3Util` constructor, `@smithy/node-http-handler` is required inside the proxy `if` block. **Critical** because `remote_storage_util.js` requires this module eagerly when `Statement` loads.
- `lib/file_transfer_agent/azure_util.js` — `@azure/storage-blob` is required inside `createClient()`. **Critical** for the same reason.
- The pattern in TypeScript files:
```ts
import type { STSClient as _STSClient } from '@aws-sdk/client-sts';
// ...inside the function that actually uses it:
const { STSClient } = require('@aws-sdk/client-sts') as { STSClient: typeof _STSClient };
```
Cast to `typeof <Type>` so callers keep their type info and the response type isn't widened.
- Verify after every upstream sync: stash `node_modules/@aws-sdk`, `@aws-crypto`, `@smithy`, `@azure`, `google-auth-library` aside, then `node -e "require('./dist'); const c = require('./dist').createConnection({account:'x',username:'u',password:'p'}); require('./dist/lib/authentication/auth_workload_identity/auth_workload_identity'); require('./dist/lib/telemetry/platform_detection');"` — must complete without `MODULE_NOT_FOUND`.
- The TS declaration in `index.d.ts` uses `declare module '@naturalcycles/snowflake-sdk'` — single divergence point for the module name. The file is copied verbatim into `dist/index.d.ts` by `ci/build_typescript.js`.

## Commands

Build (TypeScript → `dist/`):

```bash
npm run prepack # tsc + copy index.d.ts + copy minicore binaries
npm run check-ts # prepack then `tsc --noEmit dist/index.d.ts`
```

Test (mocha, 180s timeout, runs both `.js` and `.ts` via `ts-node/register` from `.mocharc.js`):

```bash
npm test # unit tests
npm run test:unit # same as `npm test`
npm run test:integration # integration — needs SNOWFLAKE_TEST_* env vars
npm run test:authentication # auth flow tests
npm run test:system # system tests
npm run test:manual # interactive auth — needs RUN_MANUAL_TESTS_ONLY=true
npm run test:ci # unit + integration combined
npm run test:ci:coverage # CI tests under nyc

# Single test file (or filter via mocha's -g):
npm run test:single -- test/unit/snowflake_test.js
npm run test:single -- test/unit/snowflake_test.js -g 'pattern'
```

A subset of integration tests requires `python3 ci/container/hang_webserver.py 12345 &` to be running, plus an active wiremock server (`npm run serve-wiremock` on port 8081) for the `test/integration/wiremock/*` cases.

Lint / format (oxlint replaces ESLint; prettier handles formatting):

```bash
npm run lint:check # oxlint .
npm run lint:fix # oxlint --fix .
npm run prettier:check # prettier --check .
npm run prettier:format # prettier -w .
```

`lint-staged` runs `prettier:format` on all staged files and `oxlint --max-warnings=0` on `.js`/`.ts` via the `husky` pre-commit hook (`.husky/pre-commit`). The separate `snowflakedb/casec_precommit` secret-scanner pre-commit (`.pre-commit-config.yaml`) is opt-in via `pre-commit install`.

## Architecture

**Entry points and build:**

- `lib/snowflake.ts` is the source entry. It calls `core()` (`lib/core.js`) with `NodeHttpClient` and the Node logger.
- Root `index.js` re-exports `./lib/snowflake` (resolved by `ts-node` during dev/test).
- The published package's `main` is `./dist/index.js`, generated by `ci/build_typescript.js` (clears `dist/`, runs `tsc`, copies `index.d.ts` and the minicore binaries). The browser build was removed in v2.x.
- `tsconfig.json` has `allowJs: true` and `module: node16`, so `.ts` and `.js` files in `lib/` and `test/` are compiled together. `paths` maps `asn1.js` to a local type stub in `lib/types/asn1.js.d.ts` (asn1.js ships no types).

**`lib/core.js`** is the **factory** that returns the public API (`createConnection`, `createPool`, `configure`, type constants, error codes). It takes pluggable `httpClientClass` and `loggerClass` — historically used to provide a browser variant, now only Node, but the indirection remains.

**Layered structure under `lib/`:**

- **`connection/`** — `Connection`, `ConnectionConfig`, `ConnectionContext`, `Statement`, bind uploading, result handling. A connection owns a `ConnectionContext` carrying config, HttpClient, and services. `normalize_connection_options.ts` and `types.ts` are the v2.x typed entry into option handling.
- **`services/`** — `sf.js` is the Snowflake session service (login, token refresh, query submission state machine). `large_result_set.js` downloads chunked S3/GCS result files.
- **`authentication/`** — one module per auth type. Legacy (`.js`): `auth_default` (password), `auth_idtoken`, `auth_keypair` (JWT), `auth_oauth`, `auth_oauth_authorization_code`, `auth_oauth_pat`, `auth_okta`, `auth_web` (browser SSO). v2.x additions (`.ts`): `auth_oauth_client_credentials`, `auth_coordinator` (orchestrates token caching across pooled connections), `spcs_token` (Snowpark Container Services), and the `auth_workload_identity/` subtree (AWS / Azure / GCP attestation). `authentication.js` is the dispatcher; `secure_storage/json_credential_manager.js` is the default disk-backed token cache.
- **`file_transfer_agent/`** — `PUT` / `GET` stage upload-download. `s3_util.js` (S3, via `@aws-sdk/client-s3` + `@smithy/node-http-handler` for proxy), `azure_util.js` (Azure via `@azure/storage-blob`), `gcs_util.js` (GCS via REST + `google-auth-library` for credentials), `local_util.js` (local stages). Cloud SDKs **must** be loaded lazily (the long-standing pattern is `typeof s3 !== 'undefined' ? s3 : require('@aws-sdk/client-s3')` inside the function that needs it). New code paths that touch a peer SDK must keep this discipline or the optional-peer install will break at first call.
- **`agent/`** — TLS layer. `https_ocsp_agent.js` + `ocsp_response_cache.js` enforce OCSP revocation. `https_proxy_agent.ts` (v2.x) handles outbound proxy and integrates with the new CRL validator. `crl_validator/` is a v2.x addition that fetches and verifies Certificate Revocation Lists, including RSASSA-PSS signature support (`rsassa_pss_parser.ts`). `socket_util.js` and `check.js` are shared helpers.
- **`http/`** — `base.js` (shared logic), `node.ts` (axios + OCSP/CRL agent), `node_untyped.js` (CJS shim), `axios_instance.ts` (single configured axios), `request_util.js` (retry, normalize response, GUID injection).
- **`logger/`** — winston-based (`logger.ts` + `logger/node.js`). `easy_logging_starter.js` reads an external `client_config.json` for log-level/path overrides. `execution_timer.js`, `logging_util.js` are shared. Browser logger was removed.
- **`configuration/`** — `connection_configuration.js` loads from a TOML file (`connections.toml`) when `createConnection()` is called without options. `client_configuration.js` handles the easy-logging JSON file.
- **`global_config.js`** — process-wide settings: `configure({ logLevel, ocspFailOpen/FailClosed/Insecure, customCredentialManager, ... })`. Mutates module state, so tests that touch it must restore it. `global_config_typed.ts` is the typed surface.
- **`secret_detector.js`** — scrubs secrets out of log messages. Anything that logs request/response bodies should go through this.
- **`queryContextCache.js`** — caches per-query context returned by the server to optimize subsequent statements.
- **`disk_cache.ts`** (v2.x) — generic disk-backed cache with permission checks; used by OAuth/PAT token caches.
- **`telemetry/`** (v2.x) — `inband_telemetry.ts` posts client telemetry to Snowflake. `platform_detection.ts`, `application_path.ts`, `libc_details.ts`, `os_details/` collect host info — `platform_detection` statically imports `@aws-sdk/client-sts` for ECS/EC2 attribution, so it's another path that needs the AWS SDK if invoked.
- **`minicore/`** (v2.x) — NAPI Rust module (`rust_minicore/`) shipping prebuilt `.node` binaries for darwin/linux/win × arm64/x64. Used for crypto/parser hot paths. `index.ts` is the JS entry; `minicore.ts` wraps the platform-specific binary. Prebuilds are checked into `lib/minicore/binaries/` and copied to `dist/lib/minicore/binaries/` by the build script.
- **`errors.js`** — central `ErrorCode` enum + `Errors.createClientError(...)`. The numeric codes are part of the public API surface (mirrored in `index.d.ts`); don't renumber them. `error_code.ts` is the typed re-export consumed by the `.d.ts`.
- **`proxy_util.js`** (v2.x) — proxy resolution from connection config / env (`HTTPS_PROXY`, `NO_PROXY`), with per-destination overrides.

**Engine and language baseline:** Node ≥ 18 (`engines.node` in `package.json`; v1.x's Node-6 check is gone). New code goes in `.ts`; the `.ts`/`.js` boundary is fine to cross in either direction. Migration guidance is in the README's "TypeScript Migration" section.

**Tests:** mocha config is in `.mocharc.js` (`ts-node/register`, `extension: ['js','ts']`, `recursive: true`, retries enabled). Wiremock-backed tests live in `test/integration/wiremock/` and consume mappings from `wiremock/mappings/`. Many of the 11 currently-failing unit tests in a fresh checkout are infrastructure-dependent (need `hang_webserver.py` / fixture files) — they fail the same way on a clean `master`, so a clean-merge baseline is `~1001 passing, ~11 failing` until that local setup is in place.

## Code style

`oxlint` (`.oxlintrc.json`) is the linter; configuration is minimal — it's there as a fast gate, not as a strict style enforcer. **Formatting** is `prettier` (`.prettierrc.js`); run `npm run prettier:format` before committing. The pre-commit hook (`.husky/pre-commit` via `lint-staged`) runs both automatically on staged files.

For JetBrains users, `webstorm-codestyle.xml` is still in the repo.

## Git

Never `git push`, unless explicitly asked about it.
15 changes: 13 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,21 @@
This is the fork of [snowflake-connector-nodejs](https://github.com/snowflakedb/snowflake-connector-nodejs)
with the following changes:

- Fixes https://github.com/snowflakedb/snowflake-connector-nodejs/issues/449 by moving non-mandatory dependencies to peerDependencies. So, folks who don't need e.g AWS SDK don't have to "download the whole internet".

Published as [@naturalcycles/snowflake-sdk](https://www.npmjs.com/package/@naturalcycles/snowflake-sdk).

## Readme

---

NodeJS Driver for Snowflake

---

<p>
<a href="https://github.com/snowflakedb/snowflake-connector-nodejs/actions?query=workflow%3A%22Build+and+Test%22+branch%3Amaster" target="_blank"><img src="https://github.com/snowflakedb/snowflake-connector-nodejs/workflows/Build%20and%20Test/badge.svg?branch=master" alt="master" /></a>
<a href="https://www.npmjs.com/package/snowflake-sdk" target="_blank"><img src="https://img.shields.io/npm/v/snowflake-sdk.svg" alt="npm" /></a>
<a href="https://www.npmjs.com/package/@naturalcycles/snowflake-sdk" target="_blank"><img src="https://img.shields.io/npm/v/@naturalcycles/snowflake-sdk.svg" alt="npm" /></a>
<a href="http://www.apache.org/licenses/LICENSE-2.0.txt" target="_blank"><img src="http://img.shields.io/:license-Apache%202-brightgreen.svg" alt="apache" /> </a>
<a href="https://codecov.io/gh/snowflakedb/snowflake-connector-nodejs" target="_blank"><img src="https://codecov.io/gh/snowflakedb/snowflake-connector-nodejs/branch/master/graph/badge.svg?token=QZMWDu35ds" alt="codecov" /></a>
</p>
Expand All @@ -14,7 +25,7 @@ NodeJS Driver for Snowflake

# Install

Run `npm i snowflake-sdk` in your existing NodeJs project.
Run `npm i @naturalcycles/snowflake-sdk` in your existing NodeJs project.

# Docs

Expand Down
2 changes: 1 addition & 1 deletion index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import ErrorCodeEnum from './lib/error_code';
* The snowflake-sdk module provides an instance to connect to the Snowflake server
* @see [source] {@link https://docs.snowflake.com/en/developer-guide/node-js/nodejs-driver}
*/
declare module 'snowflake-sdk' {
declare module '@naturalcycles/snowflake-sdk' {
export type CustomParser = (rawColumnValue: string) => any;
export type Bind = string | number | boolean | null;
export type InsertBinds = readonly Bind[][];
Expand Down
50 changes: 44 additions & 6 deletions lib/authentication/auth_workload_identity/attestation_aws.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,47 @@
import { defaultProvider } from '@aws-sdk/credential-provider-node';
import { STSClient, AssumeRoleCommand, GetWebIdentityTokenCommand } from '@aws-sdk/client-sts';
import { MetadataService } from '@aws-sdk/ec2-metadata-service';
import { HttpRequest } from '@smithy/protocol-http';
import { SignatureV4 } from '@smithy/signature-v4';
import { Sha256 } from '@aws-crypto/sha256-js';
import type { defaultProvider as _defaultProvider } from '@aws-sdk/credential-provider-node';
import type {
STSClient as _STSClient,
AssumeRoleCommand as _AssumeRoleCommand,
GetWebIdentityTokenCommand as _GetWebIdentityTokenCommand,
} from '@aws-sdk/client-sts';
import type { MetadataService as _MetadataService } from '@aws-sdk/ec2-metadata-service';
import type { HttpRequest as _HttpRequest } from '@smithy/protocol-http';
import type { SignatureV4 as _SignatureV4 } from '@smithy/signature-v4';
import type { Sha256 as _Sha256 } from '@aws-crypto/sha256-js';
import Logger from '../../logger';

// AWS / Smithy SDKs are optional peerDependencies in the @naturalcycles/snowflake-sdk fork.
// They're loaded lazily on first WIF use so consumers who don't use AWS workload-identity
// can install the package without pulling in the @aws-sdk/* tree.
type AwsSdk = {
defaultProvider: typeof _defaultProvider;
STSClient: typeof _STSClient;
AssumeRoleCommand: typeof _AssumeRoleCommand;
GetWebIdentityTokenCommand: typeof _GetWebIdentityTokenCommand;
MetadataService: typeof _MetadataService;
HttpRequest: typeof _HttpRequest;
SignatureV4: typeof _SignatureV4;
Sha256: typeof _Sha256;
};
let _sdk: AwsSdk | undefined;
function awsSdk(): AwsSdk {
if (!_sdk) {
_sdk = {
defaultProvider: require('@aws-sdk/credential-provider-node').defaultProvider,
STSClient: require('@aws-sdk/client-sts').STSClient,
AssumeRoleCommand: require('@aws-sdk/client-sts').AssumeRoleCommand,
GetWebIdentityTokenCommand: require('@aws-sdk/client-sts').GetWebIdentityTokenCommand,
MetadataService: require('@aws-sdk/ec2-metadata-service').MetadataService,
HttpRequest: require('@smithy/protocol-http').HttpRequest,
SignatureV4: require('@smithy/signature-v4').SignatureV4,
Sha256: require('@aws-crypto/sha256-js').Sha256,
};
}
return _sdk;
}

export async function getAwsCredentials(region: string, impersonationPath: string[] = []) {
const { defaultProvider, STSClient, AssumeRoleCommand } = awsSdk();
Logger().debug('Getting AWS credentials from default provider');
let credentials = await defaultProvider()();

Expand Down Expand Up @@ -41,6 +76,7 @@ export async function getAwsRegion() {
return process.env.AWS_REGION; // Lambda
} else {
Logger().debug('Getting AWS region from EC2 metadata service');
const { MetadataService } = awsSdk();
return new MetadataService().request('/latest/meta-data/placement/region', {}); // EC2
}
}
Expand Down Expand Up @@ -68,6 +104,7 @@ async function getCallerIdentityToken(
region: string,
credentials: Awaited<ReturnType<typeof getAwsCredentials>>,
) {
const { HttpRequest, SignatureV4, Sha256 } = awsSdk();
const stsHostname = getStsHostname(region);
const request = new HttpRequest({
method: 'POST',
Expand Down Expand Up @@ -103,6 +140,7 @@ async function getOutboundWebIdentityToken(
region: string,
credentials: Awaited<ReturnType<typeof getAwsCredentials>>,
) {
const { STSClient, GetWebIdentityTokenCommand } = awsSdk();
const stsClient = new STSClient({ credentials, region });
const response = await stsClient.send(
new GetWebIdentityTokenCommand({
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { DefaultAzureCredential } from '@azure/identity';
import type { DefaultAzureCredential as _DefaultAzureCredential } from '@azure/identity';
import Logger from '../../logger';

export const DEFAULT_AZURE_ENTRA_ID_RESOURCE = 'api://fd3f753b-eed3-462c-b6a7-a4b5bb650aad';
Expand All @@ -9,6 +9,10 @@ export async function getAzureAttestationToken(
entraIdResource?: string;
} = {},
) {
// @azure/identity is an optional peer in the @naturalcycles/snowflake-sdk fork; load lazily.
const { DefaultAzureCredential } = require('@azure/identity') as {
DefaultAzureCredential: typeof _DefaultAzureCredential;
};
const credential = new DefaultAzureCredential({
managedIdentityClientId: options.managedIdentityClientId,
});
Expand Down
7 changes: 6 additions & 1 deletion lib/authentication/auth_workload_identity/attestation_gcp.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
import { GoogleAuth, Impersonated } from 'google-auth-library';
import type { GoogleAuth as _GoogleAuth, Impersonated as _Impersonated } from 'google-auth-library';
import Logger from '../../logger';

export const SNOWFLAKE_AUDIENCE = 'snowflakecomputing.com';

export async function getGcpAttestationToken(impersonationPath?: string[]) {
// google-auth-library is an optional peer in the @naturalcycles/snowflake-sdk fork; load lazily.
const { GoogleAuth, Impersonated } = require('google-auth-library') as {
GoogleAuth: typeof _GoogleAuth;
Impersonated: typeof _Impersonated;
};
const auth = new GoogleAuth();

if (impersonationPath) {
Expand Down
Loading
Loading