Skip to content

Commit f4d1675

Browse files
authored
[SEA-NodeJS] Unify kernel + driver logging into one DBSQLLogger sink (#417)
* [SEA-NodeJS] Unify kernel + driver logging into one DBSQLLogger sink The Rust kernel emits its diagnostics via `tracing`; in a Node process nothing subscribed to them, so kernel logs were silently dropped — the `DBSQLLogger` only ever saw driver-side lines. This wires the kernel into the SAME logger (and file) the driver uses, so logs from all three layers — driver, napi shim, kernel — land in one place. How: - `SeaBackend.connect()` calls `installKernelLogBridge(binding, logger, level)` (new `lib/sea/SeaLogging.ts`), which registers a JS callback with the binding's `initKernelLogging`. The callback maps each kernel `LogRecord` (`{ level, target, message }`) onto the driver's `LogLevel` and re-emits it through `IClientContext.getLogger()`, tagged `[kernel <target>] …`. - Kernel verbosity follows the driver logger's level: `IDBSQLLogger` gains an optional `getLevel()` (implemented by `DBSQLLogger`); the bridge passes it to the kernel so events the sink would drop never cross the bridge. Defaults to `info` for loggers that don't expose a level. - Process-global, last-writer-wins + graceful no-op on an older binding without the bridge export (logging is advisory) — see SeaLogging docs. Requires kernel napi `initKernelLogging`/`setKernelLogLevel`/`LogRecord` (databricks-sql-kernel#126); `KERNEL_REV` bumped + binding rebuilt so the generated `native/sea/index.d.ts` carries them. Verified: tsc clean, eslint + prettier clean, SEA unit suite 256 passing (incl. new SeaLogging unit tests), and a live end-to-end run confirms a single DBSQLLogger file holds both driver lines and 29 `[kernel databricks::sql::kernel]` lines for one `SELECT 1` (committed as tests/e2e/sea/logging-e2e.test.ts). Co-authored-by: Isaac Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com> * [SEA-NodeJS] Retarget kernel log level on runtime logger.setLevel() Follow-up to the unified-logging wiring: a runtime `logger.setLevel(...)` now retargets the kernel-side bridge too, not just the driver's own transports — keeping kernel verbosity in lock-step with the driver's at runtime, not only at connect. - `IDBSQLLogger` gains an optional `onLevelChange(listener) => unsubscribe`; `DBSQLLogger` implements it (fires subscribers from `setLevel`, swallowing a throwing listener so it can't break level setting). - `installKernelLogBridge` now returns an unsubscribe handle and, when both the logger can notify (`onLevelChange`) and the binding can retarget (`setKernelLogLevel`), subscribes to forward level changes to the kernel. - `SeaBackend` stores the handle and drops it on `close()` (no stale listener; the process-global kernel sink follows the existing last-writer-wins model). Backend-agnostic: `DBSQLLogger` never references the SEA binding — the SEA layer subscribes via the interface. Loggers without `onLevelChange` keep the connect-time level; older bindings without `setKernelLogLevel` simply don't subscribe. Verified: tsc / eslint / prettier clean; unit suite 1183 passing incl. new `DBSQLLogger` level-subscription tests + `SeaLogging` runtime-retarget tests (retarget, unsubscribe, no-notify logger, no-retarget binding). Live: with the logger started at `warn`, a healthy SELECT logged 0 kernel lines; after `logger.setLevel('debug')` mid-session the next SELECT logged 12 `[kernel databricks::sql::kernel]` lines — proving the kernel was retargeted. Co-authored-by: Isaac Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com> * chore(sea): bump KERNEL_REV to kernel main 80b68e1 + regen napi contract Kernel #126 (logging bridge) and #127 (mTLS identity + custom HTTP headers) are both merged to kernel main. Pin KERNEL_REV to the unified main SHA 80b68e1eef3b613910183a50dfa4dace854d50dd and regenerate native/sea/index.* from it. The contract now carries both feature surfaces (gains the mTLS/headers exports from #127). Co-authored-by: Isaac Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com> * fix(sea): align napi package name to @databricks/databricks-sql-kernel-* Kernel #131/#135 renamed the published napi package @databricks/sql-kernel -> @databricks/databricks-sql-kernel (declaring all 8 target triples + dropping private:true so the umbrella can publish). The regenerated native/sea/index.js router now emits the new names, so update the driver to match: the packaging test regex + M0 triple assertion, the version test hint, the SeaNativeLoader install hint, and native/sea/README.md. Fixes the 2 failing native-packaging unit tests surfaced by the KERNEL_REV bump to kernel main 80b68e1. Co-authored-by: Isaac Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com> --------- Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
1 parent bcc0175 commit f4d1675

14 files changed

Lines changed: 728 additions & 45 deletions

KERNEL_REV

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
8bedaabf69f5bce5a957a8775f29dbb8dbdd2e71
1+
80b68e1eef3b613910183a50dfa4dace854d50dd

lib/DBSQLLogger.ts

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ export default class DBSQLLogger implements IDBSQLLogger {
99
file?: winston.transports.FileTransportInstance;
1010
};
1111

12+
// Subscribers notified on `setLevel(...)` — used by the SEA/kernel backend to
13+
// keep the kernel-side log bridge's verbosity in lock-step with this logger.
14+
private levelListeners: Array<(level: LogLevel) => void> = [];
15+
1216
constructor({ level = LogLevel.info, filepath }: LoggerOptions = {}) {
1317
this.transports = {
1418
console: new winston.transports.Console({ handleExceptions: true, level }),
@@ -26,10 +30,32 @@ export default class DBSQLLogger implements IDBSQLLogger {
2630
this.logger.log({ level, message });
2731
}
2832

33+
getLevel(): LogLevel {
34+
return (this.transports.console.level as LogLevel) ?? LogLevel.info;
35+
}
36+
37+
onLevelChange(listener: (level: LogLevel) => void): () => void {
38+
this.levelListeners.push(listener);
39+
return () => {
40+
const index = this.levelListeners.indexOf(listener);
41+
if (index >= 0) {
42+
this.levelListeners.splice(index, 1);
43+
}
44+
};
45+
}
46+
2947
setLevel(level: LogLevel) {
3048
this.transports.console.level = level;
3149
if (this.transports.file) {
3250
this.transports.file.level = level;
3351
}
52+
for (const listener of this.levelListeners) {
53+
// A subscriber must never break level setting for the rest.
54+
try {
55+
listener(level);
56+
} catch {
57+
// swallow — level-change notification is advisory
58+
}
59+
}
3460
}
3561
}

lib/contracts/IDBSQLLogger.ts

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,25 @@ export interface LoggerOptions {
55

66
export default interface IDBSQLLogger {
77
log(level: LogLevel, message: string): void;
8+
9+
/**
10+
* Optional: the logger's current level. When implemented, the SEA/kernel
11+
* backend uses it to set the verbosity of the kernel-side (Rust) log bridge,
12+
* so kernel logs are filtered at the same level as the driver's own logs and
13+
* land in the same sink. Loggers that don't implement it leave the kernel
14+
* bridge at its `info` default.
15+
*/
16+
getLevel?(): LogLevel;
17+
18+
/**
19+
* Optional: subscribe to runtime level changes. When implemented, the
20+
* SEA/kernel backend subscribes so a runtime `setLevel(...)` retargets the
21+
* kernel-side log bridge too (not just the driver's own transports) — keeping
22+
* kernel verbosity in lock-step with the driver's. Returns an unsubscribe
23+
* function. Loggers that don't implement it still get the connect-time level;
24+
* only *runtime* retargeting of the kernel is unavailable.
25+
*/
26+
onLevelChange?(listener: (level: LogLevel) => void): () => void;
827
}
928

1029
export enum LogLevel {

lib/sea/SeaBackend.ts

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import HiveDriverError from '../errors/HiveDriverError';
2222
import { getSeaNative, SeaNativeBinding, SeaConnection } from './SeaNativeLoader';
2323
import { decodeNapiKernelError } from './SeaErrorMapping';
2424
import { buildSeaConnectionOptions, SeaNativeConnectionOptions } from './SeaAuth';
25+
import { installKernelLogBridge } from './SeaLogging';
2526
import SeaSessionBackend from './SeaSessionBackend';
2627

2728
export interface SeaBackendOptions {
@@ -70,6 +71,11 @@ export default class SeaBackend implements IBackend {
7071

7172
private nativeOptions?: SeaNativeConnectionOptions;
7273

74+
// Drops the kernel-log level-change listener on close. No-op until connect()
75+
// installs the bridge (and a no-op closure if the logger/binding can't
76+
// retarget at runtime).
77+
private kernelLogUnsubscribe: () => void = () => {};
78+
7379
constructor(options: SeaBackendOptions) {
7480
this.context = options.context;
7581
this.binding = options.nativeBinding ?? getSeaNative();
@@ -81,6 +87,16 @@ export default class SeaBackend implements IBackend {
8187
// we ever touch the native binding.
8288
this.nativeOptions = buildSeaConnectionOptions(options);
8389

90+
// Bridge the Rust kernel's `tracing` logs into the SAME `DBSQLLogger` the
91+
// driver logs through, so logs from all three layers (driver, napi shim,
92+
// kernel) land in one place — and one file when the logger has a file
93+
// transport. Kernel verbosity follows the logger's own level; loggers that
94+
// don't expose `getLevel()` leave the bridge at `info`. No-op on a binding
95+
// that predates the bridge (logging is advisory).
96+
const logger = this.context.getLogger();
97+
const kernelLogLevel = logger.getLevel?.() ?? LogLevel.info;
98+
this.kernelLogUnsubscribe = installKernelLogBridge(this.binding, logger, kernelLogLevel);
99+
84100
// Warn on the insecure combo: a `customCaCert` paired with
85101
// `checkServerCertificate: false` is almost always a mistake — verification
86102
// is fully off, so the custom trust anchor is never used. The combo is
@@ -149,5 +165,11 @@ export default class SeaBackend implements IBackend {
149165
// No backend-level resources to release — each `SeaSessionBackend`
150166
// owns its own napi `Connection` lifecycle.
151167
this.nativeOptions = undefined;
168+
169+
// Stop retargeting the (process-global) kernel log level from this backend's
170+
// logger; the kernel sink itself is process-global and is replaced by the
171+
// next connect, matching the bridge's last-writer-wins model.
172+
this.kernelLogUnsubscribe();
173+
this.kernelLogUnsubscribe = () => {};
152174
}
153175
}

lib/sea/SeaLogging.ts

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
// Copyright (c) 2026 Databricks, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
/**
16+
* Kernel → driver log bridge.
17+
*
18+
* The Rust kernel emits its diagnostics via `tracing`. In a Node process those
19+
* events have no subscriber and are dropped — so by default the driver's
20+
* `DBSQLLogger` only ever saw JS-side lines. The napi binding's
21+
* `initKernelLogging` installs a process-global subscriber that forwards
22+
* kernel events (batched) to a JS callback; this module wires that callback
23+
* into the **same** `IDBSQLLogger` the driver logs through, so logs from all
24+
* three layers (driver, napi shim, kernel) land in one place — and one file
25+
* when the logger has a file transport.
26+
*
27+
* Verbosity follows the driver's logger level (see `installKernelLogBridge`),
28+
* filtered kernel-side so we don't pay the channel/bridge cost for events the
29+
* sink would discard anyway.
30+
*/
31+
32+
import IDBSQLLogger, { LogLevel } from '../contracts/IDBSQLLogger';
33+
import { SeaNativeBinding, SeaNativeLogRecord } from './SeaNativeLoader';
34+
35+
/**
36+
* Map a kernel level string (`error`/`warn`/`info`/`debug`/`trace`) onto the
37+
* driver's `LogLevel`. The kernel's `trace` has no `LogLevel` analogue, so it
38+
* folds into `debug` (the most verbose driver level).
39+
*/
40+
export function kernelLevelToLogLevel(level: string): LogLevel {
41+
switch (level) {
42+
case 'error':
43+
return LogLevel.error;
44+
case 'warn':
45+
return LogLevel.warn;
46+
case 'info':
47+
return LogLevel.info;
48+
case 'debug':
49+
case 'trace':
50+
return LogLevel.debug;
51+
default:
52+
// Unknown kernel level — surface it rather than drop it; debug is the
53+
// least surprising bucket for an unrecognised severity.
54+
return LogLevel.debug;
55+
}
56+
}
57+
58+
/**
59+
* Map a driver `LogLevel` onto the kernel level string the napi bridge expects.
60+
* The `LogLevel` enum values are already the kernel-compatible lower-case
61+
* strings, so this is the identity at runtime — kept as an explicit function so
62+
* the boundary is named and a future divergence has one place to live.
63+
*/
64+
export function logLevelToKernelLevel(level: LogLevel): string {
65+
return level;
66+
}
67+
68+
/**
69+
* Format one kernel log record into a single driver log line, tagged with its
70+
* origin so kernel lines are distinguishable from driver lines in a shared
71+
* sink/file.
72+
*/
73+
export function formatKernelLine(record: SeaNativeLogRecord): string {
74+
return `[kernel ${record.target}] ${record.message}`;
75+
}
76+
77+
/**
78+
* Install the kernel→driver log bridge: forward kernel `tracing` events into
79+
* `logger` at `level`.
80+
*
81+
* - **Verbosity** is set kernel-side to `level` so events the sink would drop
82+
* never cross the bridge.
83+
* - **Process-global, last-writer-wins:** the napi binding holds a single
84+
* process-global subscriber + sink (a `tracing` global subscriber installs
85+
* once). Each call retargets the sink to `logger`, so in a multi-client
86+
* process the most recently connected client's logger receives kernel logs —
87+
* mirroring the Python connector's `pyo3_log` model. Single-client apps, the
88+
* common case, are unaffected.
89+
* - **Runtime retargeting:** if `logger` exposes `onLevelChange` (as
90+
* `DBSQLLogger` does), the bridge subscribes so a later `logger.setLevel(...)`
91+
* also retargets the kernel-side filter via `setKernelLogLevel` — keeping
92+
* kernel verbosity in lock-step with the driver's at runtime, not just at
93+
* connect. Loggers without it still get the connect-time level.
94+
* - **Graceful on older bindings:** if the loaded `.node` predates
95+
* `initKernelLogging`, this is a no-op (kernel logs simply stay unbridged)
96+
* rather than a hard failure — logging is advisory.
97+
*
98+
* Returns an **unsubscribe** function the caller must invoke on teardown
99+
* (`SeaBackend.close()`) to drop the level-change listener; it is a safe no-op
100+
* when nothing was subscribed.
101+
*/
102+
export function installKernelLogBridge(binding: SeaNativeBinding, logger: IDBSQLLogger, level: LogLevel): () => void {
103+
// Defensive: a stale/older binding without the bridge export must not break
104+
// connect() — logging is non-critical.
105+
if (typeof binding.initKernelLogging !== 'function') {
106+
return () => {};
107+
}
108+
109+
const callback = (err: Error | null, records: Array<SeaNativeLogRecord>): void => {
110+
if (err || !records) {
111+
return;
112+
}
113+
for (const record of records) {
114+
logger.log(kernelLevelToLogLevel(record.level), formatKernelLine(record));
115+
}
116+
};
117+
118+
binding.initKernelLogging(callback, logLevelToKernelLevel(level));
119+
120+
// Keep the kernel's level in lock-step with runtime `logger.setLevel(...)`
121+
// calls. Requires both a logger that notifies (`onLevelChange`) and a binding
122+
// that can retarget (`setKernelLogLevel`); otherwise the connect-time level
123+
// stands and there is nothing to unsubscribe.
124+
if (typeof logger.onLevelChange === 'function' && typeof binding.setKernelLogLevel === 'function') {
125+
return logger.onLevelChange((newLevel) => {
126+
binding.setKernelLogLevel(logLevelToKernelLevel(newLevel));
127+
});
128+
}
129+
130+
return () => {};
131+
}

lib/sea/SeaNativeLoader.ts

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
*
1818
* Mirrors the load-failure-tolerant pattern of `lib/utils/lz4.ts`: the
1919
* `.node` artifact ships via per-platform optional dependencies
20-
* (`@databricks/sql-kernel-<triple>`), so its absence must not crash
20+
* (`@databricks/databricks-sql-kernel-<triple>`), so its absence must not crash
2121
* a Thrift-only consumer of the driver. Callers that actually need
2222
* SEA construct a {@link SeaNativeLoader} (or use the process-global
2323
* {@link getSeaNative}) which throws a structured error if the binding
@@ -39,6 +39,7 @@ import type {
3939
AsyncStatement as NativeAsyncStatement,
4040
AsyncResultHandle as NativeAsyncResultHandle,
4141
CancellableExecution as NativeCancellableExecution,
42+
LogRecord as NativeLogRecord,
4243
} from '../../native/sea';
4344

4445
// SEA-prefixed re-exports. The kernel-generated `.d.ts` keeps the
@@ -78,6 +79,11 @@ export type SeaNativeAsyncResultHandle = NativeAsyncResultHandle;
7879
// returns.
7980
export type SeaNativeCancellableExecution = NativeCancellableExecution;
8081

82+
// One kernel log event forwarded over the napi log bridge (see SeaLogging.ts):
83+
// `{ level, target, message }`. Re-exported so the bridge can name the shape
84+
// without re-declaring it (stays in lock-step with the kernel contract).
85+
export type SeaNativeLogRecord = NativeLogRecord;
86+
8187
/**
8288
* The full native binding surface, derived from the generated module
8389
* so it can never drift from the `.d.ts` contract: when the kernel
@@ -104,7 +110,7 @@ function loadFailureHint(err: NodeJS.ErrnoException): string {
104110
// not the bare `${platform}` shown here, so a literal example would
105111
// 404. Point at the README's supported-triple list instead.
106112
const installHint =
107-
'Install the matching @databricks/sql-kernel-* optional dependency for your platform ' +
113+
'Install the matching @databricks/databricks-sql-kernel-* optional dependency for your platform ' +
108114
'(see native/sea/README.md for the supported triples; M0 ships linux-x64-gnu only).';
109115
if (err.code === 'MODULE_NOT_FOUND') {
110116
return `SEA native binding not installed for platform ${platform} on Node ${process.version}. ${installHint}`;

native/sea/README.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
**The Rust binding source lives in the kernel repo** at
44
`databricks-sql-kernel/napi/`. Building it requires a local checkout
55
of that repo — see "Build for local dev" below. The published npm
6-
package is `@databricks/sql-kernel-<triple>`.
6+
package is `@databricks/databricks-sql-kernel-<triple>`.
77

88
## Workspace topology
99

@@ -33,10 +33,10 @@ and reintroduce the same clash. Standalone-workspace is the fix.
3333
- `index.js` — napi-rs's per-platform router shim. Gitignored;
3434
populated by `npm run build:native` for local dev. In published
3535
tarballs it ships alongside the `.d.ts` and `require()`s the
36-
right `@databricks/sql-kernel-<triple>` optional dependency.
36+
right `@databricks/databricks-sql-kernel-<triple>` optional dependency.
3737
- `index.*.node` — the actual native binary, one per platform.
3838
Gitignored. In production these live in the per-triple optional
39-
dependencies (`@databricks/sql-kernel-linux-x64-gnu`, etc.); for
39+
dependencies (`@databricks/databricks-sql-kernel-linux-x64-gnu`, etc.); for
4040
local dev `npm run build:native` copies one into this directory.
4141

4242
## Build for local dev
@@ -56,7 +56,7 @@ nodejs repo.
5656
## Production load path
5757

5858
At release time the kernel's CI publishes
59-
`@databricks/sql-kernel-<triple>` npm packages — one per supported
59+
`@databricks/databricks-sql-kernel-<triple>` npm packages — one per supported
6060
platform — each containing a single `.node` binary. `native/sea/index.js`
6161
(the napi-rs router) `require()`s the package matching the consumer's
6262
`process.platform` / `process.arch` at load time.
@@ -68,13 +68,13 @@ platform — each containing a single `.node` binary. `native/sea/index.js`
6868
> unpublished package would break every install.) Until they ship, the
6969
> binding is produced locally via `npm run build:native` (which copies
7070
> `index.<triple>.node` into this directory). Once the packages are
71-
> published, add `@databricks/sql-kernel-<triple>` back to
71+
> published, add `@databricks/databricks-sql-kernel-<triple>` back to
7272
> `optionalDependencies` — npm then installs only the matching one.
7373
7474
## Supported platforms (M0)
7575

7676
M0 targets a **single** triple: **`linux-x64-gnu`** (package
77-
`@databricks/sql-kernel-linux-x64-gnu`, once published).
77+
`@databricks/databricks-sql-kernel-linux-x64-gnu`, once published).
7878

7979
On every other platform (macOS, Windows, linux-arm64, linux-x64-musl
8080
/ Alpine, …) the SEA binding is simply absent: `SeaNativeLoader`
@@ -86,9 +86,9 @@ CI starts publishing them in later milestones.
8686

8787
## Supply-chain note
8888

89-
The unpublished triple names (`@databricks/sql-kernel-darwin-arm64`,
89+
The unpublished triple names (`@databricks/databricks-sql-kernel-darwin-arm64`,
9090
`…-win32-x64-msvc`, etc.) referenced by the router are **not**
9191
squat-able: `@databricks` is a Databricks-owned npm scope, and npm
9292
only allows org members to publish under a scope it owns. A third
93-
party therefore cannot register `@databricks/sql-kernel-*` and have
93+
party therefore cannot register `@databricks/databricks-sql-kernel-*` and have
9494
the router autoload it. No placeholder packages are required.

0 commit comments

Comments
 (0)