From 8d17f49290d23dcad391a5fa4f08b8e3477ca5ac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanis=C5=82aw=20Czech?= Date: Mon, 11 May 2026 12:40:45 +0200 Subject: [PATCH 1/6] Docs about napi --- docs/source/internal/SUMMARY.md | 1 + docs/source/internal/napi.md | 161 ++++++++++++++++++++++++++++++++ 2 files changed, 162 insertions(+) create mode 100644 docs/source/internal/napi.md diff --git a/docs/source/internal/SUMMARY.md b/docs/source/internal/SUMMARY.md index 7bc86f067..89f74f4b1 100644 --- a/docs/source/internal/SUMMARY.md +++ b/docs/source/internal/SUMMARY.md @@ -4,3 +4,4 @@ - [ParameterWrapper](./parameter-wrapper.md) - [Query options overview](./query-options.md) - [Logging](./logging.md) +- [Napi](./napi.md) diff --git a/docs/source/internal/napi.md b/docs/source/internal/napi.md new file mode 100644 index 000000000..406c9edc2 --- /dev/null +++ b/docs/source/internal/napi.md @@ -0,0 +1,161 @@ +# Napi (and napi-rs) + +## Distinction + +Node-API and napi-rs are two different things. When encountering just "napi" **in the documentation** (or PRs/issues), +what is meant in most cases is Node-API (although there may be some exceptions, especially in old docs). +However, when you encounter napi **in the code** (mostly Rust code, the napi crate), this refers to napi-rs. + +In napi-rs there are different kinds of interfaces. Some of the interfaces napi-rs exposes are just 1-to-1 mappings from Node-API (mostly `napi::sys`). +Then there are low-cost abstractions over Node-API, like `(To/From)NapiValue` traits and `napi::Env`. +Then there are high-cost interfaces. Some of those add overhead that we would anyway need to introduce (like **sync** functions: functions with the `#[napi]` macro). +Others add significant overhead that we can avoid by using lower-abstraction-level approaches to achieve sometimes very significant performance improvements. +This often comes at the cost of some limitations, however those limitations rarely pose a problem for our use cases. + +## Wrappers + +There may be cases where you have some Rust value you want to work on, but the JS code does not need to have access to the members of such value, +only its methods (think: [client session](https://github.com/adespawn/nodejs/blob/452f2acd2d8794161a1866c4b336df75c038cd22/src/session.rs#L29-L32)). +In those cases you can create a wrapper that keeps the value private to Rust code and exposes methods through napi. + +## Our interfaces + +The main benefit of high-cost napi-rs interfaces is that they provide abstractions that very significantly reduce required verbosity. +This section briefly describes the available internal interfaces and their use cases: + +The napi-rs offers [napi-rs classes](https://napi.rs/docs/concepts/class). They offer a quite easy to understand interface. +Using them is however [very](https://www.scylladb.com/wp-content/uploads/image10-5-768x768.png) [inefficient](https://www.scylladb.com/wp-content/uploads/image2-23-768x768.png), +as we [have](https://github.com/scylladb/nodejs-rs-driver/pull/182) [discovered](https://github.com/scylladb/nodejs-rs-driver/pull/181). + +So why is it so slow? + +Creating an [Object](https://nodejs.org/api/n-api.html#napi_create_object) through Node-API is quite slow, and possibly even slower than doing the same from JS code +(note: this is my recollection of things that happened over a year ago as of writing this documentation). Additionally, when [updating](https://github.com/scylladb/nodejs-rs-driver/pull/291) +to [napi-v3](https://napi.rs/docs/more/v2-v3-migration-guide), our existing use cases stopped working due to +[some problems](https://github.com/scylladb/nodejs-rs-driver/pull/291/commits/7c41208dc106f7ff20e1d0ffb9d33ecedd69e1ce) with BigInt. + +To solve this problem, we have come up with a few workarounds, depending on the specific situation. For objects containing information, we design them +not to have any methods on them. This is necessary since our solutions introduce some limitations at the benefit of better usability / performance in our specific use cases. + +### To Napi Value + +This section describes solutions for converting values from Rust to JS code. + +#### Solutions for performance + +As mentioned above, creation of JS [Objects](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object) is quite expensive. +What we have come up with, when returning multiple values from performance-critical paths, is to return a tuple ([array](https://nodejs.org/api/n-api.html#napi_create_array)) +of values we want to return. + +You can find examples of such use cases in the [Rust part](https://github.com/scylladb/nodejs-rs-driver/blob/2e04a1701e4cfa039bac35c96d8b84b3a5f93867/src/paging.rs#L53-L73) +of the code, as well as the [js parsing](https://github.com/scylladb/nodejs-rs-driver/blob/f4d6a68641bec162f066641b599eb31ec6efc0fa/lib/client.js#L354-L377) +of such values. + +#### Clean use for non critical paths + +Sometimes we want to return objects with [clear structure](https://github.com/adespawn/nodejs/blob/452f2acd2d8794161a1866c4b336df75c038cd22/src/metadata/host.rs#L5-L12), +where performance is not so critical (think: metadata). In this case helpers defined in [to_napi_obj.rs](../../../src/utils/to_napi_obj.rs) come to play. +They allow you to convert: + +- Rust structs and enums with values in variants to JS +- Rust maps (String -> Any) into JS objects + +Details on how to use them are present in the code docs for that file. +Those helpers are a specialized replacement for [napi-rs classes](https://napi.rs/docs/concepts/class). +They provide slightly better performance than napi-rs classes (since they do a bit less stuff on the returned object) and a reasonable user interface. + +### From Napi Value + +Similar to the previous section, we have helpers for converting JS objects to Rust structs. To see more, go to [from_napi_obj.rs](../../../src/utils/from_napi_obj.rs). +This macro provides a much cleaner interface than [napi-rs classes](https://napi.rs/docs/concepts/class). +Compared to napi-rs classes, it allows you to convert arbitrary JS objects with the desired structure, rather than only objects created through the napi-rs layer. + +When using structs from this macro, all fields are converted from JS to Rust. This means that when you have a struct with a high number of fields but use only a few of them, +you will still pay the cost of converting all unused fields. + +### Casync (async bridge) + +This is an open improvement that attempts to cut the very high CPU cost of synchronization in async functions. +You can see the benchmarks for this feature in [#414](https://github.com/scylladb/nodejs-rs-driver/pull/414). +TODO: finish this section once #414 is merged. + +### JSResults + +We use a custom JSResult for results that may end with an error. This replaces napi-rs Results, as those do not allow for custom errors. +We need custom errors since the errors returned by the Rust driver contain additional information. For now we just pass the error name. +We then need to use JSResult over regular Result since we cannot implement the ToNapiValue trait on a regular Result. +This complicates the code slightly but is the solution we found with the lowest verbosity. + +## Typing + +When you pass values between JS and Rust it's nice to have it typed in some way, to improve both readability and correctness of the code. +When using napi-classes or node-built in type representations (integers, bigInt, arrays, `Result`*, strings, buffers), in regular sync function, +Napi-rs will generate type annotations correctly in the `index.d.ts` file upon compilation of the code. + +\* result type is not "visible" across, since napi-rs does not annotate what the function can throw, but is properly handled as a passthrough type. + +However, when using either custom (To/From)NapiValue implementation, one of the helper macros or casync functions, you will need to do a bit more work to properly annotate the types. +Generally there are two things you need to do, in order to properly annotate those uses. What you need to do depends on the case. + +- [`custom-types.d.ts`](../../../src/custom-types.d.ts): This file defines custom types. This means when you define new type, that you accept as argument or return from rust function, + and it is **not** napi-class, you have define its JS(TS) type there. + This file will then tell the JS/TS parser what does the magic type annotated in Rust function ex. `PagingResultWithExecutor` means. + + So in the Rust code you just return `PagingResultWithExecutor`: + + ```rs + // Simple version + #[napi] + pub async fn query_single_page_encoded( + &self, + ... + ) -> Result { + + + // Full version with manual type override (see next point) + #[napi(ts_return_type = "Promise")] + pub async fn query_single_page_encoded( + &self, + ... + ) -> JsResult { + ``` + + Napi-rs then annotates the TS definition of this function as returning `PagingResultWithExecutor`: + + ```ts + export declare class SessionWrapper { + ... + querySinglePage(...): Promise + } + ``` + + The custom definition in the `custom-types.d.ts`: + + ```ts + export type PagingResultWithExecutor = [PagingStateWrapper | null, QueryResultWrapper, QueryExecutor] + ``` + + provides the meaning for this type. This is something you need to add manually to `custom-types.d.ts` file + for every new type you add. This file will then be prepended to `index.d.ts` (used by JS/TS engine when parsing types in VScode) at the rust compilation time. + + Note: this could be possibly automated with `pub trait TypeName` napi-rs trait. +- Manual type overrides: When the napi-rs cannot properly deduce the JS name, you need to manually override the type in the endpoint that utilize it. + By default, when you have a type `T` in Rust, and you were to have rust function return value of that type, it will be annotated in TS as `T`. + Similarly, if you have a generic type: `G` it will be annotated as `G` in TS. + + Here comes the problem: We have a type `JsResult`, which should be opaque (the same way regular `Result` is opaque) - meaning it should be annotated as + `T` in TS (instead of the default `JsResult`). Unfortunately you cannot handle it automatically. + + To properly annotate such functions you need to use napi-rs `ts_return_type` for every function that returns such type. + When using this tag, you need to specify the full type like this: `#[napi(ts_return_type = "Promise")]` + (meaning you need to remember about Promise part for async functions). + + It's annoying, but allows to properly annotate the functions. + You can see me question about this feature on the [napi-rs server](https://discord.com/channels/874290842444111882/874290843262021705/1505966134615085088). + +## The napi problem + +The napi-rs library is not perfect. We have already [found and fixed](https://github.com/napi-rs/napi-rs/issues?q=author%3Aadespawn) multiple bugs in the library. +The library is developed with heavy LLM usage. The quality and readability of the code in some places is less than ideal. +However, this is still the best option we have. To reduce the impact of problems with the napi-rs code, we avoid using high-abstraction parts of napi-rs, +and where possible switch to self-written, specialized abstractions. From 40c0c6212a5fb4e959c42b7f8d2db0f22e070b53 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanis=C5=82aw=20Czech?= Date: Thu, 14 May 2026 12:40:03 +0200 Subject: [PATCH 2/6] More internal documentation --- README.md | 2 +- docs/source/internal/SUMMARY.md | 27 ++++++++++ docs/source/internal/miscellaneous.md | 72 +++++++++++++++++++++++++++ 3 files changed, 100 insertions(+), 1 deletion(-) create mode 100644 docs/source/internal/miscellaneous.md diff --git a/README.md b/README.md index dc1d8bb33..abe8f165c 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# ScyllaDB Node.js-RS Driver +# ScyllaDB Node.js RS Driver ![test workflow](https://github.com/scylladb-zpp-2024-javascript-driver/scylladb-javascript-driver/actions/workflows/integration-tests.yml/badge.svg?branch=main) ![quality workflow](https://github.com/scylladb-zpp-2024-javascript-driver/scylladb-javascript-driver/actions/workflows/check-docs.yml/badge.svg?branch=main) diff --git a/docs/source/internal/SUMMARY.md b/docs/source/internal/SUMMARY.md index 89f74f4b1..0c6ec31c0 100644 --- a/docs/source/internal/SUMMARY.md +++ b/docs/source/internal/SUMMARY.md @@ -1,7 +1,34 @@ # Summary +## Used shortcuts + +Within this document (and internal code documentation) I may use informal names for different products. +Here you can find a list of those names in all their variants, with the versions we decided on in user-facing parts of the driver. + +JS - JavaScript +TS - TypeScript + +Node, NodeJs. Official: [Node.js](https://nodejs.org/) + +Napi, NodeAPI, Napi-rs: see [napi.md](./napi.md#distinction). Official [Node-API](https://nodejs.org/api/n-api.html) and [NAPI-RS](https://napi.rs/). + +DSx: Official: [DataStax](https://www.ibm.com/products/datastax) + +DSx driver, old driver, DataStax driver, cassandra driver. Official: +[cassandra-driver/cassandra-nodejs-driver](https://github.com/apache/cassandra-nodejs-driver) +(The version depends on whether the language of the driver is obvious). +This driver was initially developed by DSx and was transferred to Apache during development of this driver. +For this reason some places may use the name referring to DSx. Any remaining public documentation should refer to this driver as +`cassandra-driver`, with possible annotation like: `formerly known as DataStax Node.js Driver` (see main [README.md](../../../README.md)). + +this driver, new driver, scylladb-driver, scylladb-driver-alpha. Official: ScyllaDB Node.js RS Driver / scylladb-driver-alpha / nodejs-rs-driver +(standalone name / package name / repository name). Confusing? No worries, this repo probably breaks this convention quite a lot. + +## Sections + - [Errors](./error-throwing.md) - [ParameterWrapper](./parameter-wrapper.md) - [Query options overview](./query-options.md) - [Logging](./logging.md) - [Napi](./napi.md) +- [Miscellaneous](./miscellaneous.md) diff --git a/docs/source/internal/miscellaneous.md b/docs/source/internal/miscellaneous.md new file mode 100644 index 000000000..1eeed82cf --- /dev/null +++ b/docs/source/internal/miscellaneous.md @@ -0,0 +1,72 @@ +# Miscellaneous stuff + +## DataStax API + +DSx driver had some external APIs that were related to DSx-specific features: + +- Custom auth providers +- The entire datastax & geometry modules +- Some client / query options + +How did we handle those? + +For functions/methods, we implemented stubs in JS code: + +```js +/** + * @deprecated Not supported by the driver. Usage will throw an error. + */ +class Vertex { + constructor() { + throwNotSupported("Vertex"); + } +} +``` + +See [`structure.js`](../../../lib/datastax/graph/structure.js). The goal of those stubs is to loudly inform +JS users of the driver that our driver does not support those features. This approach, while good for users transitioning +from the old DSx driver, may provide some unnecessary noise for new users. When stabilizing the API in 1.0, those +stubs can be removed, with proper documentation for transitioning users. (If there is such a need, +you could create a temporary transition version for users transferring from the DSx driver, with those stubs enabled). + +For the TS users of the driver, it was enough to remove those endpoints from the TS files. +When someone attempts to use those features, they will get a compilation error, +meaning we achieved the goal of explicitly informing the user of the deprecation of those endpoints. + +When it comes to no longer supported options, we have a very similar approach: +When a JS user provides a no-longer-supported option, +[we throw an error](https://github.com/adespawn/nodejs/blob/ee439f78ed4b4e6c0a73b8aa2f649e45493ae1c5/lib/client-options.js#L971-L978). +Those options are not mentioned in documentation, avoiding the noise for new users. +When it comes to TS users, deleting those options from the TS API is again enough. + +## Personal vendettas + +Well, maybe not exactly personal, but still. This list is composed of small and big annoyances with the current state of the driver. + +### API overloads + +```js + * @example Overloads + * client.eachRow(query, rowCallback); + * client.eachRow(query, params, rowCallback); + * client.eachRow(query, params, options, rowCallback); + * client.eachRow(query, params, rowCallback, callback); + * client.eachRow(query, params, options, rowCallback, callback); +``` + +From this [piece of code](https://github.com/adespawn/nodejs/blob/ee439f78ed4b4e6c0a73b8aa2f649e45493ae1c5/lib/client.js#L418-L423). + +Allowing for overloading API in this way is (while perfectly fine by JS standards) very annoying when it comes to ensuring type safety. +We have to guess based on the value types what overload the user is providing us with. But touching it will break the API. +Is someone using those overloads? Idk. Will it be hard for users to remove usages of those overloads? Probably not. +Feel free to re-visit this when creating 1.0. + +### Long + +The driver was initially developed when BigInt [was not a thing](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt#browser_compatibility). +This means there are internal (and external) parts of the driver that use [Long](https://www.npmjs.com/package/long). +This is the only JS dependency (excluding development dependencies) that the driver uses. +While internal pruning of Long should be (probably annoying but still) possible, removing it from external parts of the driver +may break some APIs. There is an issue related to this: [#41](https://github.com/scylladb/nodejs-rs-driver/issues/41). +This topic would need to be re-visited before 1.0. You would have to decide how to approach this: keep Long as a legacy part of the API, +or fully remove it? From 162894c454eab3068cf98e2d5f9a61bc9bf8dda6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanis=C5=82aw=20Czech?= Date: Thu, 28 May 2026 16:07:20 +0200 Subject: [PATCH 3/6] Release --- docs/source/internal/release-process.md | 29 +++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 docs/source/internal/release-process.md diff --git a/docs/source/internal/release-process.md b/docs/source/internal/release-process.md new file mode 100644 index 000000000..2eb59f9e1 --- /dev/null +++ b/docs/source/internal/release-process.md @@ -0,0 +1,29 @@ +# Release process + +Currently we have support for n=2 platforms. This limitation for specific platforms comes from the need to compile and test a given platform in CI. +We have n+1 npm packages. 1 general package with all of the JS code, and additional packages 1 per each platform: + +- [Main package](https://www.npmjs.com/package/@scylladb/driver) +- [Linux x64 platform package](https://www.npmjs.com/package/@scylladb/driver-linux-x64-gnu) +- [Linux Arm platform package](https://www.npmjs.com/package/@scylladb/driver-linux-arm64-gnu) + +## Adding a new package + +To release a new package, follow npm documentation. Once the package is added, add support for [trusted publishing](https://docs.npmjs.com/trusted-publishers). +DO NOT USE tokens/keys for releasing through CI. You may use those only for the first version release. +What I have done to release a new package is create an empty index.js and package.json: + +```json +{ + "name": "@scylladb/driver-linux-arm64-gnu", + "version": "0.0.0", + "description": "", + "license": "Apache-2.0", + "repository": { + "type": "git", + "url": "https://github.com/scylladb/nodejs-rs-driver" + } +} +``` + +Release it through npm cli (see documentation) to add it to npm registry. After that it becomes visible on the npm side, and you can set it up from there. From 9c7fa603d6c1385cf2b3e6f2786beea5bb7abc42 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanis=C5=82aw=20Czech?= Date: Wed, 3 Jun 2026 15:21:24 +0200 Subject: [PATCH 4/6] Types --- docs/source/internal/SUMMARY.md | 1 + docs/source/internal/on-types.md | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+) create mode 100644 docs/source/internal/on-types.md diff --git a/docs/source/internal/SUMMARY.md b/docs/source/internal/SUMMARY.md index 0c6ec31c0..bf5d8751b 100644 --- a/docs/source/internal/SUMMARY.md +++ b/docs/source/internal/SUMMARY.md @@ -32,3 +32,4 @@ this driver, new driver, scylladb-driver, scylladb-driver-alpha. Official: Scyll - [Logging](./logging.md) - [Napi](./napi.md) - [Miscellaneous](./miscellaneous.md) +- [On types](./on-types.md) diff --git a/docs/source/internal/on-types.md b/docs/source/internal/on-types.md new file mode 100644 index 000000000..ba62c8850 --- /dev/null +++ b/docs/source/internal/on-types.md @@ -0,0 +1,26 @@ +# Some rambling on JS types + +JavaScript allows calls to this library with arbitrary argument values. +In general using this driver API with values of types different than described in the documentation +(JS Docs) can be considered UB. + +Within JavaScript Objects we allow for some duck-typing - this applies to all of the config options and named parameters. +Any object passed there that has required fields is considered valid. +There are some exceptions where we rely on aspects of the objects other than the field being present. + +- Policies: In policies we expect objects with specified constructors. This means you have to use objects as provided through the driver. +- Type guessing & encoding: Parts of type guessing and encoding depend on receiving values of specified types. + +When it comes to built-in types, we may perform some explicit checks to ensure you have provided expected built-in types. + +In general, there is no specific policy on allowing duck-typing: in many places like configuration it's allowed, but there are exceptions, +such as the above-mentioned policies. + +## TS fixes something, but exposes you to other problems + +When you call a TS function from JS code, you can NOT assume that someone will call it with the types as specified in TS. +You can only assume this is what the function should accept and correctly handle. +When calling internally with wrong types, consider it a bug. +When calling it internally, but with wrong types provided by the user, consider it the same as direct calls from the user with different types: undefined behavior. +Either way try to gracefully handle those cases. Clear errors (remember, napi will complain when receiving invalid types: [#251](https://github.com/scylladb/nodejs-rs-driver/issues/251)), +or where possible allow for duck-typing (i.e. don't explicitly disallow it where it makes sense). From 0c3ce30265490a1c6798e1cc2fa91d0253e506f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanis=C5=82aw=20Czech?= Date: Wed, 17 Jun 2026 11:40:03 +0200 Subject: [PATCH 5/6] Update maintenance Include new step about docs, and remove last step replaced by CI step --- MAINTENANCE.md | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/MAINTENANCE.md b/MAINTENANCE.md index 3719b5825..dcc766ad6 100644 --- a/MAINTENANCE.md +++ b/MAINTENANCE.md @@ -92,19 +92,13 @@ Remember to run `npm i` in the `examples/` and `benchmark/` directories once you 2. Bump the package version. Remember to update the version in `package-lock.json` in the main directory, `examples/`, and `benchmark/`. You can do this by running `npm i` in all 3 directories (see [example commit](https://github.com/scylladb/nodejs-rs-driver/pull/363/changes/41250609737052975129c7514439869324478008) on how to do that). -3. Create a new tag. -4. Ensure the extended CI passes. -5. Create release notes on GitHub. The version tag must match version from `package.json` with `v` prefix (for example: `v0.2.0`). +3. Add the new tag to the documentation configuration (see #470) +4. Create a new tag. +5. Ensure the extended CI passes. +6. Create release notes on GitHub. The version tag must match version from `package.json` with `v` prefix (for example: `v0.2.0`). Once you publish release notes, CI action will trigger automatically. This action will build and publish the npm package. -6. Once the CI action finishes, check if it succeeded. If it failed, you will have to fix the underlying issue, and re-run the CI action. -7. Verify that the new release is visible at [npmjs site](https://www.npmjs.com/package/@scylladb/driver). -8. Test the package, by installing it directly from npm. Go to `examples` directory, in `package.json` update the line: -`"@scylladb/driver": "file:./../"` -to: -`"@scylladb/driver": ""`, -then run the following command: -`npm i && node ./runner.js` - +7. Once the CI action finishes, check if it succeeded. If it failed, you will have to fix the underlying issue, and re-run the CI action. +8. Verify that the new release is visible at [npmjs site](https://www.npmjs.com/package/@scylladb/driver). ### Updating packages From 2b42fc9856219536231d69df0edc7d5a524bdaeb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanis=C5=82aw=20Czech?= Date: Fri, 19 Jun 2026 13:32:12 +0200 Subject: [PATCH 6/6] Important note We want to avoid breaking any additional releases, right? --- MAINTENANCE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/MAINTENANCE.md b/MAINTENANCE.md index dcc766ad6..82529fd8b 100644 --- a/MAINTENANCE.md +++ b/MAINTENANCE.md @@ -135,6 +135,7 @@ When converting existing code from JS to TS you can do it in 2 parts: - Ensuring type safety within JS code, - Converting JS to TS. +- Ensure empty `.npmignore` is present in the directory you just introduced the typescript file in. The repository is set up in a way that supports files at all 3 conversion steps (including fully unconverted files).