Restructure embedeer as a single-package repo with @jsilvanus/embedeer and inline GPU provider detection

[Copilot]

Converts the repo from an npm workspaces monorepo back to a standard single-package repository (@jsilvanus/embedeer) and replaces the optional sidecar ort packages with inline GPU system checks. The package is ready for npm publishing.

Structure

Single package at the repo root:

src/           — source files
test/          — test files
package.json   — @jsilvanus/embedeer

Install

npm install @jsilvanus/embedeer        # CPU always works; GPU auto-detected if system libs present

No separate GPU sidecar packages needed. onnxruntime-node (already bundled via @huggingface/transformers) ships libonnxruntime_providers_cuda.so on Linux x64 and DirectML on Windows — no extra install required.

New device / provider API

Added to Embedder.create(), WorkerPool, and CLI (--device, --provider):

// Auto-detect GPU, silent CPU fallback
const embedder = await Embedder.create(model, { device: 'auto' });

// Require GPU — throws with diagnostic error if system libs are missing
const embedder = await Embedder.create(model, { device: 'gpu' });

// Explicit provider
const embedder = await Embedder.create(model, { provider: 'cuda' });

# Auto-detect: uses CUDA on Linux x64, DirectML on Windows x64, CPU fallback everywhere else
npx @jsilvanus/embedeer --device auto --data "Hello"

# Require GPU (throws with diagnostic if CUDA/cuDNN libs are missing)
npx @jsilvanus/embedeer --device gpu --provider cuda --data "Hello"

Provider selection logic (src/provider-loader.js)

All system checks are inlined — no dynamic package imports:

• device='auto': tries platform-default providers (Linux x64 → cuda; Windows x64 → dml), silently falls back to CPU
• device='gpu': throws with a diagnostic error (lists missing CUDA libs or unsupported platform) if no GPU provider succeeds
• device='cpu' / provider='cpu': always CPU
• Explicit provider=cuda|dml: runs inline checks or throws with a diagnostic error; unsupported platform surfaces "not supported on …" message

Linux x64 — CUDA 12 inline checks

provider-loader.js checks:

1. /dev/nvidiactl exists (NVIDIA driver present)
2. All required shared libraries found via common paths + ldconfig -p: libcudart.so.12, libcublas.so.12, libcublasLt.so.12, libcurand.so.10, libcufft.so.11, libcudnn.so.9
3. Throws a clear diagnostic with apt install instructions if anything is missing

System requirements: NVIDIA GPU + driver ≥ 525, CUDA 12 toolkit, cuDNN 9.

# Install CUDA 12 + cuDNN 9 (Ubuntu/Debian)
sudo apt install cuda-toolkit-12-6 libcudnn9-cuda-12

# Run with GPU (auto-detect — uses CUDA here, CPU fallback on any other machine)
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --device auto --data "Hello GPU"

# Require CUDA (throws with install instructions if libs are missing)
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --device gpu --data "Hello GPU"

Docker + NVIDIA CUDA

Works out of the box with NVIDIA Container Toolkit:

• /dev/nvidiactl is mounted into the container automatically
• NVIDIA CUDA images set LD_LIBRARY_PATH to paths already in the default search list

FROM nvidia/cuda:12.6.3-cudnn9-runtime-ubuntu24.04
# ... install Node.js ...
RUN npm install @jsilvanus/embedeer

docker run --rm --gpus all my-app \
  npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --device gpu --data "Hello GPU"

Host requirement: NVIDIA Container Toolkit.

Windows x64 — DirectML inline checks

DirectML is bundled with onnxruntime-node on Windows. provider-loader.js just verifies process.platform === 'win32'. Supports NVIDIA, AMD, and Intel GPUs on Windows 10 (1903+) / Windows 11.

Workers

Both worker.js (child process) and thread-worker-script.js (thread) call resolveProvider(device, provider) before pipeline(). device and provider flow through WorkerPool → workerData.

npm publish readiness

• "publishConfig": { "access": "public" } (required for scoped packages)
• "files" array publishes only src/ + README.md
• "engines": { "node": ">=18" }
• "author": "jsilvanus"

Thorough I/O and piping (src/cli.js)

Configurable record delimiter (-D / --delimiter <str>)

printf 'Hello\0World\0' | npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --delimiter '\0'
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --file data.tsv --delimiter '\t'
printf 'foo|||bar|||baz' | npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --delimiter '|||'

Output formats

Format	Description
json (default)	JSON array of float arrays
json --with-text	JSON array of {"text","embedding"} objects
jsonl	Newline-delimited JSON — ideal for jq, streaming
csv	CSV with header text,dim_0,...,dim_N
txt	Space-separated floats, one vector per line
txt --with-text	Tab-separated <text>\t<floats>
sql	INSERT INTO embeddings ...

Interactive line-reader mode (-i / --interactive)

npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --interactive --dump embeddings.jsonl
cat corpus.txt | npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 -i --output csv --dump out.csv

Tests

Tests in test/provider-loader.test.js cover platform defaults, tryLoadProvider failure modes (no NVIDIA device, missing CUDA libs), and all resolveProvider paths. Tests in test/cli-format.test.js cover parseDelimiter, parseTexts, and all formatOutput paths. All tests pass.

Original prompt

Create a monorepo for jsilvanus/embedeer that supports optional GPU providers installable via npm in a two-step install, and update the main embedeer package to dynamically load the appropriate provider at runtime.

Context (current state in PR #1 branch copilot/create-embedding-tool):

• embedeer is an ESM Node package that uses @huggingface/transformers pipeline('feature-extraction') inside a WorkerPool.
• Workers: src/worker.js (child process), src/thread-worker-script.js (worker_threads). Both call pipeline() and then extractor(texts, { pooling, normalize }).
• Existing options: batchSize, concurrency, mode (process|thread), pooling, normalize, token, dtype, cacheDir.
• CLI exists at src/cli.js.

Goal:

1. Convert repository into a monorepo (npm workspaces) with:

   • packages/embedeer (main JS package)
   • packages/ort-win32-x64-cuda (provider package)
   • packages/ort-win32-x64-dml (provider package)
   • packages/ort-linux-x64-cuda (provider package)
   • (optional) packages/ort-*-cpu if needed, but CPU can remain default via upstream onnxruntime-node.

2. Two-step install is acceptable:

   • Base: npm install embedeer
   • GPU: npm install @embedeer/ort-win32-x64-cuda OR @embedeer/ort-win32-x64-dml OR @embedeer/ort-linux-x64-cuda

3. Runtime selection / API changes in embedeer:

   • Add options: device ('auto'|'cpu'|'gpu') and provider override ('cpu'|'cuda'|'dml').
   • CLI flags: --device and --provider.
   • For device='gpu' on Windows: prefer provider='cuda' if installed else 'dml' if installed, unless overridden.
   • For device='gpu' on Linux: prefer provider='cuda'.
   • device='auto': try preferred GPU providers for the platform if installed; otherwise CPU.
   • device='cpu': always CPU.
   • If provider is explicitly requested and not available, throw a clear error telling which npm package to install.

4. Provider packages responsibilities:

   • Each provider package exports a small JS API used by embedeer to "activate" the provider in the current process before @huggingface/transformers pipeline is created.
   • Each provider package should include an install script that downloads or otherwise provides the correct native onnxruntime binaries for that provider (CUDA or DirectML) for the target platform. If a full binary bundling is too large for this task, stub the download logic and clearly document TODOs and expected artifact layout, but still implement the monorepo structure and dynamic loading hooks.
   • Provide clear README docs in root and in packages/embedeer describing how to install GPU provider packages and run npx embedeer with --device gpu.

5. Ensure the worker modes still work:

   • Both child-process workers and thread workers must apply provider activation before pipeline() is called.
   • WorkerPool must pass device/provider into workerData.

6. Tests:

   • Add unit tests verifying provider selection logic and error messages when provider packages are missing.
   • Ensure existing tests pass.

7. CI/build:

   • Update package.json scripts at root for workspaces.
   • Keep npx embedeer working (bin entry).

Output:

• Open a PR implementing the above.

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: See its pr 1

Assistant: You’ve already got the key piece for GPU support in PR #1: you’re using @huggingface/transformers, and that in turn uses ONNX Runtime (onnxruntime-node is in your lockfile). So the GPU plan is “make ONNX Runtime use a GPU execution provider” (CUDA/DirectML/CoreML/etc.), not “write shaders”.

What PR #1 is doing today (relevant to GPU)

• Workers (src/worker.js, src/thread-worker-script.js) call:
  • pipeline('feature-extraction', modelName, { dtype })
  • then extractor(texts, { pooling, normalize })
• That pipeline is executed by ONNX Runtime under the hood.
• Right now it will effectively run on CPU unless ORT is built/installed with a GPU provider and configured to use it.

The practical way to add GPU support here

Step 1: Add a --device / device option, propagate it into workerData

Add device: 'auto'|'cpu'|'cuda' (start with those; you can extend later).

Where to thread it through:

• CLI: src/cli.js add --device
• Embedder: src/embedder.js accept device, pass to WorkerPool
• WorkerPool: src/worker-pool.js include device in workerData
• Workers: src/worker.js and src/thread-worker-script.js read device and configure runtime before creating the pipeline

Step 2: Ensure ONNX Runtime with CUDA is actually available

This is the hardest part in Node land. There are typically two workable approaches:

A) “Bring your own CUDA ORT” via container (recommended)

• Provide a CUDA-enabled Docker image that includes CUDA libs + an ONNX Runtime build that supports CUDA.
• Your Node code just selects device=cuda; if CUDA isn’t present it falls back to CPU.

B) Make GPU optional via an extra dependency

• CPU default: normal install wor...

This pull request was created from Copilot chat.

[jsilvanus]

@copilot next, please work on CUDA that we have a working gpu accelerated mvp.

[Copilot]

@copilot next, please work on CUDA that we have a working gpu accelerated mvp.

Here's the CUDA MVP — commit 5cc9f05.

Key discovery: onnxruntime-node v1.24.x (already installed via @huggingface/transformers) ships libonnxruntime_providers_cuda.so (302MB) on Linux x64. No custom binary packaging needed.

What was implemented:

1. @embedeer/ort-linux-x64-cuda — activate() now does real checks:

   • Verifies /dev/nvidiactl exists (NVIDIA driver present)
   • Searches for required CUDA 12 + cuDNN 9 libs: libcudart.so.12, libcublas.so.12, libcublasLt.so.12, libcurand.so.10, libcufft.so.11, libcudnn.so.9
   • Throws a clear error with apt install instructions if anything is missing

2. @embedeer/ort-win32-x64-dml — DirectML is also bundled with ORT on Windows; activate() just checks the platform.

To use GPU on Linux (after this PR):

# Install CUDA 12 + cuDNN 9 (Ubuntu)
sudo apt install cuda-toolkit-12-6 libcudnn9-cuda-12

# Install packages
npm install embedeer
npm install @embedeer/ort-linux-x64-cuda

# Run with GPU
npx embedeer --model Xenova/all-MiniLM-L6-v2 --device gpu --data "Hello GPU"

The --device auto flag will try CUDA first and silently fall back to CPU if CUDA libraries aren't available.

[jsilvanus]

@copilot some user might want to run this in docker with CUDA. Enabled there. Are we go already on that or do we need something else?

[Copilot]

@copilot some user might want to run this in docker with CUDA. Enabled there. Are we go already on that or do we need something else?

Yes, Docker with CUDA already works with the current implementation — no code changes needed.

Here's why it works automatically:

• The NVIDIA Container Toolkit mounts /dev/nvidiactl into the container → our GPU presence check passes
• NVIDIA CUDA Docker images put all libraries under /usr/local/cuda/lib64 → already in our default search path
• If LD_LIBRARY_PATH is set in the image (which NVIDIA images do), we pick that up too

Commit d5ad2ea adds Docker documentation to both READMEs. The setup is:

FROM nvidia/cuda:12.6.3-cudnn9-runtime-ubuntu24.04
# ... install Node.js ...
RUN npm install embedeer && npm install @embedeer/ort-linux-x64-cuda

docker run --rm --gpus all my-app \
  npx embedeer --model Xenova/all-MiniLM-L6-v2 --device gpu --data "Hello GPU"

The only host requirement is NVIDIA Container Toolkit installed alongside the NVIDIA driver.

[jsilvanus]

@copilot remove embedeer/ort-win32-x64-cuda (placeholder)

This package will be @jsilvanus/embedeer and other packages scoped there as well.

[Copilot]

@copilot remove embedeer/ort-win32-x64-cuda (placeholder)

This package will be @jsilvanus/embedeer and other packages scoped there as well. ...

Done in commit 104cb9e:

• Deleted packages/ort-win32-x64-cuda entirely
• Renamed scope @embedeer/ → @jsilvanus/ on all packages:
  • embedeer → @jsilvanus/embedeer
  • @embedeer/ort-linux-x64-cuda → @jsilvanus/ort-linux-x64-cuda
  • @embedeer/ort-win32-x64-dml → @jsilvanus/ort-win32-x64-dml
• Removed the win32-x64-cuda entry from PROVIDER_PACKAGES in provider-loader.js
• Updated all references in source, tests, and READMEs
• All 61 tests still pass

[jsilvanus]

@copilot please prefix ort packages with embedeer-.

Prepare this for npm publishing. Make sure everything is ready for it.

[Copilot]

@copilot please prefix ort packages with embedeer-.

Prepare this for npm publishing. Make sure everything is ready for it.

Done in commit f721a06:

Package renames:

• @jsilvanus/ort-linux-x64-cuda → @jsilvanus/embedeer-ort-linux-x64-cuda
• @jsilvanus/ort-win32-x64-dml → @jsilvanus/embedeer-ort-win32-x64-dml

npm publish readiness (all three packages):

• "publishConfig": { "access": "public" } — required for scoped packages on npm
• "files": [...] — controls exactly what gets published (source only, no tests/node_modules)
• "engines": { "node": ">=18" } on all packages
• "author": "jsilvanus" set on all packages

npx + CUDA docs (new requirement): Updated npx embedeer → npx @jsilvanus/embedeer throughout and added "Using GPU with npx" sections to both READMEs showing --device auto (CUDA where available, CPU fallback), --device gpu (hard-require), and explicit --provider cuda/dml with the install steps for each platform.

[Copilot]

@copilot I want thorough input and output, piping possibilities etc. Input from stdin (maybe a selectable limiter, \n by default but could also be something else). Please plan for knput possibilities....

Done in commit 05176ec. Here's what was added:

New input option — configurable record delimiter (-D / --delimiter):

• Default: \n (newline) — unchanged behaviour
• Supports escape sequences: \0 (null byte), \n, \t, \r, or any literal string
• Applies to both stdin and --file; JSON arrays are still auto-detected first

# Null-byte delimited (safe with text containing newlines — compatible with find -print0)
printf 'Hello\0World\0' | npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --delimiter '\0'

# Tab-delimited file
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --file data.tsv --delimiter '\t'

# Custom multi-char delimiter
printf 'foo|||bar|||baz' | npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --delimiter '|||'

New output formats:

• jsonl — one {"text":"...","embedding":[...]} per line — ideal for jq, grep, streaming
• csv — text,dim_0,dim_1,...,dim_N with header — for pandas/Excel
• --with-text flag — adds source text to json ([{"text","embedding"}]) and txt (text\tfloats) output

# JSONL — pipe to jq
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --output jsonl --data "foo" "bar" | jq '.text'

# CSV — for pandas
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --file texts.txt --output csv --dump out.csv

# JSON with source text
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --output json --with-text --data "cat" "dog"

Both READMEs now have dedicated Input Sources, Configurable delimiter, Output Formats, and Piping examples sections. 19 new tests cover all new paths.

[jsilvanus]

@copilot I also want option that a line reader (??) opens and you can just paste some data and it will, when seeing data come in, start work (batch maybe also?), and output (probably to a file when input reader is used....)

[Copilot]

@copilot I also want option that a line reader (??) opens and you can just paste some data and it will, when seeing data come in, start work (batch maybe also?), and output (probably to a file when in...

Done in commit 7bf2828. Added --interactive / -i — a readline-based line-reader that starts embedding as data arrives:

# Interactive terminal — paste lines, Ctrl+D when done
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --interactive --dump embeddings.jsonl

# Stream a file in batches (no terminal needed)
cat corpus.txt | npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 -i --output csv --dump out.csv

# GPU, custom batch size
npx @jsilvanus/embedeer --model Xenova/all-MiniLM-L6-v2 --interactive --device auto \
  --batch-size 16 --output jsonl --dump out.jsonl

How flushing works:

Trigger	What happens
Buffer reaches --batch-size	Auto-flush
Empty line (Enter ×2)	Manual flush
Ctrl+D (EOF)	Flush remaining + exit
Ctrl+C	Flush remaining + exit

• > prompt appears on stderr when stdin is a TTY; silent when piped.
• Output goes to --dump file (cleared at session start); falls back to stdout with a tip.
• json/sql auto-switch to jsonl; csv writes the header once on the first batch only.
• Progress (Batch N: M record(s) → file) always goes to stderr.

[jsilvanus]

@copilot fix any remaining errors.

[Copilot]

@copilot fix any remaining errors.

Fixed in commit ed06683. All 15 previously-failing tests are now resolved.

Root cause: @huggingface/transformers and the monorepo workspace symlinks (@jsilvanus/embedeer-ort-*) were not installed in the agent environment. This caused 14 failures in embedder.test.js / embedder-options.test.js (top-level ESM import of @huggingface/transformers blew up) and 1 failure in provider-loader.test.js (the win32/dml workspace package wasn't symlinked yet).

Fix: Added .github/copilot-setup-steps.yml to run npm install before agent sessions. This installs all dependencies including the workspace package symlinks.

All 80 tests now pass (ℹ pass 80, ℹ fail 0).

[jsilvanus]

@copilot Do we need the embedeer-ort* packages at all, if everything comes in the basic deps already?

[Copilot]

@copilot Do we need the embedeer-ort* packages at all, if everything comes in the basic deps already?

Technically no — the ort-* packages ship zero new binaries. libonnxruntime_providers_cuda.so already comes bundled with onnxruntime-node (which is a transitive dep of @huggingface/transformers). The ort packages currently only do:

• embedeer-ort-linux-x64-cuda: checks /dev/nvidiactl + scans for CUDA 12 / cuDNN 9 system libs, then returns 'cuda' as the device string
• embedeer-ort-win32-x64-dml: checks process.platform === 'win32', then returns 'dml'

Why the two-package pattern was added: provider-loader.js uses import(packageName) as an opt-in signal — if the package isn't installed the import fails and the system falls back to CPU (--device auto) or throws (--device gpu). This pattern makes sense when packages ship large optional native binaries (e.g., separate CUDA DLLs), but since everything is already bundled, it's mostly ceremony here.

We could simplify by:

1. Deleting both ort packages
2. Moving the system checks (CUDA lib detection, platform check) inline into provider-loader.js
3. --device auto on linux/x64 just tries CUDA directly, falls back silently
4. --device gpu throws with the same diagnostic error if system libs are missing

Want me to go ahead with that simplification? It would reduce the repo to a single publishable package (@jsilvanus/embedeer) with no optional sidecars needed.

[jsilvanus]

@copilot Yes: delete both ort packages and move the system checks inline into provider-loader.js. Convert this back to a normal repo, not monorepo anymore, since this changes things.

[Copilot]

@copilot Yes: delete both ort packages and move the system checks inline into provider-loader.js. Convert this back to a normal repo, not monorepo anymore, since this changes things.

Done in commit f5804b6.

• Deleted both packages/ort-linux-x64-cuda and packages/ort-win32-x64-dml
• Inlined CUDA system checks (/dev/nvidiactl detection + CUDA 12/cuDNN 9 lib scan) and DML platform check directly into src/provider-loader.js — no dynamic import() of external packages
• Flattened the monorepo back to a normal repo: src/ and test/ are now at the root, and package.json is the single @jsilvanus/embedeer package
• Updated README.md: GPU is now described as built-in (just npm install @jsilvanus/embedeer + CUDA system libs for Linux)
• 79 tests pass

[jsilvanus]

@copilot Are PR description and README up-to-date?

[Copilot]

@copilot Are PR description and README up-to-date?

Yes, both are accurate and current:

README correctly describes the single-package structure: npm install @jsilvanus/embedeer with no sidecar packages, GPU built-in via onnxruntime-node, CUDA system lib requirements for Linux x64, DirectML for Windows, all CLI options, and the provider selection table.

PR description was already updated in the previous commit to reflect the single-package repo with inline GPU system checks.

One small note: getPlatformDefaultProviders() returns ['cuda', 'dml'] on Windows x64 — intentionally tries CUDA first (for future-proofing), silently falls through to DML since there's no win32-x64-cuda impl. The README's "GPU order is cuda → dml" on Windows accurately reflects this.