[Bug]: PyTorch loads unconditionally regardless of embedding provider — prevents memory savings when using external embeddings

[zip-it-e-dee-doo-dah]

Bug Description

When HINDSIGHT_API_EMBEDDINGS_PROVIDER is set to an external provider (e.g., openai), the hindsight-api Python process still loads PyTorch and local model files at startup, resulting in ~1.15 GB RSS baseline with no local model configured — only ~950 MB less than a BGE-local instance.

Steps to Reproduce

• Configure hindsight-api with HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai and a valid HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY

• Start the daemon (no local embedding model needed or configured)

• Check mapped files: grep -c "torch|bge" /proc//maps

• Observe PyTorch and BGE file mappings present despite no local model being used

• Hindsight version: v0.6.5

• Operating system: Ubuntu 24.04.4 LTS, Linux 6.8.0-111-generic x86_64

• Install method: uvx hindsight-api@latest via uv, running as background daemon process

• Model: N/A (embedding-provider configuration issue, not LLM)

• Provider / routing chain: HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai, HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=text-embedding-3-small. No local sentence-transformers model configured.

Additional context
Discovered while migrating from local BGE-small embeddings to OpenAI external embeddings to reduce CPU load and RSS. The migration successfully eliminates embedding inference CPU saturation (the primary goal), but the memory savings were significantly smaller than expected due to PyTorch loading unconditionally.

Likely cause: import torch or from sentence_transformers import ... at module top-level in embeddings.py or a dependency, rather than inside the LocalEmbeddings class initializer. A lazy import pattern (importing only when EMBEDDINGS_PROVIDER=local) would fix this.

Expected Behavior

When an external embedding provider is configured, PyTorch and local model weights should not be imported or loaded. RSS baseline should reflect only the API server, database client, and LLM client — expected ~200–400 MB, not ~1.15 GB.

Actual Behavior

64 PyTorch/BGE file mappings present in /proc//maps on a daemon configured exclusively for OpenAI embeddings. RSS sits at ~1,152 MB at idle — consistent with PyTorch runtime being fully loaded despite serving zero inference requests.

'''
On daemon with HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
grep -c "torch|bge" /proc/249656/maps
Output: 64
'''

'''
RSS
grep VmRSS /proc/249656/status
VmRSS: 1178492 kB (~1.15 GB)
'''

For comparison, a BGE-local instance on the same host:
'''
grep VmRSS /proc/142163/status
VmRSS: 2166572 kB (~2.1 GB)
'''

Switching to external embeddings saves ~950 MB — but the expected saving was ~1.5–1.8 GB (PyTorch runtime + model weights). PyTorch appears to be imported unconditionally at module load time rather than lazily when a local provider is actually selected.

Version

Hindsight version: v0.6.5, Operating system: Ubuntu 24.04.4 LTS, Linux 6.8.0-111-generic x86_64

LLM Provider

OpenAI

[gtoxlili]

Lazy imports are already in place — torch / sentence_transformers / transformers are all imported inside class methods, nothing at module top level in either embeddings.py or cross_encoder.py.

The RSS is from the reranker, not embeddings. It defaults to `local` even when `EMBEDDINGS_PROVIDER=openai`, so the cross-encoder still loads torch + transformers. Visible in startup:

Reranker: initializing local provider with model cross-encoder/ms-marco-MiniLM-L-6-v2

Setting `HINDSIGHT_API_RERANKER_PROVIDER` to a remote option (cohere / siliconflow / tei / openrouter / litellm / ...) drops torch entirely.

[zip-it-e-dee-doo-dah]

1. Updating this thread with refined diagnosis after applying your suggested fix and continuing to investigate. Configuration is working correctly. HINDSIGHT_API_RERANKER_PROVIDER=zeroentropy is present in the running daemon's environment (confirmed via /proc/<pid>/environ), and the ZeroEntropy API key and model (zerank-2) are correctly injected via systemd EnvironmentFile. The env var path is working end-to-end. Residual issue: torch C libraries are still mapped at startup. After setting the remote provider, the running Hindsight Python worker process still has the following mapped: libtorch.so, libtorch_cpu.so, libtorch_cuda.so, libc10.so, libc10_cuda.so, libtorch_python.so, torch/_C.cpython-312.so (11 unique torch library files total). Daemon RSS stays at ~1.2 GB (vs. our pre-migration baseline of ~2.1 GB with the local reranker fully loaded), so the remote-provider config is meaningfully reducing footprint, but not eliminating torch. Diagnostic work (against hindsight-api==0.6.1, current @latest at time of writing): We searched for top-level import torch / from torch / from transformers in the Hindsight source — none found. The lazy imports inside class methods are in place as you described. However, hindsight_api/__init__.py (line 8) and hindsight_api/engine/__init__.py (line 10) both export LocalSTCrossEncoder and RemoteTEICrossEncoder as package-level imports from cross_encoder. This forces the cross_encoder module to load at startup regardless of which provider is configured, which transitively pulls in the torch C extension chain before any provider check runs. The local model weights themselves are not loaded — no .bin or .safetensors files appear in the process map. So the runtime cost is bounded to library mappings, not a full model load. But those libraries wouldn't be in RSS at all if cross_encoder didn't load on startup. 2. One direction that would address this: guarding the package-level exports in __init__.py behind a provider check, or deferring them to lazy resolution. Happy to test a patch if that's useful.

…

On Tue, May 12, 2026 at 2:37 AM Gtio ***@***.***> wrote: *gtoxlili* left a comment (vectorize-io/hindsight#1587) <#1587 (comment)> Lazy imports are already in place — torch / sentence_transformers / transformers are all imported inside class methods, nothing at module top level in either embeddings.py or cross_encoder.py. The RSS is from the reranker, not embeddings. It defaults to `local` even when `EMBEDDINGS_PROVIDER=openai`, so the cross-encoder still loads torch + transformers. Visible in startup: Reranker: initializing local provider with model cross-encoder/ms-marco-MiniLM-L-6-v2 Setting `HINDSIGHT_API_RERANKER_PROVIDER` to a remote option (cohere / siliconflow / tei / openrouter / litellm / ...) drops torch entirely. — Reply to this email directly, view it on GitHub <#1587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CDDPM7MH55SLMM2DBER3TBT42LBEHAVCNFSM6AAAAACYZ7BBQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIMRXHE2DKMJTHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>