[feat] [doc improvement]: Where does context chat backend get its compute?

[tareko]

Describe the feature you'd like to request
I finally got context chat working, and it has tremendous potential. I am running our server on a digitalocean droplet, and do our computing on a separate ollama server.

I looked at the configuration file /nc_app_context_chat_backend_data/config.yaml, but I don't have the skills to understand where the computing is happening. If I had to guess, I would guess it's all happening locally. For sure any request takes a very long time (up to 60 minutes) and doesn't seem to register on the GPU server. I can't tell, and the documentation doesn't help too much.

Describe the solution you'd like

1. Please take a moment to explain to me where models are being computed
2. If it is as I suspect and everything happens local, then a small section in the documentation that explains this would be useful - I'm happy to write this and submit a PR
3. Is it possible to use other computing resources? If so, how?
4. If yes to above, then this should also be documented - again happy to write this and submit a PR

Describe alternatives you've considered

[tareko]

I forked the repo and asked OpenAI Codex the following question:

Can you explain to me where this program loads models? Does it load them on the local server / CPU? Or does it draw from another configuration to load them, even if it's a remote server?

I got the following answer:

How models are loaded

At startup the service prepares its working directories (including MODEL_DIR) and picks a configuration file (CPU or GPU) that lives in persistent storage.

The configuration is parsed by get_config, which keeps only the first supported LLM entry it encounters (e.g., nc_texttotext, llama, ctransformer, or hugging_face).
When a request needs an LLM, LLMModelLoader.load() consults that configuration, caches the instantiated model in the FastAPI app, and reuses it until it is offloaded.

Local options

The llama and ctransformer adapters both look up MODEL_DIR (defaulting to persistent_storage/model_files) and open a model file on disk, so they execute locally on the server’s CPU/GPU depending on the runtime parameters you supply (e.g., n_gpu_layers).

The Hugging Face adapter does the same when you point it at a model_path; otherwise it will load by model ID through the Hugging Face pipeline, which downloads/caches the model but still runs it in-process on this backend.

A helper map (models_to_fetch) ensures required local artifacts (GGUF files, tokenizer) are downloaded into persistent storage so those loaders can find them.

Remote option

If the chosen LLM is nc_texttotext, the backend doesn’t host a model at all; instead, the CustomLLM wrapper schedules and polls Nextcloud TaskProcessing jobs via HTTP to whichever remote Text-to-Text provider is installed on the linked Nextcloud instance. Results are streamed back once the remote task completes.
Likewise, document embeddings are obtained by sending HTTP requests to an embedding server defined in the config (protocol, host, port), so embeddings can be served from a remote process even when the LLM itself runs locally.
In short: the backend decides at runtime, based on the configuration in persistent storage, whether to load a local model file into its own process or delegate to a remote provider. Both patterns are supported out of the box.

As such, it seems that it is possible to make this remote if nc_texttotext is activated.

[tareko]

Excellent discussion at issue #175 . It's probably worth writing a little section for the documents and manual to explain this. I'll take a look and try to do this over the next 1-2 weeks.

[kyteinsky]

Nice that you followed up that discussion. The documentation update would be much appreciated, especially coming from a user's perspective.

Likewise, document embeddings are obtained by sending HTTP requests to an embedding server defined in the config (protocol, host, port), so embeddings can be served from a remote process even when the LLM itself runs locally.

A little clarification here, the requests are sent to a local process running the embedding server inside the docker container. So this runs on the same server as context_chat_backend and uses the compute device set in the deploy daemon.
It's planned to externalise this at some point so it can contact any external service for the embeddings.