[model-config] feat: add environment variables support for backends#10721
[model-config] feat: add environment variables support for backends#10721Nold360 wants to merge 2 commits into
Conversation
…rations - Add field to model configuration to pass environment variables to backend processes - Update backend options and model configuration handling - Add documentation for environment variables configuration with examples including CUDA_VISIBLE_DEVICES Assisted-by: qwen-agentworld-35b-a3b Signed-off-by: nold <nold42@pm.me>
| clientAddr := fmt.Sprintf("127.0.0.1:%d", port) | ||
|
|
||
| proc, err := s.ml.StartProcess(backendPath, backend, bindAddr) | ||
| proc, err := s.ml.StartProcess(backendPath, backend, bindAddr, nil) |
There was a problem hiding this comment.
seems we don't pass the env vars here in distributed mode?
There was a problem hiding this comment.
ah, i didn't really know about distributed.. seems like we need to pass it through installbackend, too for that. i'll look into it.
There was a problem hiding this comment.
seems like we need to pass the env variables through InstallBackend for that, too.
if you are fine with more params on those, i'll try to get it done
There was a problem hiding this comment.
Pull request overview
Adds support for specifying per-model environment variables in model.yaml, threading them through model config → backend options → backend process spawn so operators can control backend runtime behavior (e.g., CUDA_VISIBLE_DEVICES).
Changes:
- Add
envto model configuration schema and UI metadata registry. - Plumb env vars into model loader options and backend process spawning.
- Extend gRPC
ModelOptionsproto and document the new YAML field.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/model/process.go | Extends backend process spawn to accept model-specific env vars. |
| pkg/model/loader_options.go | Adds loader option storage + setter for env vars. |
| pkg/model/initializers.go | Passes env vars through when spawning the gRPC backend. |
| core/services/worker/supervisor.go | Updates StartProcess callsite for new signature. |
| core/config/model_config.go | Adds env field to the model config schema. |
| core/config/model_config_test.go | Adds a test section intended to cover env var config. |
| core/config/meta/registry.go | Registers env for UI/config metadata (map editor). |
| core/backend/options.go | Threads config env vars into model options and gRPC options. |
| backend/backend.proto | Adds EnvVars to ModelOptions proto. |
| docs/content/advanced/model-configuration.md | Documents env: usage in model configuration YAML. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| env = append(env, vulkanICDEnv(workDir)...) | ||
|
|
||
| // Add model-specific environment variables | ||
| if envVars != nil { | ||
| for key, value := range envVars { |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: nold <Nold360@users.noreply.github.com>
Description
Simple change to be able to specify env variables for backend processes in model config. This way the user is able to further customizes processes. Like in my case i want to be able to set
CUDA_VISIBLE_DEVICESto control which model gets loaded onto which of my GPUs. it also slightly reduces VRAM usage on unused GPUs when using tensor_split.Assisted-by: qwen-agentworld-35b-a3b
Notes for Reviewers
Signed commits