Skip to content

Commit 6584db9

Browse files
localai-botmudler
localai-bot
and
mudler
authored
fix(nodes): never schedule a model onto a node that cannot store it (#11054)
* fix(nodes): never schedule a model onto a node that cannot store it A worker whose models filesystem was 100% full kept advertising `status: healthy`, stayed a scheduling candidate, was picked to host a 70 GB video model, accepted the staging request, transferred ~17 GB and only then failed: staging .../whisper-large-v3/model.fp32-00001-of-00002.safetensors: upload to node b7bacbf4-... failed with status 500: writing file: /models/longcat-video-avatar-1.5/...: no space left on device The node was at 937G/937G/0-avail. Total elapsed before the truth surfaced: 16 minutes, for a decision that could never have succeeded. The worker health signal only ever proved liveness. `/readyz` (WorkerReadiness/NATSReadiness) checks the NATS link; `status: healthy` in the registry is driven by heartbeat recency. Node capacity carried VRAM and RAM but no disk figure at all, and the router compared model size against VRAM only — nothing anywhere looked at free space on the filesystem that staging actually writes to. Report it, then use it: - Workers now measure the filesystem backing their MODELS directory (not `/` -- staged weights land in the models path, and that mount is very often separate) and report `total_disk`/`available_disk` on registration and on every heartbeat. Free disk moves faster than VRAM under staging traffic, so the per-heartbeat refresh matters. - The SmartRouter drops nodes that cannot store the model before it picks one. The requirement comes from `modelPayloadBytes` -- the same local paths `stageModelFiles` uploads, already computed for the size-derived load budget -- plus a 5% / 1 GiB margin, rather than a fixed percentage of the node's disk. A percentage threshold would take a small-but-usable node out of rotation for models it could hold, and on a homogeneous cluster would strand every node at once. - When no node fits, scheduling fails immediately with an error naming the requirement and each node's free space, instead of picking one and discovering it mid-transfer. Two deliberate non-changes. Low disk does not mark a node `unhealthy`: the check is per model, so a node too small for one model stays a valid target for smaller ones. And `total_disk == 0` means "does not report disk" (pre-upgrade worker, or a failed stat), not "full" -- such nodes pass through untouched so a rolling upgrade never empties the candidate pool. A genuinely full node is distinguishable: non-zero total, zero available. Registry read failures are logged and scheduling continues unfiltered; a database hiccup must not wedge a cluster. Free space is surfaced on the node detail page next to VRAM, since the incident's signature was a node that looked entirely healthy. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] * feat(nodes): make the disk-headroom check operator-controllable The admission check added in the previous commit had no off switch. A scheduler-side veto with no escape hatch is a liability: our size estimate can be wrong (deduplicating or compressing filesystems, a backend that fetches its own weights rather than loading the staged copy), and an operator who hits that has no way out but a downgrade. Add one knob with two surfaces that share a single source of truth: - `--distributed-disk-headroom-check` / `LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK` (default true), following the `--distributed-prefix-cache` pattern for a default-on distributed feature. - `distributed_disk_headroom_check` in the runtime-settings registry, so it can be flipped without a restart from `POST /api/settings` and from Settings -> Distributed in the WebUI. Both write `DistributedConfig.DiskHeadroomDisabled`, and the SmartRouter reads that member LIVE on every scheduling decision through a closure over the application config rather than a value snapshotted at construction. Env/CLI sets the boot value, the runtime setting overrides it live, last write wins, and there is exactly one member to read. Snapshotting would have made the runtime toggle a no-op until restart. Disabled means WARN, not SKIP. Selection goes back to ignoring free disk -- byte for byte the pre-check behaviour -- but the check still runs, and when it would have rejected every node it says so, naming the knob that suppressed it. Going quiet when switched off would reproduce the exact condition that made the original incident expensive: a cluster doing something that could not work and saying nothing. Disabling is also logged once at startup. Warning only on the total-rejection case keeps it actionable rather than chatty on a heterogeneous cluster. Also fixes a false positive in the check itself: shared-models mode (LOCALAI_DISTRIBUTED_SHARED_MODELS) stages nothing at all -- every node already mounts this models directory at this path -- so demanding the full checkpoint size of free space per node would have rejected a cluster that needs no new bytes. The check is skipped there entirely. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
1 parent f317da7 commit 6584db9

28 files changed

Lines changed: 901 additions & 54 deletions

‎core/application/distributed.go‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -356,6 +356,12 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
356356
PrefixConfig: prefixCfg,
357357
Pressure: pressure,
358358
SharedModels: cfg.Distributed.SharedModels,
359+
// A closure over the live ApplicationConfig, NOT a snapshot: the
360+
// runtime setting (distributed_disk_headroom_check) mutates this exact
361+
// member, so a snapshot here would make the toggle a no-op until
362+
// restart. env/CLI sets the boot value, POST /api/settings overrides it
363+
// live, and this is the single member both write.
364+
DiskHeadroomEnabled: func() bool { return !cfg.Distributed.DiskHeadroomDisabled },
359365
// RAW, not OrDefault: zero means "derive the budget per model from the
360366
// checkpoint size" (config.ModelLoadTimeoutForSize), which is what makes
361367
// a 70 GB video checkpoint work without the operator first hitting a
@@ -377,6 +383,13 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
377383
// replica, not just the one performing the transfer. Without this, a
378384
// /api/operations poll that round-robins onto a peer sees no staging row and
379385
// the progress flickers. The origin publishes; peers mirror via the wildcard.
386+
// A silently disabled safety check is how the original incident stayed
387+
// invisible for sixteen minutes. Say so once, loudly, at startup.
388+
if cfg.Distributed.DiskHeadroomDisabled {
389+
xlog.Info("Disk-headroom admission check is DISABLED: node selection will ignore whether a worker can store the model, and staging may fail with ENOSPC partway through a transfer",
390+
"knob", config.FlagDiskHeadroomCheck, "env", "LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK")
391+
}
392+
380393
router.StagingTracker().SetPublisher(natsClient)
381394
if _, err := router.StagingTracker().SubscribeBroadcasts(natsClient); err != nil {
382395
xlog.Warn("Failed to subscribe to staging progress broadcasts", "error", err)

‎core/cli/run.go‎

Lines changed: 33 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -158,35 +158,36 @@ type RunCMD struct {
158158
DefaultAPIKeyExpiry string `env:"LOCALAI_DEFAULT_API_KEY_EXPIRY" help:"Default expiry for API keys (e.g. 90d, 1y; empty = no expiry)" group:"auth"`
159159

160160
// Distributed / Horizontal Scaling
161-
Distributed bool `env:"LOCALAI_DISTRIBUTED" default:"false" help:"Enable distributed mode (requires PostgreSQL + NATS)" group:"distributed"`
162-
InstanceID string `env:"LOCALAI_INSTANCE_ID" help:"Unique instance ID for distributed mode (auto-generated UUID if empty)" group:"distributed"`
163-
NatsURL string `env:"LOCALAI_NATS_URL" help:"NATS server URL (e.g., nats://localhost:4222)" group:"distributed"`
164-
StorageURL string `env:"LOCALAI_STORAGE_URL" help:"S3-compatible storage endpoint URL (e.g., http://minio:9000)" group:"distributed"`
165-
StorageBucket string `env:"LOCALAI_STORAGE_BUCKET" default:"localai" help:"S3 bucket name for object storage" group:"distributed"`
166-
StorageRegion string `env:"LOCALAI_STORAGE_REGION" default:"us-east-1" help:"S3 region" group:"distributed"`
167-
StorageAccessKey string `env:"LOCALAI_STORAGE_ACCESS_KEY" help:"S3 access key ID" group:"distributed"`
168-
StorageSecretKey string `env:"LOCALAI_STORAGE_SECRET_KEY" help:"S3 secret access key" group:"distributed"`
169-
RegistrationToken string `env:"LOCALAI_REGISTRATION_TOKEN" help:"Token that backend nodes must provide to register (empty = no auth required)" group:"distributed"`
170-
RegistrationRequireAuth bool `env:"LOCALAI_REGISTRATION_REQUIRE_AUTH" default:"false" help:"Fail startup when distributed mode is enabled but LOCALAI_REGISTRATION_TOKEN is empty (node endpoints and worker file-transfer server would otherwise be unauthenticated)" group:"distributed"`
171-
DistributedRequireAuth bool `env:"LOCALAI_DISTRIBUTED_REQUIRE_AUTH" default:"false" help:"Umbrella switch: require BOTH NATS JWT credentials and a registration token when distributed mode is enabled (implies --nats-require-auth and --registration-require-auth)" group:"distributed"`
172-
AutoApproveNodes bool `env:"LOCALAI_AUTO_APPROVE_NODES" default:"false" help:"Auto-approve new worker nodes (skip admin approval)" group:"distributed"`
173-
DistributedSharedModels bool `env:"LOCALAI_DISTRIBUTED_SHARED_MODELS" default:"false" help:"Assert that every node mounts the SAME models directory at the SAME path (shared volume). When true, the router skips staging model files to workers and loads them directly from the shared path, avoiding re-downloads." group:"distributed"`
174-
DistributedPrefixCache bool `env:"LOCALAI_DISTRIBUTED_PREFIX_CACHE" default:"true" help:"Enable prefix-cache-aware routing in distributed mode (default true). When false, routing falls back to round-robin." group:"distributed"`
175-
DistributedPrefixCacheTTL string `env:"LOCALAI_DISTRIBUTED_PREFIX_CACHE_TTL" help:"Idle-timeout for prefix-cache index entries; also drives the background eviction cadence (every TTL/2). Default 5m." group:"distributed"`
176-
BackendInstallTimeout string `env:"LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT" help:"NATS round-trip timeout for backend.install requests sent to worker nodes (default 15m). Increase for slow links pulling multi-GB images." group:"distributed"`
177-
BackendUpgradeTimeout string `env:"LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT" help:"NATS round-trip timeout for backend.upgrade requests (default 15m)." group:"distributed"`
178-
ModelLoadTimeout string `env:"LOCALAI_NATS_MODEL_LOAD_TIMEOUT" help:"Fixed gRPC deadline for the remote LoadModel call sent to a worker node once its backend is installed and model files are staged. Unset (the default), the deadline is derived from the checkpoint size instead: 5m plus 20s per GiB, capped at 6h, so multi-tens-of-GB diffusion/video checkpoints get the minutes they need without a fixed cliff. Set this only to pin a specific budget; the value is used verbatim, including when it is shorter than the derived one." group:"distributed"`
179-
NatsAccountSeed string `env:"LOCALAI_NATS_ACCOUNT_SEED" help:"NATS account signing seed (SU...) used to mint per-node worker JWTs at registration" group:"distributed"`
180-
NatsServiceJWT string `env:"LOCALAI_NATS_SERVICE_JWT" help:"NATS user JWT for the frontend (and agent workers) to publish control-plane messages" group:"distributed"`
181-
NatsServiceSeed string `env:"LOCALAI_NATS_SERVICE_SEED" help:"NATS user signing seed (SU...) paired with LOCALAI_NATS_SERVICE_JWT" group:"distributed"`
182-
NatsWorkerJWTTTL string `env:"LOCALAI_NATS_WORKER_JWT_TTL" help:"Lifetime of minted per-node NATS JWTs (e.g. 24h, default 24h)" group:"distributed"`
183-
NatsRequireAuth bool `env:"LOCALAI_NATS_REQUIRE_AUTH" default:"false" help:"Require NATS JWT credentials (service JWT + account seed) when distributed mode is enabled" group:"distributed"`
184-
NatsTLSCA string `env:"LOCALAI_NATS_TLS_CA" type:"existingfile" help:"PEM file for NATS server CA (private PKI); use with tls:// in --nats-url" group:"distributed"`
185-
NatsTLSCert string `env:"LOCALAI_NATS_TLS_CERT" type:"existingfile" help:"Client certificate for NATS mTLS" group:"distributed"`
186-
NatsTLSKey string `env:"LOCALAI_NATS_TLS_KEY" type:"existingfile" help:"Client private key for NATS mTLS" group:"distributed"`
187-
ExposeNodeHeader bool `env:"LOCALAI_EXPOSE_NODE_HEADER" default:"false" help:"Set the X-LocalAI-Node response header on inference responses (OpenAI chat/completions/embeddings, Anthropic /v1/messages, Ollama /api/chat,/api/generate,/api/embed) with the ID of the worker that served the request. Disabled by default: the node ID reveals internal topology and should not be exposed on a public endpoint. Best-effort: under heavy concurrency the header may reflect a recent routing decision rather than this exact request's." group:"distributed"`
188-
ModelScheduling string `env:"LOCALAI_MODEL_SCHEDULING" help:"Declarative per-model scheduling config applied at startup (inline JSON list of {model_name,node_selector,min_replicas,max_replicas,replicas:\"all\"}). Authoritative: overwrites matching models on every boot. Distributed mode only." group:"distributed"`
189-
ModelSchedulingConfig string `env:"LOCALAI_MODEL_SCHEDULING_CONFIG" help:"Path to a YAML file with the same per-model scheduling list as LOCALAI_MODEL_SCHEDULING. Distributed mode only." group:"distributed"`
161+
Distributed bool `env:"LOCALAI_DISTRIBUTED" default:"false" help:"Enable distributed mode (requires PostgreSQL + NATS)" group:"distributed"`
162+
InstanceID string `env:"LOCALAI_INSTANCE_ID" help:"Unique instance ID for distributed mode (auto-generated UUID if empty)" group:"distributed"`
163+
NatsURL string `env:"LOCALAI_NATS_URL" help:"NATS server URL (e.g., nats://localhost:4222)" group:"distributed"`
164+
StorageURL string `env:"LOCALAI_STORAGE_URL" help:"S3-compatible storage endpoint URL (e.g., http://minio:9000)" group:"distributed"`
165+
StorageBucket string `env:"LOCALAI_STORAGE_BUCKET" default:"localai" help:"S3 bucket name for object storage" group:"distributed"`
166+
StorageRegion string `env:"LOCALAI_STORAGE_REGION" default:"us-east-1" help:"S3 region" group:"distributed"`
167+
StorageAccessKey string `env:"LOCALAI_STORAGE_ACCESS_KEY" help:"S3 access key ID" group:"distributed"`
168+
StorageSecretKey string `env:"LOCALAI_STORAGE_SECRET_KEY" help:"S3 secret access key" group:"distributed"`
169+
RegistrationToken string `env:"LOCALAI_REGISTRATION_TOKEN" help:"Token that backend nodes must provide to register (empty = no auth required)" group:"distributed"`
170+
RegistrationRequireAuth bool `env:"LOCALAI_REGISTRATION_REQUIRE_AUTH" default:"false" help:"Fail startup when distributed mode is enabled but LOCALAI_REGISTRATION_TOKEN is empty (node endpoints and worker file-transfer server would otherwise be unauthenticated)" group:"distributed"`
171+
DistributedRequireAuth bool `env:"LOCALAI_DISTRIBUTED_REQUIRE_AUTH" default:"false" help:"Umbrella switch: require BOTH NATS JWT credentials and a registration token when distributed mode is enabled (implies --nats-require-auth and --registration-require-auth)" group:"distributed"`
172+
AutoApproveNodes bool `env:"LOCALAI_AUTO_APPROVE_NODES" default:"false" help:"Auto-approve new worker nodes (skip admin approval)" group:"distributed"`
173+
DistributedSharedModels bool `env:"LOCALAI_DISTRIBUTED_SHARED_MODELS" default:"false" help:"Assert that every node mounts the SAME models directory at the SAME path (shared volume). When true, the router skips staging model files to workers and loads them directly from the shared path, avoiding re-downloads." group:"distributed"`
174+
DistributedPrefixCache bool `env:"LOCALAI_DISTRIBUTED_PREFIX_CACHE" default:"true" help:"Enable prefix-cache-aware routing in distributed mode (default true). When false, routing falls back to round-robin." group:"distributed"`
175+
DistributedDiskHeadroomCheck bool `env:"LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK" default:"true" help:"Reject worker nodes that lack free space to store the model, at scheduling time rather than partway through staging (default true). Free space is measured on the filesystem backing each worker's models directory, and compared against the model's own size plus a small margin. When false, node selection ignores free disk (pre-#11054 behaviour); the check still runs and warns when it would have rejected every node. Can also be toggled at runtime via the distributed_disk_headroom_check setting." group:"distributed"`
176+
DistributedPrefixCacheTTL string `env:"LOCALAI_DISTRIBUTED_PREFIX_CACHE_TTL" help:"Idle-timeout for prefix-cache index entries; also drives the background eviction cadence (every TTL/2). Default 5m." group:"distributed"`
177+
BackendInstallTimeout string `env:"LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT" help:"NATS round-trip timeout for backend.install requests sent to worker nodes (default 15m). Increase for slow links pulling multi-GB images." group:"distributed"`
178+
BackendUpgradeTimeout string `env:"LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT" help:"NATS round-trip timeout for backend.upgrade requests (default 15m)." group:"distributed"`
179+
ModelLoadTimeout string `env:"LOCALAI_NATS_MODEL_LOAD_TIMEOUT" help:"Fixed gRPC deadline for the remote LoadModel call sent to a worker node once its backend is installed and model files are staged. Unset (the default), the deadline is derived from the checkpoint size instead: 5m plus 20s per GiB, capped at 6h, so multi-tens-of-GB diffusion/video checkpoints get the minutes they need without a fixed cliff. Set this only to pin a specific budget; the value is used verbatim, including when it is shorter than the derived one." group:"distributed"`
180+
NatsAccountSeed string `env:"LOCALAI_NATS_ACCOUNT_SEED" help:"NATS account signing seed (SU...) used to mint per-node worker JWTs at registration" group:"distributed"`
181+
NatsServiceJWT string `env:"LOCALAI_NATS_SERVICE_JWT" help:"NATS user JWT for the frontend (and agent workers) to publish control-plane messages" group:"distributed"`
182+
NatsServiceSeed string `env:"LOCALAI_NATS_SERVICE_SEED" help:"NATS user signing seed (SU...) paired with LOCALAI_NATS_SERVICE_JWT" group:"distributed"`
183+
NatsWorkerJWTTTL string `env:"LOCALAI_NATS_WORKER_JWT_TTL" help:"Lifetime of minted per-node NATS JWTs (e.g. 24h, default 24h)" group:"distributed"`
184+
NatsRequireAuth bool `env:"LOCALAI_NATS_REQUIRE_AUTH" default:"false" help:"Require NATS JWT credentials (service JWT + account seed) when distributed mode is enabled" group:"distributed"`
185+
NatsTLSCA string `env:"LOCALAI_NATS_TLS_CA" type:"existingfile" help:"PEM file for NATS server CA (private PKI); use with tls:// in --nats-url" group:"distributed"`
186+
NatsTLSCert string `env:"LOCALAI_NATS_TLS_CERT" type:"existingfile" help:"Client certificate for NATS mTLS" group:"distributed"`
187+
NatsTLSKey string `env:"LOCALAI_NATS_TLS_KEY" type:"existingfile" help:"Client private key for NATS mTLS" group:"distributed"`
188+
ExposeNodeHeader bool `env:"LOCALAI_EXPOSE_NODE_HEADER" default:"false" help:"Set the X-LocalAI-Node response header on inference responses (OpenAI chat/completions/embeddings, Anthropic /v1/messages, Ollama /api/chat,/api/generate,/api/embed) with the ID of the worker that served the request. Disabled by default: the node ID reveals internal topology and should not be exposed on a public endpoint. Best-effort: under heavy concurrency the header may reflect a recent routing decision rather than this exact request's." group:"distributed"`
189+
ModelScheduling string `env:"LOCALAI_MODEL_SCHEDULING" help:"Declarative per-model scheduling config applied at startup (inline JSON list of {model_name,node_selector,min_replicas,max_replicas,replicas:\"all\"}). Authoritative: overwrites matching models on every boot. Distributed mode only." group:"distributed"`
190+
ModelSchedulingConfig string `env:"LOCALAI_MODEL_SCHEDULING_CONFIG" help:"Path to a YAML file with the same per-model scheduling list as LOCALAI_MODEL_SCHEDULING. Distributed mode only." group:"distributed"`
190191

191192
Version bool
192193

@@ -409,6 +410,9 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
409410
if !r.DistributedPrefixCache {
410411
opts = append(opts, config.DisablePrefixCache)
411412
}
413+
if !r.DistributedDiskHeadroomCheck {
414+
opts = append(opts, config.DisableDiskHeadroomCheck)
415+
}
412416
if r.DistributedPrefixCacheTTL != "" {
413417
d, err := time.ParseDuration(r.DistributedPrefixCacheTTL)
414418
if err != nil {

‎core/config/disk_headroom_setting_test.go‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
package config_test
2+
3+
import (
4+
. "github.com/onsi/ginkgo/v2"
5+
. "github.com/onsi/gomega"
6+
7+
"github.com/mudler/LocalAI/core/config"
8+
)
9+
10+
// The disk-headroom admission check has two operator surfaces that must not
11+
// drift into two sources of truth: an env/CLI flag that sets the boot value,
12+
// and a runtime setting that can override it live. Both write the SAME
13+
// ApplicationConfig member, which the SmartRouter reads on every scheduling
14+
// decision.
15+
var _ = Describe("distributed disk-headroom check setting", func() {
16+
It("is enabled by default", func() {
17+
o := config.NewApplicationConfig()
18+
19+
Expect(o.Distributed.DiskHeadroomDisabled).To(BeFalse(),
20+
"the check must default ON — it prevents a measured production failure")
21+
Expect(*o.ToRuntimeSettings().DistributedDiskHeadroomCheck).To(BeTrue())
22+
})
23+
24+
It("reports the env/CLI value through the runtime settings snapshot", func() {
25+
o := config.NewApplicationConfig(config.DisableDiskHeadroomCheck)
26+
27+
Expect(*o.ToRuntimeSettings().DistributedDiskHeadroomCheck).To(BeFalse())
28+
})
29+
30+
It("lets a runtime override beat the env value, in both directions", func() {
31+
// Env said "off"; the operator turns it back on at runtime.
32+
o := config.NewApplicationConfig(config.DisableDiskHeadroomCheck)
33+
on := true
34+
o.ApplyRuntimeSettings(&config.RuntimeSettings{DistributedDiskHeadroomCheck: &on})
35+
Expect(o.Distributed.DiskHeadroomDisabled).To(BeFalse())
36+
37+
// Env said "on" (the default); the operator turns it off at runtime.
38+
o = config.NewApplicationConfig()
39+
off := false
40+
o.ApplyRuntimeSettings(&config.RuntimeSettings{DistributedDiskHeadroomCheck: &off})
41+
Expect(o.Distributed.DiskHeadroomDisabled).To(BeTrue())
42+
})
43+
44+
It("leaves the value alone when the runtime settings do not mention it", func() {
45+
o := config.NewApplicationConfig(config.DisableDiskHeadroomCheck)
46+
47+
o.ApplyRuntimeSettings(&config.RuntimeSettings{})
48+
49+
Expect(o.Distributed.DiskHeadroomDisabled).To(BeTrue())
50+
})
51+
})

‎core/config/distributed_config.go‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,22 @@ type DistributedConfig struct {
8888
AgentWorkerConcurrency int `yaml:"agent_worker_concurrency" json:"agent_worker_concurrency" env:"LOCALAI_AGENT_WORKER_CONCURRENCY"`
8989
JobWorkerConcurrency int `yaml:"job_worker_concurrency" json:"job_worker_concurrency" env:"LOCALAI_JOB_WORKER_CONCURRENCY"`
9090

91+
// DiskHeadroomDisabled turns off the scheduler's free-disk admission check,
92+
// restoring the pre-#11054 behaviour where node selection ignores whether a
93+
// node can actually store the model. The check is ON by default because it
94+
// prevents a measured failure (a node with 0 bytes free accepted a 70GB
95+
// model and failed 16 minutes into staging); this is the escape hatch for
96+
// setups where our size estimate is wrong (deduplicating filesystems, a
97+
// worker that fetches its own weights), not the norm.
98+
//
99+
// Disabling does NOT silence the check: it still runs and warns when it
100+
// would have rejected every node, so the operator keeps the diagnosis
101+
// without being blocked. See SmartRouter.scheduleNewModel.
102+
//
103+
// Stored as the negation of the CLI/runtime flag so the zero value is
104+
// "enabled" (mirrors PrefixCacheDisabled).
105+
DiskHeadroomDisabled bool
106+
91107
// PrefixCacheDisabled turns off prefix-cache-aware routing, falling back to
92108
// round-robin (the floor). Prefix-cache routing is ON by default in
93109
// distributed mode; this flag exists so operators can opt out. The CLI
@@ -310,6 +326,13 @@ var EnableDistributedSharedModels = func(o *ApplicationConfig) {
310326
o.Distributed.SharedModels = true
311327
}
312328

329+
// DisableDiskHeadroomCheck turns off the scheduler's free-disk admission
330+
// check (see DistributedConfig.DiskHeadroomDisabled). The check is enabled by
331+
// default in distributed mode.
332+
var DisableDiskHeadroomCheck = func(o *ApplicationConfig) {
333+
o.Distributed.DiskHeadroomDisabled = true
334+
}
335+
313336
// DisablePrefixCache turns off prefix-cache-aware routing (falls back to
314337
// round-robin). Prefix-cache routing is enabled by default in distributed mode.
315338
var DisablePrefixCache = func(o *ApplicationConfig) {
@@ -356,6 +379,10 @@ const (
356379
FlagBackendInstallTimeout = "backend-install-timeout"
357380
FlagBackendUpgradeTimeout = "backend-upgrade-timeout"
358381
FlagModelLoadTimeout = "model-load-timeout"
382+
// FlagDiskHeadroomCheck names the disk-headroom toggle. It is quoted in
383+
// the warning the check emits while disabled, so the operator reading a
384+
// log line knows exactly which knob produced it.
385+
FlagDiskHeadroomCheck = "distributed-disk-headroom-check"
359386
)
360387

361388
// Defaults for distributed timeouts.

0 commit comments

Comments
 (0)