feat(modelartifacts): support bounded parallel Hugging Face file downloads

[Dennisadira]

Closes #11114.

Problem

Snapshot materialization fetched every file through the sequential executor in DownloadFilesWithContext (pkg/downloader/download_plan.go), so a repository split into many shards spent most of its wall clock in per-file request latency rather than moving bytes.

What changed

DownloadFilesWithConcurrency runs up to N whole-file transfers at once via an errgroup with SetLimit. DownloadFilesWithContext stays as a wrapper passing a limit of 1, so the two non-artifact callers — core/gallery/models.go and core/config/model_config_loader.go — keep exactly the behaviour they had: tasks still run in slice order, and the first failure still returns before any later task starts.

Only whole files run in parallel. A single file is never split, so the .partial resume machinery and the per-file SHA check in downloadTaskWithRetry are untouched.

Two details the parallel path forced, both worth a look during review:

• completedBytes is now an atomic.Int64 (pkg/modelartifacts/materializer.go). Several AfterDownload hooks add to it while other files' progress callbacks read it. This is not a precaution — with a plain int64 the race detector reports three races on the new specs.
• The caller's status callback is serialized. The sequential path gave it an implicit guarantee of never being entered twice at once, and it belongs to the caller, so the executor keeps that promise rather than pushing locking onto every call site. AfterDownload is deliberately not serialized: it does the verify-and-promote work that parallelism exists to overlap, so hooks must be safe to run concurrently.

Manifest ordering needed no code change — each hook already writes its own manifest.Files slot by snapshot index, so entries stay in snapshot order regardless of completion order — but nothing pinned that, so there is now a spec for it.

Configuration

Default is 1, i.e. current behaviour. A shared models volume is often the bottleneck rather than the link, so raising this is a deployment decision rather than something to assume.

Flag	Env	Default
--artifact-download-concurrency	LOCALAI_ARTIFACT_DOWNLOAD_CONCURRENCY	1

Available on both local-ai run and local-ai models install; documented in docs/content/reference/cli-reference.md.

Verification

go build, go vet, and go test -race clean on current master for pkg/downloader, pkg/modelartifacts, and core/cli. Ten new specs (6 in pkg/downloader, 4 in pkg/modelartifacts).

I checked the specs actually catch the bugs they claim to, by reverting each part and confirming the expected failure:

Reverted	Result
Materializer back to the sequential executor	records the manifest in snapshot order fails: "files never overlapped, so this proves nothing about ordering"
group.SetLimit(concurrency) removed	3 specs fail, including "more transfers ran at once than the configured limit"
atomic.Int64 back to a plain int64	3 DATA RACE warnings under -race

The concurrency specs assert on the peak number of simultaneous in-flight requests observed by the test server, so they distinguish configured concurrency from actual concurrency, and a limit of 1 (or 0, or negative) is asserted to never overlap at all.

Throughput

Measured locally against an instrumented server rather than a real Hub repo, in the two regimes that bracket real behaviour. 32 files at 64 KiB with a fixed per-request delay, and 16 files at 2 MiB against a single server-wide byte budget shared across all in-flight responses:

Regime	c=1	c=2	c=4	c=8
Latency-bound (50 ms RTT per file)	1669 ms	832 ms (2.0×)	420 ms (4.0×)	213 ms (7.8×)
Bandwidth-bound (shared cap)	803 ms	799 ms (1.0×)	800 ms (1.0×)	800 ms (1.0×)

Near-linear while per-file round-trip latency dominates, and exactly flat once the link or the volume is saturated — parallelism buys nothing there, it just multiplies concurrent load. That asymmetry is the reason the default is 1: the win depends entirely on which side of it a given deployment sits, which is not something LocalAI can infer.

Against the real Hub

End-to-end through the materializer, fetching 11 small files (0.7 MiB total) from sentence-transformers/all-MiniLM-L6-v2, four rounds with the concurrency levels interleaved so link drift hits both arms equally:

	samples	median
concurrency 1	5975, 6689, 8056 ms	6689 ms
concurrency 4	2075, 2184, 2314, 2986 ms	2249 ms

~3.0× median, consistent in both orderings. This is a small-file, latency-dominated repo, which is exactly the case the change targets.

Three caveats I would rather state than bury:

• One further concurrency-1 sample took 240 s — a stalled connection plus retries, discarded as an outlier rather than counted as a win. Sequential timings on a real link are heavy-tailed, so treat the ~3× as indicative, not precise.
• In an earlier run concurrency 8 was slower than 4 (4869 ms vs 4049 ms), consistent with per-host connection limits. More is not better, which is another reason not to raise the default centrally.
• The large-file arm (a few ~23 MiB ONNX blobs) could not be completed at all: HF's xet CDN repeatedly returned download stalled: no data received for 1m0s and http2: timeout awaiting response headers. That failed at concurrency 1 as well, i.e. on the untouched sequential path, so it is a Hub/CDN condition rather than anything this PR introduces — but it does mean I have no real-link data for the bandwidth-bound regime, only the synthetic figures above.

This is not the "representative sharded HF model" the issue asks for — a real 70 GB multi-shard fetch is not something I can run reproducibly here. Happy to measure a specific repo if you have one in mind.

Not done here

• No chunk-level parallelism within a single file — the issue explicitly scopes that out.

[localai-org-maint-bot]

@mudler Good to merge from my review. The bounded executor preserves the sequential compatibility path, cancels sibling work on failure, serializes the legacy status callback, and the materializer uses atomic aggregate accounting while writing each manifest slot by stable snapshot index. The CLI flag is default-safe and documented. Fresh verification on exact head 3bb6c9ef: git diff --check passes, the full race-enabled pkg/modelartifacts suite passes, and all six new DownloadFilesWithConcurrency specs pass under -race. DCO passes. Repository Actions have not run on this fork head yet and still need maintainer authorization before merge.

[mudler]

no need to document it only for huggingface, this has effect for any url, no?

[mudler]

would actually make sense to wire it in the runtime settings as well, so users can also configure it via WebUI

[localai-org-maint-bot]

@mudler I pushed the two requested minor changes to the contributor branch in 4951ca1e: the CLI/docs wording is now artifact-source-neutral, and artifact_download_concurrency is a persisted live runtime setting exposed in the WebUI. The default manager uses an atomic limit so live updates are race-safe; injected materializers remain compatible through an optional setter. Fresh focused config tests and go test -count=1 -race ./pkg/modelartifacts pass, and git diff --check is clean. The React build could not run because this isolated worktree has no installed vite dependency. I did not add a Signed-off-by; DCO may require human handling once GitHub refreshes the new head.