fix(agentpool): close truncate-then-read race in agent_jobs.json persistence#9811
Merged
Conversation
…istence
Three call sites wrote and read agent_jobs.json (and agent_tasks.json)
through three independent mutexes:
- AgentJobService.ExecuteJob spawns go saveJobs(job) -> fileJobPersister
holding p.mu
- AgentJobService.SaveJobsToFile holding service.fileMutex
- AgentJobService.LoadJobsFromFile on a separate service instance holding
a different service.fileMutex
Nothing serialized those mutexes, and both writers used os.WriteFile, which
opens O_TRUNC. A reader landing between the truncate and the write saw a
zero-byte file and surfaced as `unexpected end of JSON input` at offset 0.
The macOS tests-apple job started hitting this consistently once the path
filter was removed from .github/workflows/test.yml and the file-mode race
test ran on every push (run 25823124797 was the first observed failure).
Two changes close the window:
1. fileJobPersister.saveTasksToFile / saveJobsToFile now write to a
same-directory temp file and os.Rename to the final path. rename(2) is
atomic on POSIX, so concurrent readers see either the prior contents or
the new contents and never a zero-byte window. The helper Syncs before
close so a crash mid-write leaves either the old file intact or the temp
behind (cleaned up on next save).
2. AgentJobService.{Load,Save}{Tasks,Jobs}{FromFile,ToFile} are collapsed
to thin wrappers around fileJobPersister, removing the duplicate write
path and the redundant service.fileMutex / service.tasksFile /
service.jobsFile fields. Within a single service all task/job I/O now
serializes on the persister's mutex; the atomic rename handles the
cross-instance case the tests exercise.
Adds a regression test that hammers SaveJobsToFile and LoadJobsFromFile
concurrently for 500ms across two service instances on the same paths.
On master this reproduces `unexpected end of JSON input` on Linux within
~500ms; with the fix the suite ran -until-it-fails for 30s (54 attempts,
all green).
Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…terface
The first cut of the race fix made AgentJobService.{Save,Load}{Tasks,Jobs}*
type-assert s.persister to *fileJobPersister so they could reach the
unexported saveTasksToFile / saveJobsToFile helpers. That defeats the
JobPersister interface: the service is back to reasoning about a concrete
implementation instead of an abstraction.
Promote the bulk-flush operations to the interface as FlushTasks / FlushJobs:
- fileJobPersister.FlushTasks/FlushJobs call the existing private helpers
(atomic temp+rename writes from the prior commit).
- dbJobPersister.FlushTasks/FlushJobs are no-ops because SaveTask/SaveJob
are already write-through to the database.
The service's four file-named methods now talk only to the interface:
LoadTasks/LoadJobs read through s.persister.LoadTasks/LoadJobs, and the
Save side calls FlushTasks/FlushJobs. The "FromFile"/"ToFile" suffixes
stay for backward compat with user_services.go and the existing tests,
but they no longer claim a file-only contract.
Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
approved these changes
May 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
os.WriteFileinfileJobPersisterwith a same-directory temp file +os.Rename. POSIX rename is atomic, so concurrent readers see either the prior contents or the new contents — never the zero-byte window thatO_TRUNCexposes.AgentJobService.{Load,Save}{Tasks,Jobs}{FromFile,ToFile}to thin wrappers overfileJobPersister, removing the duplicate write path and the redundantservice.fileMutex/service.tasksFile/service.jobsFilefields.Why
Three call sites wrote and read
agent_jobs.jsonthrough three different mutex instances:go saveJobs(job)spawned fromExecuteJob->fileJobPersister.saveJobsToFilefileJobPersister.muAgentJobService.SaveJobsToFileservice.fileMutexAgentJobService.LoadJobsFromFileon a second service instanceservice.fileMutexNothing serialized them, and both writers used
os.WriteFile(which opensO_WRONLY|O_CREATE|O_TRUNC). A reader that landed between the truncate and the write saw a zero-byte file and surfaced asfailed to parse jobs file: unexpected end of JSON inputat offset 0.The macOS
tests-applejob started hitting this every push oncepaths-ignorewas dropped from.github/workflows/test.ymlin 6d2b7d8; run 25823124797 was the first observed failure. The race has existed since #9124 (the file persister was introduced); it just took more sample runs on macOS for the scheduler to land in the bad window.The combination here closes both axes:
fileJobPersister.mu, so the asyncsaveJobsgoroutine and an explicitSaveJobsToFileserialize against each other.Test plan
does not surface a partial file when saves and loads racehammersSaveJobsToFileandLoadJobsFromFileconcurrently across two service instances for 500 ms; on master this reproduces the exact macOS error on Linux within that window, on this branch it passes.go run github.com/onsi/ginkgo/v2/ginkgo --until-it-fails --focus="does not surface a partial file" ./core/services/agentpool/— 54 attempts in 30 s, all green.go test -race ./core/services/agentpool/...clean.golangci-lint run --config=.golangci.yml ./core/services/agentpool/...clean../core/services/agentpool/...suite (52 specs + the new one) passes.