Skip to content

fix(agentpool): close truncate-then-read race in agent_jobs.json persistence#9811

Merged
mudler merged 2 commits into
masterfrom
worktree-fix-agentjobs-race
May 13, 2026
Merged

fix(agentpool): close truncate-then-read race in agent_jobs.json persistence#9811
mudler merged 2 commits into
masterfrom
worktree-fix-agentjobs-race

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Summary

  • Replaces os.WriteFile in fileJobPersister with a same-directory temp file + os.Rename. POSIX rename is atomic, so concurrent readers see either the prior contents or the new contents — never the zero-byte window that O_TRUNC exposes.
  • Collapses AgentJobService.{Load,Save}{Tasks,Jobs}{FromFile,ToFile} to thin wrappers over fileJobPersister, removing the duplicate write path and the redundant service.fileMutex / service.tasksFile / service.jobsFile fields.

Why

Three call sites wrote and read agent_jobs.json through three different mutex instances:

# Site Mutex
1 go saveJobs(job) spawned from ExecuteJob -> fileJobPersister.saveJobsToFile fileJobPersister.mu
2 AgentJobService.SaveJobsToFile service.fileMutex
3 AgentJobService.LoadJobsFromFile on a second service instance a different service.fileMutex

Nothing serialized them, and both writers used os.WriteFile (which opens O_WRONLY|O_CREATE|O_TRUNC). A reader that landed between the truncate and the write saw a zero-byte file and surfaced as failed to parse jobs file: unexpected end of JSON input at offset 0.

The macOS tests-apple job started hitting this every push once paths-ignore was dropped from .github/workflows/test.yml in 6d2b7d8; run 25823124797 was the first observed failure. The race has existed since #9124 (the file persister was introduced); it just took more sample runs on macOS for the scheduler to land in the bad window.

The combination here closes both axes:

  • Inside one service, all save paths now share fileJobPersister.mu, so the async saveJobs goroutine and an explicit SaveJobsToFile serialize against each other.
  • Across two service instances pointing at the same file (the bootstrap and test patterns), the atomic rename guarantees readers never see a partial file.

Test plan

  • New regression spec does not surface a partial file when saves and loads race hammers SaveJobsToFile and LoadJobsFromFile concurrently across two service instances for 500 ms; on master this reproduces the exact macOS error on Linux within that window, on this branch it passes.
  • go run github.com/onsi/ginkgo/v2/ginkgo --until-it-fails --focus="does not surface a partial file" ./core/services/agentpool/ — 54 attempts in 30 s, all green.
  • go test -race ./core/services/agentpool/... clean.
  • golangci-lint run --config=.golangci.yml ./core/services/agentpool/... clean.
  • Full ./core/services/agentpool/... suite (52 specs + the new one) passes.

mudler added 2 commits May 13, 2026 21:43
…istence

Three call sites wrote and read agent_jobs.json (and agent_tasks.json)
through three independent mutexes:

  - AgentJobService.ExecuteJob spawns go saveJobs(job) -> fileJobPersister
    holding p.mu
  - AgentJobService.SaveJobsToFile holding service.fileMutex
  - AgentJobService.LoadJobsFromFile on a separate service instance holding
    a different service.fileMutex

Nothing serialized those mutexes, and both writers used os.WriteFile, which
opens O_TRUNC. A reader landing between the truncate and the write saw a
zero-byte file and surfaced as `unexpected end of JSON input` at offset 0.
The macOS tests-apple job started hitting this consistently once the path
filter was removed from .github/workflows/test.yml and the file-mode race
test ran on every push (run 25823124797 was the first observed failure).

Two changes close the window:

1. fileJobPersister.saveTasksToFile / saveJobsToFile now write to a
   same-directory temp file and os.Rename to the final path. rename(2) is
   atomic on POSIX, so concurrent readers see either the prior contents or
   the new contents and never a zero-byte window. The helper Syncs before
   close so a crash mid-write leaves either the old file intact or the temp
   behind (cleaned up on next save).

2. AgentJobService.{Load,Save}{Tasks,Jobs}{FromFile,ToFile} are collapsed
   to thin wrappers around fileJobPersister, removing the duplicate write
   path and the redundant service.fileMutex / service.tasksFile /
   service.jobsFile fields. Within a single service all task/job I/O now
   serializes on the persister's mutex; the atomic rename handles the
   cross-instance case the tests exercise.

Adds a regression test that hammers SaveJobsToFile and LoadJobsFromFile
concurrently for 500ms across two service instances on the same paths.
On master this reproduces `unexpected end of JSON input` on Linux within
~500ms; with the fix the suite ran -until-it-fails for 30s (54 attempts,
all green).

Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…terface

The first cut of the race fix made AgentJobService.{Save,Load}{Tasks,Jobs}*
type-assert s.persister to *fileJobPersister so they could reach the
unexported saveTasksToFile / saveJobsToFile helpers. That defeats the
JobPersister interface: the service is back to reasoning about a concrete
implementation instead of an abstraction.

Promote the bulk-flush operations to the interface as FlushTasks / FlushJobs:

  - fileJobPersister.FlushTasks/FlushJobs call the existing private helpers
    (atomic temp+rename writes from the prior commit).
  - dbJobPersister.FlushTasks/FlushJobs are no-ops because SaveTask/SaveJob
    are already write-through to the database.

The service's four file-named methods now talk only to the interface:
LoadTasks/LoadJobs read through s.persister.LoadTasks/LoadJobs, and the
Save side calls FlushTasks/FlushJobs. The "FromFile"/"ToFile" suffixes
stay for backward compat with user_services.go and the existing tests,
but they no longer claim a file-only contract.

Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
mudler merged commit ab01ed1 into master May 13, 2026
21 checks passed
@mudler
mudler deleted the worktree-fix-agentjobs-race branch May 13, 2026 21:58
@mudler mudler added the bug Something isn't working label May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants