Skip to content

fix(cli): converge service election after restart #35964

Description

@kitlangton

Summary

Running opencode2 service restart while several TUI clients are connected can leave multiple serve --service processes acting as servers. The registration file can simultaneously appear absent: service status reports stopped while /api/health reaches one of the live servers.

This appears to regress the intended single-winner service election behavior.

Environment

  • Channel: next
  • Version: 0.0.0-next-15145
  • macOS arm64
  • origin/v2: 212c9f99ee2d082082a00f7074df950974747a2a
  • Several long-running TUI clients connected to one managed service

Reproduction

  1. Open several TUI clients against the managed service.

  2. Run:

    opencode2 service restart
  3. Allow clients to reconnect and rediscover the service.

  4. Inspect service status, health, and processes.

Observed timeline

Times are UTC.

21:26:06.305  service restart begins
21:26:06.881-21:26:06.993  at least eight clients start `serve --service` contenders
21:26:06.993  one contender continues with role=server
21:26:20.005  another `serve --service` contender starts and continues as a server
21:26:45       another live `serve --service` process starts
21:26:56       two role=server processes remain alive

At 21:26:56:

opencode2 service status
stopped

opencode2 api get /api/health
{"healthy":true,"version":"0.0.0-next-15145","pid":<newer-pid>}

The process table contained two live compiled binaries with serve --service:

<older-pid>  ... opencode2 serve --service
<newer-pid>  ... opencode2 serve --service

Both had initialized Location services and loaded project/global plugins. At the sampled instant, both used over 1 GB RSS and the newer process was above 100% CPU while initializing.

User-visible behavior during the split state

A prompt was durably admitted and session.execution.started was recorded, but no session.prompt.promoted followed before interruption. A second prompt was admitted while the execution remained pending. The user interrupt was durably recorded about 1.4 seconds after the second admission.

This timing is correlated with the split service state; it does not by itself prove which server owned the pending execution.

Expected behavior

  • Exactly one serve --service contender should initialize and remain alive.
  • All losers should exit before initializing Location graphs.
  • The service registration should always identify the elected healthy PID.
  • service status and /api/health should agree.
  • Reconnecting clients should converge on the elected service instead of starting replacements while a healthy server is alive.

Relevant implementation

packages/cli/src/server-process.ts acquires a service lock, discovers an existing registration, starts/registers when none exists, then releases the lock. A registered process polls the registration ID every 10 seconds and terminates itself if displaced.

The current test in packages/cli/test/service.test.ts covers two concurrent processes in a clean temporary environment. It does not cover:

  • service restart with many already-connected clients
  • reconnect loops racing after the registration is removed
  • a displaced server's cleanup interacting with its successor's registration
  • status/health agreement during replacement
  • contenders arriving after the first winner has started expensive Location initialization

Questions to investigate

  • Which process removed or lost the registration while a healthy server remained reachable?
  • Can an older process finalizer remove a successor's registration despite the ID guard?
  • Can Service.stop/Service.start overlap reconnect-triggered Service.start calls across clients in a way the process-local lock does not serialize?
  • Should the service lock remain held until registration is visible and contenders have observed it?
  • Should losers be prevented from initializing the HTTP server and Location graphs before election is settled?

Suggested coverage

  • Start one registered service, connect several simulated clients, remove/restart it, and race their reconnect discovery loops.
  • Assert only one server remains after the displacement polling interval.
  • Assert the registration file exists and matches the healthy PID throughout convergence.
  • Assert a displaced server cannot remove the winner's registration.
  • Assert prompts submitted during reconnection are either processed or return a clear temporary-unavailable result rather than remaining started but unpromoted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    2.0bugSomething isn't workingcoreAnything pertaining to core functionality of the application (opencode server stuff)

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions