Skip to content

Service thread deadlocks and crashes when enqueue() blocks with no reader in non-verbose mode #8985

@deadnews

Description

@deadnews

Summary

When a service task (backup/restore via Services API) runs without isc_spb_verbose, the BURP thread can deadlock in Service::enqueue() because no client is reading from svc_stdout. On large databases with enough warning/error output to fill the 1024-byte buffer, this leads to a full server process crash (terminated abnormally, exit code -1).

Environment

  • Firebird 5.x (SuperServer)
  • Database: ~1TB, thousands of metadata objects
  • Client: Go driver using Services API for backup/restore

Steps to reproduce

  1. Start a metadata-only backup + test restore via Services API on a large database (~1TB)
  2. Do not set isc_spb_verbose in the service parameter block
  3. Poll isc_info_svc_running to wait for completion (the standard non-verbose approach)
  4. During restore, BURP generates enough BURP_print() output (index messages, warnings) to fill the 1024-byte svc_stdout buffer

Expected: Restore completes, isc_info_svc_running returns false.
Actual: Firebird process crashes. Guardian restarts it.

Setting isc_spb_verbose and reading output via isc_info_svc_line makes the same operation complete successfully every time.

Root cause analysis

The issue is in Service::enqueue() (src/jrd/svc.cpp):

while (full()) {
    svc_sem_empty.tryEnter(1, 0);  // wait for reader
    if (checkForShutdown() || (svc_flags & SVC_detached))
        return;
}

When the buffer is full, enqueue() waits on svc_sem_empty. This semaphore is only signaled by Service::get(), which is called when the client queries isc_info_svc_line. In non-verbose mode, the client never calls isc_info_svc_line - it only polls isc_info_svc_running, which checks svc_flags without touching the buffer.

The output comes from BURP_print(), which writes to the buffer regardless of verbose mode:

// src/burp/burp.cpp
void BURP_print(USHORT number, const SafeArg& arg) {
    BURP_msg_partial(true, 169, dummy);  // unconditional
    BURP_msg_put(true, number, arg);     // unconditional
}
// routes through burp_output() → outputError()/outputVerbose() → enqueue()

void BURP_verbose(USHORT number, const SafeArg& arg) {
    if (tdgbl->gbl_sw_verbose)  // only this one checks the flag
        BURP_message(number, arg, true);
}

So in non-verbose mode: BURP writes warnings/errors → buffer fills → enqueue() blocks forever → BURP thread holds engine resources → server crash.

firebird.log evidence

Each failed run correlates with a process crash at the exact same second:

Sat Apr 11 10:27:06 2026
    /opt/firebird/bin/fbguard: /opt/firebird/bin/firebird terminated abnormally (-1)

Client-side error at the same moment:

2026-04-11 10:27:06.270 error="metadata restore: restore failed:
    read tcp 127.0.0.1:39774->127.0.0.1:3050: read: connection reset by peer"

The same database completes successfully with isc_spb_verbose enabled.

Suggested fix

enqueue() should not block indefinitely when no reader exists. Possible approaches:

  1. Discard output when no reader — if isc_spb_verbose was not set, skip writing to svc_stdout entirely (since nobody will read it)
  2. Drop oldest data on overflow — advance svc_stdout_head when the buffer is full, discarding unread output instead of blocking
  3. Bounded wait with discard — if svc_sem_empty is not signaled within N iterations, discard and continue

Option 1 seems cleanest: outputVerbose() already checks usvcDataMode — a similar check for whether a verbose reader exists would prevent the deadlock entirely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions