SIGSEGV in SegmentReadTaskScheduler::schedLoop due to use-after-free race with SegmentReaderPoolManager::stop()

[JaySon-Huang]

Bug Report

1. Minimal reproduce step (Required)

Run the unit test DeltaMergeStoreTest.ReadLegacyStringDataCFTiny:

./gtests_dbms --gtest_filter="DeltaMergeStoreTest.ReadLegacyStringDataCFTiny"

The test itself passes, but a SIGSEGV occurs during the global tear-down phase after RUN_ALL_TESTS() returns.

Root cause analysis

The crash is a use-after-free race condition between two global singletons during shutdown:

• SegmentReadTaskScheduler — a singleton with a background thread (schedLoop) that continuously schedules read tasks and pushes MergedTask objects to SegmentReaderPoolManager via SegmentReaderPoolManager::instance().addTask().
• SegmentReaderPoolManager — another singleton that manages SegmentReaderPool objects, each containing a WorkQueue<MergedTaskPtr> with an internal std::mutex.

The race window:

In both gtests_dbms_main.cpp and Server.cpp, the shutdown sequence explicitly destroys SegmentReaderPoolManager's pools before stopping SegmentReadTaskScheduler's background thread:

SegReaderPoolManager::stop()              // Destroys all SegmentReaderPool objects
  → reader_pools.clear()
    → ~SegmentReaderPool()
      → task_queue.finish()              // WorkQueue & its mutex destroyed
      → ~SegmentReader() / t.join()
// ... schedLoop thread is STILL RUNNING ...
// Later, during global static destruction:
// ~SegmentReadTaskScheduler()           // setStop() + sched_thread.join()

SegmentReadTaskScheduler::schedLoop() has no stop flag set during the window between explicit stop() and ~SegmentReadTaskScheduler(). When it wakes up from its 2ms sleep, it calls scheduleOneRound() which may invoke SegmentReaderPoolManager::instance().addTask() → SegmentReaderPool::addTask() → task_queue.push() → lock(mu) — but the mutex has already been destroyed by stop().

Key code locations:

• SegmentReadTaskScheduler::scheduleOneRound() (line 267 of SegmentReadTaskScheduler.cpp): calls SegmentReaderPoolManager::instance().addTask(std::move(merged_task)) without any check that the pool manager is still alive.
• SegmentReaderPoolManager::addTask() (line 208–212 of SegmentReader.cpp): accesses reader_pools[idx] without any mutex protection against concurrent stop().
• SegmentReaderPoolManager::stop() (line 223–227 of SegmentReader.cpp): calls reader_pools.clear() without any mutex protection.

2. What did you expect to see? (Required)

The test should pass and the process should exit cleanly without any signal.

3. What did you see instead (Required)

The test passes ([OK] DeltaMergeStoreTest.ReadLegacyStringDataCFTiny (30 ms)), but a SIGSEGV is raised during the tear-down phase:

Received signal Segmentation fault
 0  StackTrace::StackTrace()
 1  fault_signal_handler(int)
 2  <unknown symbol> [libc.so.6]
 3  __GI___pthread_mutex_lock [libpthread.so.0]
 4  std::__1::mutex::lock() [libc++.so.1]
 5  WorkQueue::push()                                   ← trying to lock destroyed mutex
 6  SegmentReaderPool::addTask()                        ← on destroyed pool
 7  SegmentReaderPoolManager::addTask()                 ← reader_pools already cleared
 8  SegmentReadTaskScheduler::scheduleOneRound()        ← schedLoop still running
 9  SegmentReadTaskScheduler::schedLoop()               ← background thread never stopped

The schedLoop thread (the scheduler's background thread) is still alive and tries to push a MergedTask to a WorkQueue whose mutex has already been destroyed by SegmentReaderPoolManager::stop().

4. What is your TiFlash version? (Required)

v9.0.0-beta.2.pre-199-g9a9c68c1ec, branch pipeline_col

Affected code paths

This race condition exists in both the test harness and the production server:

File	Line	Role
dbms/src/TestUtils/gtests_dbms_main.cpp	125	SegmentReaderPoolManager::instance().stop() — no prior scheduler stop
dbms/src/Server/Server.cpp	1117	SegmentReaderPoolManager::instance().stop() — no prior scheduler stop (SCOPE_EXIT)

In production, the race is less likely to trigger because the TCP listener is closed before SCOPE_EXIT runs, so no new read requests arrive. However, it remains a latent bug that could manifest during fast restart / rolling update scenarios where queries are forcibly terminated mid-read.

Suggested fix

SegmentReadTaskScheduler needs a public stop() method that sets the stop flag and joins the background thread. This must be called before SegmentReaderPoolManager::stop() in both gtests_dbms_main.cpp and Server.cpp:

// In SegmentReadTaskScheduler.h — add public method:
void stop() { setStop(); if (sched_thread.joinable()) sched_thread.join(); }

// In both gtests_dbms_main.cpp and Server.cpp — reorder shutdown:
DB::DM::SegmentReadTaskScheduler::instance().stop();    // ← ADD: stop scheduler FIRST
DB::DM::SegmentReaderPoolManager::instance().stop();    // ← THEN stop reader pools