Skip to content

fix: keep async query callback closures alive to prevent use-after-free crash#390

Open
qevolg wants to merge 3 commits into
taosdata:mainfrom
qevolg:fix/async-query-callback-use-after-free
Open

fix: keep async query callback closures alive to prevent use-after-free crash#390
qevolg wants to merge 3 commits into
taosdata:mainfrom
qevolg:fix/async-query-callback-use-after-free

Conversation

@qevolg

@qevolg qevolg commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Problem

Running docs/examples/python/async_query_example.py (and any code using the native async query API) crashes with a segmentation fault:

Fatal Python error: Segmentation fault
Current thread ... (most recent call first):
  <no Python frame>

Root cause

taos_query_a, taos_query_a_with_reqid and taos_fetch_rows_a wrap the Python callback in a ctypes CFUNCTYPE closure that is created inline as a call argument, e.g.:

_libtaos.taos_query_a(connection, c_char_p(sql.encode("utf-8")), async_query_callback_type(callback), param)

These APIs are asynchronous: the native client only stores the function pointer and invokes it later, from a worker thread, when the server responds. But the temporary async_query_callback_type(callback) closure has no remaining Python reference once the wrapper returns, so it can be garbage-collected before the callback fires.

A gdb backtrace shows the crash precisely — the native taskWorkPool thread calls into a freed ctypes closure (callable=0x0):

Thread N "taskWorkPool" received signal SIGSEGV
#0  _CallPythonObject (... callable=0x0, converters=0x0 ...) callbacks.c:151
#1  closure_fcn (libffi)
#4  doRequestCallback        client/src/clientImpl.c:3164
#5  schedulerExecCb          ...

This is a use-after-free, independent of the query data and of the Python version (reproduced on CPython 3.12 with taospy 2.8.9 + native client 3.3.6.0).

Fix

Retain each in-flight callback closure in a module-level registry and release it right after it has been invoked, so the pointer stays valid for the native side while keeping memory usage bounded (the closure is freed once the async callback completes).

Verification

  • async_query_example.py now exits 0 and returns rows correctly (previously: segfault / core dumped).
  • Added an end-to-end check confirming the registry is empty before and after an async query (inflight before: 0 / inflight after : 0) — no leak.
  • python -m py_compile passes; lines stay within the project's black line-length = 119.

Note / out of scope

taos_subscribe (legacy subscription API) uses the same inline-closure pattern (callback = subscribe_callback_type(callback)) and likely has the same latent issue, but its callback lifetime differs (fires repeatedly, tied to the subscription handle) and it is superseded by TMQ, so it is left out of this focused fix.

…ee crash

taos_query_a / taos_query_a_with_reqid / taos_fetch_rows_a wrapped the
Python callback in a ctypes CFUNCTYPE closure that was created inline as a
call argument. These APIs are asynchronous: the native client stores the
function pointer and invokes it later from a worker thread. The temporary
closure had no remaining Python reference once the wrapper function
returned, so it could be garbage collected before the callback fired,
leading to a segfault (use-after-free) in the native worker thread.

Reproduced with docs/examples/python/async_query_example.py:
the crash happens in _CallPythonObject with callable=0x0, on the native
"taskWorkPool" thread (doRequestCallback -> libffi -> _CallPythonObject).

Fix: retain each in-flight callback closure in a module-level registry and
release it right after it has been invoked, so the pointer stays valid for
the native side while keeping memory usage bounded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a mechanism to retain references to asynchronous ctypes callbacks to prevent them from being garbage collected before the native client worker thread invokes them, which could lead to segmentation faults. The feedback suggests handling cases where the callback is None to avoid a TypeError when the native library invokes it, providing a code suggestion to return None immediately in such cases.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread taos/cinterface.py
Comment on lines +525 to +536
def _retain_async_callback(callback_type, callback):
# type: (type, callable) -> object
def wrapper(*args):
try:
callback(*args)
finally:
_inflight_async_callbacks.pop(token, None)

closure = callback_type(wrapper)
token = id(closure)
_inflight_async_callbacks[token] = closure
return closure

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If callback is passed as None, the original behavior of the ctypes wrapper was to pass a NULL pointer to the C library (since callback_type(None) evaluates to a NULL pointer).

With the current implementation, passing None will wrap it in wrapper, creating a valid function pointer. When the native library invokes this callback, it will attempt to call None(*args), resulting in a TypeError: 'NoneType' object is not callable inside the callback thread.

To preserve the original behavior and handle None callbacks defensively, we should check if callback is None and return None immediately.

Suggested change
def _retain_async_callback(callback_type, callback):
# type: (type, callable) -> object
def wrapper(*args):
try:
callback(*args)
finally:
_inflight_async_callbacks.pop(token, None)
closure = callback_type(wrapper)
token = id(closure)
_inflight_async_callbacks[token] = closure
return closure
def _retain_async_callback(callback_type, callback):
# type: (type, callable) -> object
if callback is None:
return None
def wrapper(*args):
try:
callback(*args)
finally:
_inflight_async_callbacks.pop(token, None)
closure = callback_type(wrapper)
token = id(closure)
_inflight_async_callbacks[token] = closure
return closure

qevolg and others added 2 commits July 1, 2026 22:42
…oads

TDengine main no longer builds taosadapter as part of its cmake/make
(removed in the monorepo build-system sync), so the Build workflow's
"Start TDengine" step failed with `taosadapter: No such file or
directory`; the REST health-check curl then hit a refused :6041 and,
under `bash -e`, pytest never ran and no coverage.xml was produced.

Mirror the taos-connector-node recipe:
- update TDengine cmake flags (-DBUILD_TOOLS=false -DBUILD_TEST=off
  -DBUILD_CONTRIB=ON, drop obsolete -DBUILD_HTTP and custom prefix),
  install to the default prefix and run ldconfig so libtaos is found
- expand apt deps needed to build TDengine main from source
- checkout taosdata/taosadapter, build it with Go 1.26.0 and install it
- start the installed taosd/taosadapter and wait for :6041 readiness
- default TDENGINE_TEST_USERNAME/PASSWORD to root/taosdata when the
  secrets are empty (fork PRs)
- set fail_ci_if_error: false on the Codecov upload

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_rest_connection.py::test_token and ::test_wrong_token connect to
external TDengine Cloud endpoints with hardcoded tokens. test_token
failed the Build with a cloud-side "HTTP 503 system resources is not
sufficient", making the run non-deterministic and blocking the Codecov
upload. Deselect both so the local build's suite is deterministic; the
cloud endpoints can be exercised in a dedicated pipeline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 16.66667% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.19%. Comparing base (73cd399) to head (01aff0e).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
taos/cinterface.py 16.66% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #390      +/-   ##
==========================================
- Coverage   82.00%   80.19%   -1.81%     
==========================================
  Files          24       24              
  Lines        3717     3727      +10     
==========================================
- Hits         3048     2989      -59     
- Misses        669      738      +69     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants