Skip to content

Commit 45ee912

Browse files
Ignacio Van Droogenbroeckclaude
andcommitted
fix: Close DuckDB cursor to release internal buffers
Root cause: DuckDB cursor objects were never being explicitly closed, causing internal buffers to accumulate in memory. The pattern was: ```python rows = conn.execute(sql).fetchall() ``` This creates a cursor object from `conn.execute(sql)`, fetches all rows, but never closes the cursor. Even after `fetchall()` and `del rows`, the cursor object holds references to DuckDB's internal result buffers. These buffers accumulate across queries, causing steady memory growth that GC cannot collect (they're managed by DuckDB's C++ layer, not Python). Solution: Explicitly close cursor after fetching results ```python cursor = conn.execute(sql) rows = cursor.fetchall() # ... process data ... cursor.close() # Release DuckDB internal buffers del cursor ``` This ensures DuckDB's C++ result buffers are freed immediately after the query data is extracted, rather than accumulating until the connection is closed/reset. Expected impact: Memory should stabilize immediately. Each query will fully clean up after itself, preventing the steady growth pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent f4b7362 commit 45ee912

1 file changed

Lines changed: 8 additions & 3 deletions

File tree

api/duckdb_engine.py

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -373,16 +373,21 @@ def _execute_with_pool(self, sql: str) -> Dict[str, Any]:
373373
with self.connection_pool.get_connection(timeout=5.0) as conn:
374374
# Execute query
375375
query_start = time.time()
376-
rows = conn.execute(sql).fetchall()
377-
columns = [desc[0] for desc in conn.description] if conn.description else []
376+
377+
# CRITICAL: Keep reference to cursor to close it properly
378+
cursor = conn.execute(sql)
379+
rows = cursor.fetchall()
380+
columns = [desc[0] for desc in cursor.description] if cursor.description else []
378381
query_time = time.time() - query_start
379382

380383
# Convert rows to list immediately
381384
data = [list(row) for row in rows]
382385
row_count = len(data)
383386

384-
# CRITICAL: Delete rows reference immediately to free DuckDB result memory
387+
# CRITICAL: Delete ALL references to free DuckDB memory
385388
del rows
389+
cursor.close() # Close cursor to release DuckDB internal buffers
390+
del cursor
386391

387392
total_time = time.time() - start_time
388393

0 commit comments

Comments
 (0)