Merge pull request #1 from choiyounggi/knowledge/streaming-large-result-sets-20260713-120141

knowledge: streaming large result sets (server-side cursor blocked → fetchmany + write-only)

Author: choiyounggi

.dev-loop/INGEST_REPORT.md

@@ -0,0 +1,35 @@
+# Knowledge flush — 1 insight
+
+Source: RNR-3440 (사내 잠재매물 주간 추출 스크립트 메모리 피크 저감). Candidate:
+"QueryPie 프록시 경유로 대용량 결과를 스트리밍할 때 server-side named cursor 대신
+일반 커서 + `fetchmany` + openpyxl `write_only`."
+
+## Verified best-practice
+
+**Claim 1 — psycopg2 server-side (named) cursor requires a transaction; fails under autocommit.**
+- Source: psycopg2 usage docs — `https://github.com/psycopg/psycopg2/blob/master/doc/src/usage.rst` (via Context7). Quote: "Named cursors are typically created 'WITHOUT HOLD', meaning they exist only within the current transaction. Attempting to fetch from them after a commit or in autocommit mode raises an exception."
+- Matches my live repro (`can't use a named cursor outside of transactions`). → **verified**
+
+**Claim 2 — a client-side (default) cursor pulls the whole result set to the client on execute; `fetchmany` only caps the Python-list explosion.**
+- Source: psycopg2 cursor/usage docs + FAQ (named-cursor advantage = "data is fetched in chunks … minimal client memory"). By contrast the default cursor buffers the full result in libpq. → **verified**
+
+**Claim 3 — openpyxl `write_only` gives near-constant memory (<10 MB); one save only; lxml is for serialization speed, not the memory saving.**
+- Source: openpyxl Optimised Modes — `https://openpyxl.readthedocs.io/en/stable/optimized.html` (via WebSearch). "keeping memory usage under 10Mb"; "A write-only workbook can only be saved once"; "make sure you have lxml installed" for large dumps (speed).
+- This **corrects** the raw candidate's "lxml unnecessary" → precise form: unnecessary *for the memory win*, recommended *for large-dump speed*. Confirmed by my server test (write-only worked with lxml absent). → **verified**
+
+**Claim 4 — QueryPie blocks `BEGIN`, so server-side cursor is impossible there.**
+- Environment-specific, no external source. Live repro in gui context: `autocommit=False` + named cursor → `[ENGINE] No permission to execute BEGIN statement`. → **field-tested**. Generalized in the page to "a read-only access-control proxy that blocks transaction control", with QueryPie as the concrete example (not a product-specific page).
+- Memory figure 838 MB → 38 MB (300k synthetic rows) is my RNR-3440 measurement (`ru_maxrss`, separate processes).
+
+## Existing-layer check
+
+- Pages read: `databases/index.md`, `databases/query-optimization/keyset-pagination.md`, `backend/python/index.md`.
+- Overlap: keyset-pagination is the nearest neighbor (both handle large result sets) but a **distinct** topic — pagination splits the read into many bounded queries; this page streams a *single* query's result in chunks. Not a duplicate → new page + **bidirectional `related` link** added to both.
+- backend/python has no DB-cursor page; the psycopg2/openpyxl specifics live as concrete examples inside the databases page rather than a separate python page (no duplication).
+- Conflicts: none found.
+
+## Routing decision
+
+- Target: **`databases/query-optimization/streaming-large-result-sets.md`** (new page).
+- Category `query-optimization` fits (memory-bounding how a query's result is pulled into the app is query-execution optimization); no new category needed.
+- Registered in `databases/index.md` (query-optimization section) and appended to `log.md`.

log.md

@@ -32,3 +32,4 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-07-11] ingest | security +1 (new category api-exposure): exposing-an-origin-http-api — code-vs-edge control split for putting an origin HTTP API behind a public tunnel/proxy. Captures the non-obvious 5xx-bypasses-request-middleware gotcha (set security headers in the exception handler too — verify with a forced 500), header-only fail-closed auth, docs/OpenAPI off at the origin by default, sanitized 5xx bodies, and the code(validate/headers/errors)-vs-edge(TLS/HSTS/rate-limit/WAF) boundary. Derived from hardening korea-data-suite for external exposure + independent pentest; sources OWASP Secure Headers, MDN, Starlette middleware, FastAPI docs.
 ## [2026-07-12] ingest | databases +1 (new category sqlite): concurrent-access-for-a-read-api — WAL + busy_timeout for a read-heavy API with a background writer; set journal_mode once (per-connection pragma cost ~0.4ms measured vs 0.04ms busy_timeout), single writer, run the batch writer as a separate process (interpreter-lock isolation), scale reads with worker processes not threads. Derived from load-testing korea-data-suite (measured connect/query/pragma costs, GIL-bound concurrency, separate-sync-process deployment). Sources: SQLite WAL/pragma/FAQ docs.
 ## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
+## [2026-07-13] ingest | databases +1 (query-optimization): streaming-large-result-sets — memory-bounded export of a huge single-query result. Client-side cursor pulls the whole set to libpq on execute (fetchmany caps only the Python-list explosion); only a server-side/named cursor truly streams but needs a transaction, so it fails under autocommit or a proxy that blocks BEGIN → fall back to client-side fetchmany + disk spool + openpyxl write_only (measured 300k rows 838MB→38MB). Derived from RNR-3440 (potential-listing weekly extract memory peak); QueryPie BEGIN-block generalized to "read-only access proxy", field-tested. Sources: psycopg2 usage/cursor docs (named-cursor WITHOUT HOLD + autocommit exception), openpyxl optimized-modes (write-only near-constant memory, lxml=speed-not-memory).

wiki/databases/index.md

@@ -21,6 +21,7 @@ Match your situation to a "load when" line; load only matching pages.
 |------|-----------|
 | [reading-execution-plans](query-optimization/reading-execution-plans.md) | A single query/statement is slow; verifying an index/query change with EXPLAIN before shipping (endpoint slow because it runs *many* fast queries → n-plus-one-queries) |
 | [keyset-pagination](query-optimization/keyset-pagination.md) | Implementing pagination, infinite scroll, or batch table walks |
+| [streaming-large-result-sets](query-optimization/streaming-large-result-sets.md) | Exporting/reading a very large single-query result into the app; process memory peaks on `fetchall` or building a big file; server-side cursor blocked by autocommit or a read-only proxy |
 | [large-in-lists](query-optimization/large-in-lists.md) | Building `IN (...)` queries whose list size can grow (batch lookups, fetch-by-ids) |
 | [n-plus-one-queries](query-optimization/n-plus-one-queries.md) | Loading a list plus per-row associations via an ORM; query count scales with result size |
 | [existence-and-count-checks](query-optimization/existence-and-count-checks.md) | Writing "is there any…", counts, badges, or gating logic on row presence |

wiki/databases/query-optimization/keyset-pagination.md

@@ -8,7 +8,7 @@ sources:
   - https://use-the-index-luke.com/no-offset
   - https://www.postgresql.org/docs/current/queries-limit.html
 last_verified: 2026-07-10
-related: [databases-indexing-composite-index-column-order]
+related: [databases-indexing-composite-index-column-order, databases-query-optimization-streaming-large-result-sets]
 ---
 
 # Paginating Large Result Sets

wiki/databases/query-optimization/streaming-large-result-sets.md

@@ -0,0 +1,58 @@
+---
+id: databases-query-optimization-streaming-large-result-sets
+domain: databases
+category: query-optimization
+applies_to: [postgresql, python, general]
+confidence: verified
+sources:
+  - https://github.com/psycopg/psycopg2/blob/master/doc/src/usage.rst
+  - https://github.com/psycopg/psycopg2/blob/master/doc/src/cursor.rst
+  - https://openpyxl.readthedocs.io/en/stable/optimized.html
+last_verified: 2026-07-13
+related: [databases-query-optimization-keyset-pagination]
+---
+
+# Streaming Large Query Result Sets into the App
+
+## When this applies
+
+A single query returns a very large result (hundreds of thousands of rows or
+more) and your process reads it all into the app — typically to export to a file
+(Excel/CSV) or feed another sink. The pain is process **memory peak**, not query
+speed. (Splitting the read into many bounded queries → keyset-pagination.)
+
+## Do this
+
+| Situation | Do |
+|-----------|----|
+| Stream a large single-query result to the app | Server-side cursor (`cursor(name=...)` in psycopg2 → PostgreSQL `DECLARE`) fetching in chunks via `itersize`/`fetchmany` — the DB sends batches, client memory stays minimal |
+| Server-side cursor is unavailable (env blocks transactions/`BEGIN`) | Client-side cursor + `fetchmany(N)` batches so you never build the full Python list (`fetchall`), and write output through a **streaming writer** instead of buffering it |
+| Output is Excel | openpyxl `Workbook(write_only=True)` + `ws.append(row)` — never holds the whole workbook; near-constant memory (<10 MB). Install `lxml` for large dumps (serialization speed, **not** the memory saving) |
+| The result must be traversed twice (e.g. a post-pass needs the whole set first) | Don't hold it all in Python — spool each row to a local file (pickle/CSV) once, then stream-read the spool for the second pass |
+
+Why, in order of memory impact:
+
+1. **A client-side (default) cursor pulls the entire result set to the client on
+   `execute`.** `fetchmany` only hands you slices of what libpq already buffered,
+   so it caps the *Python object* explosion but not the driver buffer. Avoiding
+   `fetchall` (the full Python list) is necessary but not sufficient.
+2. **Only a server-side (named) cursor truly streams** — the DB ships chunks. But
+   it lives *inside a transaction* (created `WITHOUT HOLD`), so it fails under
+   autocommit or wherever transaction control is blocked.
+3. **The output writer is often the bigger share.** A normal openpyxl workbook
+   holds every cell object until `save()`; write-only mode removes that.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| Named cursor under `autocommit=True` | Fetching raises "named cursor isn't valid anymore" / can't use outside a transaction — set `autocommit=False` |
+| A read-only access proxy blocks `BEGIN` (e.g. an access-control proxy like QueryPie) | Server-side cursor is impossible (`No permission to execute BEGIN statement`) → fall back to client-side `fetchmany` + spool + streaming writer. Measured: 300k rows `fetchall`+normal workbook 838 MB → `fetchmany`+write-only **38 MB** |
+| write-only workbook re-save/append | Only one `save()` is allowed (`WorkbookAlreadySaved`). Set column widths / `freeze_panes` **before** the first `append`; compute `auto_filter` from a row count, not `ws.max_row` (unreliable in write-only) |
+| Two-pass over the result via re-running the query | Re-executing a heavy query doubles DB load — spool the single fetch to disk and re-read it instead |
+
+## Sources
+
+- https://github.com/psycopg/psycopg2/blob/master/doc/src/usage.rst — server-side (named) cursors are `WITHOUT HOLD`, invalid after commit / under autocommit; stream large datasets in chunks
+- https://github.com/psycopg/psycopg2/blob/master/doc/src/cursor.rst — `fetchmany`/`itersize` semantics
+- https://openpyxl.readthedocs.io/en/stable/optimized.html — write-only mode, near-constant memory, one save only, lxml recommended for large dumps