Skip to content

Refactor jq/result paths for spaniter v0.2.0#46

Open
apstndb wants to merge 6 commits into
mainfrom
feat/spaniter-v0.2-resultset
Open

Refactor jq/result paths for spaniter v0.2.0#46
apstndb wants to merge 6 commits into
mainfrom
feat/spaniter-v0.2-resultset

Conversation

@apstndb

@apstndb apstndb commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adopt unreleased spaniter APIs via local replace until apstndb/spaniter#6 ships as v0.3.0 (WithStatsEncoding, RowIteratorResult.StatsProto / ResultSet, PullRowIteratorSeq).
  • Add resultset package for RowIterator → *sppb.ResultSet materialization (Materialize, FromIteratorResult, CollectListValues, RowToListValue).
  • Slim jqresult to jq-specific concerns: protojson maps, lazy/eager gojq pipeline, RowToJSON.
  • Drive lazy RowIter through PullRowIteratorSeq with WithResult; Stop() stops the pull sequence when started, otherwise stops the raw iterator directly.
  • Coerce read-write jq to eager materialization before jq so DML executes even when the filter ignores input (for example true).
  • Pass WithStatsEncoding(DMLExact) for executed read-write DML via spaniterStatsOpts; PLAN mode keeps default encoding so row_count_exact: 0 is not fabricated.
  • Propagate terminal iterator errors from PullRowIteratorSeq through lazy RowIter.

Merge note: This PR depends on spaniter v0.3.0. Merge spaniter#6 first, then bump go.mod to the released tag and remove replace. GitHub CI build/lint will fail until then because the published v0.2.0 module does not include these APIs.

Test plan

  • go test ./... (with local replace github.com/apstndb/spaniter => ../spaniter)
  • go test -run TestWithCloudSpannerEmulator . (emulator)
  • read-write lazy DML executes with filter true and preserves rowCountExact: "0" for zero-row UPDATE
  • PLAN DML materialization omits fabricated rowCountExact
  • jqresult lazy unit tests (stats deferral, duplicate .rows, Stop without drain, terminal error propagation)
  • resultset / protojson DML zero-row-count stats tests

Add a resultset package for RowIterator materialization, drive lazy RowIter
through spaniter, and adopt nil-return ResultSetStats semantics from v0.2.0.

Co-authored-by: Cursor <cursoragent@cursor.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the codebase to extract the logic for draining Cloud Spanner row iterators into a new resultset package, utilizing the updated spaniter v0.2.0 library. It also updates jqresult.RowIter to stream rows using Go's iter.Pull2 sequence mechanism. Feedback on the changes highlights a potential goroutine leak in RowIter.Stop(). When r.stopSeq is non-nil, the method returns early without calling r.rowIter.Stop(), which could leave the background goroutine blocked on an active Spanner network call. It is recommended to ensure r.rowIter.Stop() is always called to properly release resources.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread jqresult/rowiter.go
Propagate terminal iterator errors, stop both pull sequence and RowIterator,
drop WithDrainOnEarlyStop from ordinary streaming, and use ResultSetStatsForDML
on read-write DML paths so row_count_exact:0 is preserved.

Co-authored-by: Cursor <cursoragent@cursor.com>
@apstndb

apstndb commented Jun 26, 2026

Copy link
Copy Markdown
Owner Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the codebase to extract result set materialization logic into a new resultset package, which leverages the updated github.com/apstndb/spaniter dependency (v0.2.0). The jqresult package has been updated to use spaniter sequences and pull-based iteration for row streaming, and to handle DML-specific statistics correctly via a new dml parameter. Additionally, obsolete helper functions in main.go have been removed, and comprehensive unit tests have been added for the new and refactored components. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Drain read-write lazy iterators on cleanup so DML runs when jq ignores input,
use ResultSetStatsForDML only outside PLAN mode, and avoid double-stopping
RowIterator after spaniter.RowIteratorSeq takes ownership.

Co-authored-by: Cursor <cursoragent@cursor.com>
@apstndb

apstndb commented Jun 26, 2026

Copy link
Copy Markdown
Owner Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the jqresult package to leverage spaniter's iterator sequence and introduces a new resultset package to handle draining Cloud Spanner row iterators into protobuf ResultSets. It also adds support for preserving standard DML row counts and ensuring lazy JQ filters fully consume the RowIterator during read-write transactions. A review comment identified a potential bug in the test helper runLazyReadWriteDML where retrying a transaction could lead to duplicate elements in the output slice; resetting the slice inside the transaction block was suggested to resolve this.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread integration_test.go
Comment on lines +36 to +38
var out []any
_, err := client.ReadWriteTransaction(ctx, func(ctx context.Context, tx *spanner.ReadWriteTransaction) error {
rowIter := tx.QueryWithOptions(ctx, spanner.Statement{SQL: sql}, spanner.QueryOptions{})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Cloud Spanner, the transaction function passed to ReadWriteTransaction can be executed multiple times if the transaction aborts and retries. Since out is declared outside the transaction scope and appended to inside the transaction, retrying will result in duplicate elements in out. Resetting out to nil at the start of the transaction function ensures that retries do not accumulate duplicate values.

Suggested change
var out []any
_, err := client.ReadWriteTransaction(ctx, func(ctx context.Context, tx *spanner.ReadWriteTransaction) error {
rowIter := tx.QueryWithOptions(ctx, spanner.Statement{SQL: sql}, spanner.QueryOptions{})
var out []any
_, err := client.ReadWriteTransaction(ctx, func(ctx context.Context, tx *spanner.ReadWriteTransaction) error {
out = nil
rowIter := tx.QueryWithOptions(ctx, spanner.Statement{SQL: sql}, spanner.QueryOptions{})

apstndb and others added 3 commits June 26, 2026 21:50
Read-write statements always materialize before jq so lazy mode applies
only to read-only queries. Drop completeOnStop and use spaniter
StatsEncoding, ResultSet, and PullRowIteratorSeq (local replace until release).

Co-authored-by: Cursor <cursoragent@cursor.com>
Pass stats encoding through spaniter drain options, use StatsMapFromResult
and ResultSet without bool flags, and keep lazy jq read-only only.

Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid accumulating jq output across Spanner transaction retries in
runEagerReadWriteDML integration helper.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant