Skip to content

Commit da8a80c

Browse files
committed
Support batching different queries together.
Fix #4195 by adding an emit_query_batch codegen option that creates a QueryBatch type when using pgx v5. When emit_query_batch: true is set in config (pgx/v5 only), sqlc generates: - A QueryBatch struct wrapping pgx.Batch - NewQueryBatch() constructor - Queue* methods for each query (:one, :many, :exec, :execrows, :execresult) - ExecuteBatch() on Queries to send all queued queries in one round-trip This expands the batching support of existing :batchone/:batchmany/:batchexec annotations, which required separate query definitions and only supported baching the same query with different parameters. Batching multiple instances of the same query with different parameters is also supported via the new interface. While there is probably not a good reason to use both in the same package, you can generate and use both without error for backwards compatibility. Key design decisions - Follows pgx v5's recommended QueuedQuery callback pattern - QueryBatch.Batch is exported so users can mix generated Queue* calls with custom pgx batch operations - :exec queries have no callback (consistent with pgx - errors propagate via Close/ExecuteBatch) - :one callbacks receive a bool indicating whether a row was found Changes - internal/codegen/golang/gen.go - Wire up EmitQueryBatch option, file generation, and validation - internal/codegen/golang/imports.go - Add queryBatchImports() with import logic that skips struct field types (struct definitions live in query.sql.go, not the batch file). Extract queryUsesType() from inline closure for reuse. - internal/codegen/golang/opts/options.go - Add EmitQueryBatch and OutputQueryBatchFileName options - internal/codegen/golang/templates/pgx/queryBatchCode.tmpl - New template for all query batch methods - internal/codegen/golang/templates/template.tmpl - Add queryBatchFile and queryBatchCode template definitions Test coverage - emit_query_batch - Full test with :one, :many, :exec, :execrows, :execresult - emit_query_batch_db_arg - With emit_methods_with_db_argument: true - emit_query_batch_minimal - Non-struct return types (verifies import handling for e.g. pgtype.Timestamptz) - emit_query_batch_overrides - With custom type overrides (verifies batch file doesn't import types only used in struct fields) - emit_query_batch_with_batch - Combines old-style :batchexec with emit_query_batch (verifies both batch.go and query_batch.sql.go coexist correctly) Documentation - docs/reference/config.md - Added emit_query_batch and output_query_batch_file_name - docs/reference/query-annotations.md - New section with usage examples contrasting with :batch* annotations
1 parent ce83d3f commit da8a80c

File tree

43 files changed

+1565
-3
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+1565
-3
lines changed

docs/reference/config.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,8 @@ The `gen` mapping supports the following keys:
165165
- `emit_all_enum_values`:
166166
- If true, emit a function per enum type
167167
that returns all valid enum values.
168+
- `emit_query_batch`:
169+
- If true, generate a `QueryBatch` type with `Queue*` methods that batch multiple different queries into a single round-trip. Uses pgx v5's `QueuedQuery` callback API. Only supported with `sql_package: pgx/v5`. Defaults to `false`.
168170
- `emit_sql_as_comment`:
169171
- If true, emits the SQL statement as a code-block comment above the generated function, appending to any existing comments. Defaults to `false`.
170172
- `build_tags`:
@@ -179,6 +181,8 @@ The `gen` mapping supports the following keys:
179181
- If `true`, sqlc won't generate table and enum structs that aren't used in queries for a given package. Defaults to `false`.
180182
- `output_batch_file_name`:
181183
- Customize the name of the batch file. Defaults to `batch.go`.
184+
- `output_query_batch_file_name`:
185+
- Customize the name of the query batch file. Defaults to `query_batch.sql.go`.
182186
- `output_db_file_name`:
183187
- Customize the name of the db file. Defaults to `db.go`.
184188
- `output_models_file_name`:
@@ -448,6 +452,8 @@ Each mapping in the `packages` collection has the following keys:
448452
- `emit_all_enum_values`:
449453
- If true, emit a function per enum type
450454
that returns all valid enum values.
455+
- `emit_query_batch`:
456+
- If true, generate a `QueryBatch` type with `Queue*` methods that batch multiple different queries into a single round-trip. Uses pgx v5's `QueuedQuery` callback API. Only supported with `sql_package: pgx/v5`. Defaults to `false`.
451457
- `build_tags`:
452458
- If set, add a `//go:build <build_tags>` directive at the beginning of each generated Go file.
453459
- `json_tags_case_style`:
@@ -456,6 +462,8 @@ Each mapping in the `packages` collection has the following keys:
456462
- If `true`, sqlc won't generate table and enum structs that aren't used in queries for a given package. Defaults to `false`.
457463
- `output_batch_file_name`:
458464
- Customize the name of the batch file. Defaults to `batch.go`.
465+
- `output_query_batch_file_name`:
466+
- Customize the name of the query batch file. Defaults to `query_batch.sql.go`.
459467
- `output_db_file_name`:
460468
- Customize the name of the db file. Defaults to `db.go`.
461469
- `output_models_file_name`:

docs/reference/query-annotations.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,74 @@ func (b *CreateBookBatchResults) Close() error {
223223
}
224224
```
225225

226+
## `emit_query_batch` (batching different queries)
227+
228+
The `:batchexec`, `:batchmany`, and `:batchone` annotations above batch the
229+
**same query** with different parameters. If you need to batch **different
230+
queries** into a single round-trip, use the `emit_query_batch` configuration
231+
option instead.
232+
233+
When `emit_query_batch` is enabled, sqlc generates a `QueryBatch` type that
234+
uses pgx v5's `QueuedQuery` callback API. Each regular query (`:one`, `:many`,
235+
`:exec`, `:execrows`, `:execresult`) gets a `Queue*` method on `QueryBatch`.
236+
All queued queries are sent in a single round-trip when `ExecuteBatch` is
237+
called.
238+
239+
__NOTE: This option only works with PostgreSQL using the `pgx/v5` driver and outputting Go code.__
240+
241+
```yaml
242+
# sqlc.yaml
243+
version: "2"
244+
sql:
245+
- engine: "postgresql"
246+
schema: "schema.sql"
247+
queries: "query.sql"
248+
gen:
249+
go:
250+
package: "db"
251+
out: "db"
252+
sql_package: "pgx/v5"
253+
emit_query_batch: true
254+
```
255+
256+
```sql
257+
-- name: GetUser :one
258+
SELECT * FROM users WHERE id = $1;
259+
260+
-- name: ListUsers :many
261+
SELECT * FROM users ORDER BY id;
262+
263+
-- name: UpdateUser :exec
264+
UPDATE users SET name = $1 WHERE id = $2;
265+
```
266+
267+
```go
268+
// Generated QueryBatch API:
269+
batch := db.NewQueryBatch()
270+
271+
batch.QueueGetUser(userID, func(user db.User, found bool) error {
272+
if !found {
273+
return nil // no row matched
274+
}
275+
fmt.Println(user.Name)
276+
return nil
277+
})
278+
279+
batch.QueueListUsers(func(users []db.User) error {
280+
fmt.Println("found", len(users), "users")
281+
return nil
282+
})
283+
284+
batch.QueueUpdateUser(db.UpdateUserParams{Name: "Alice", ID: 1})
285+
286+
// Send all queries in one round-trip:
287+
err := queries.ExecuteBatch(ctx, batch)
288+
```
289+
290+
The `QueryBatch.Batch` field is exported so you can mix generated `Queue*`
291+
calls with custom pgx batch operations on the same `pgx.Batch`. This feature
292+
can be used alongside `:batch*` annotations in the same package.
293+
226294
## `:copyfrom`
227295

228296
__NOTE: This command is driver and package specific, see [how to insert](../howto/insert.md#using-copyfrom)

internal/codegen/golang/gen.go

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ type tmplCtx struct {
3939
EmitAllEnumValues bool
4040
UsesCopyFrom bool
4141
UsesBatch bool
42+
EmitQueryBatch bool
4243
OmitSqlcVersion bool
4344
BuildTags string
4445
WrapErrors bool
@@ -182,7 +183,8 @@ func generate(req *plugin.GenerateRequest, options *opts.Options, enums []Enum,
182183
EmitEnumValidMethod: options.EmitEnumValidMethod,
183184
EmitAllEnumValues: options.EmitAllEnumValues,
184185
UsesCopyFrom: usesCopyFrom(queries),
185-
UsesBatch: usesBatch(queries),
186+
UsesBatch: usesBatch(queries) || options.EmitQueryBatch,
187+
EmitQueryBatch: options.EmitQueryBatch,
186188
SQLDriver: parseDriver(options.SqlPackage),
187189
Q: "`",
188190
Package: options.Package,
@@ -205,10 +207,14 @@ func generate(req *plugin.GenerateRequest, options *opts.Options, enums []Enum,
205207
tctx.SQLDriver = opts.SQLDriverGoSQLDriverMySQL
206208
}
207209

208-
if tctx.UsesBatch && !tctx.SQLDriver.IsPGX() {
210+
if usesBatch(queries) && !tctx.SQLDriver.IsPGX() {
209211
return nil, errors.New(":batch* commands are only supported by pgx")
210212
}
211213

214+
if options.EmitQueryBatch && tctx.SQLDriver != opts.SQLDriverPGXV5 {
215+
return nil, errors.New("emit_query_batch is only supported by pgx/v5")
216+
}
217+
212218
funcMap := template.FuncMap{
213219
"lowerTitle": sdk.LowerTitle,
214220
"comment": sdk.DoubleSlashComment,
@@ -289,6 +295,11 @@ func generate(req *plugin.GenerateRequest, options *opts.Options, enums []Enum,
289295
batchFileName = options.OutputBatchFileName
290296
}
291297

298+
queryBatchFileName := "query_batch.sql.go"
299+
if options.OutputQueryBatchFileName != "" {
300+
queryBatchFileName = options.OutputQueryBatchFileName
301+
}
302+
292303
if err := execute(dbFileName, "dbFile"); err != nil {
293304
return nil, err
294305
}
@@ -305,11 +316,16 @@ func generate(req *plugin.GenerateRequest, options *opts.Options, enums []Enum,
305316
return nil, err
306317
}
307318
}
308-
if tctx.UsesBatch {
319+
if usesBatch(queries) {
309320
if err := execute(batchFileName, "batchFile"); err != nil {
310321
return nil, err
311322
}
312323
}
324+
if tctx.EmitQueryBatch {
325+
if err := execute(queryBatchFileName, "queryBatchFile"); err != nil {
326+
return nil, err
327+
}
328+
}
313329

314330
files := map[string]struct{}{}
315331
for _, gq := range queries {

internal/codegen/golang/imports.go

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,10 @@ func (i *importer) Imports(filename string) [][]ImportSpec {
101101
if i.Options.OutputBatchFileName != "" {
102102
batchFileName = i.Options.OutputBatchFileName
103103
}
104+
queryBatchFileName := "query_batch.sql.go"
105+
if i.Options.OutputQueryBatchFileName != "" {
106+
queryBatchFileName = i.Options.OutputQueryBatchFileName
107+
}
104108

105109
switch filename {
106110
case dbFileName:
@@ -113,6 +117,8 @@ func (i *importer) Imports(filename string) [][]ImportSpec {
113117
return mergeImports(i.copyfromImports())
114118
case batchFileName:
115119
return mergeImports(i.batchImports())
120+
case queryBatchFileName:
121+
return mergeImports(i.queryBatchImports())
116122
default:
117123
return mergeImports(i.queryImports(filename))
118124
}
@@ -506,6 +512,65 @@ func hasPrefixIgnoringSliceAndPointerPrefix(s, prefix string) bool {
506512
return strings.HasPrefix(trimmedS, trimmedPrefix)
507513
}
508514

515+
func (i *importer) queryBatchImports() fileImports {
516+
// Filter to only non-batch, non-copyfrom queries
517+
regularQueries := make([]Query, 0, len(i.Queries))
518+
for _, q := range i.Queries {
519+
if q.Cmd != metadata.CmdCopyFrom && !usesBatch([]Query{q}) {
520+
regularQueries = append(regularQueries, q)
521+
}
522+
}
523+
std, pkg := buildImports(i.Options, regularQueries, queryBatchUsesType(regularQueries))
524+
525+
for _, q := range regularQueries {
526+
switch q.Cmd {
527+
case metadata.CmdOne:
528+
// :one queries use errors.Is for pgx.ErrNoRows check
529+
std["errors"] = struct{}{}
530+
case metadata.CmdExecRows, metadata.CmdExecResult:
531+
// Exec queries need pgconn.CommandTag to handle results.
532+
// metadata.CmdExecLastId is unsupported in Postgres.
533+
pkg[ImportSpec{Path: "github.com/jackc/pgx/v5/pgconn"}] = struct{}{}
534+
}
535+
}
536+
537+
// context is always needed for ExecuteBatch
538+
std["context"] = struct{}{}
539+
// pgx/v5 is always needed for pgx.Batch and pgx.Rows
540+
pkg[ImportSpec{Path: "github.com/jackc/pgx/v5"}] = struct{}{}
541+
return sortedImports(std, pkg)
542+
}
543+
544+
// queryBatchUsesType returns a predicate that checks whether a type name is
545+
// directly referenced in the generated query batch file. This skips struct
546+
// field types because struct definitions live in query.sql.go, not
547+
// query_batch.sql.go. The batch file only references structs by name.
548+
func queryBatchUsesType(queries []Query) func(string) bool {
549+
return func(name string) bool {
550+
for _, q := range queries {
551+
if q.hasRetType() {
552+
// Only check non-struct return types. Struct definitions
553+
// live in the query file, not the batch file.
554+
if !q.Ret.EmitStruct() {
555+
if hasPrefixIgnoringSliceAndPointerPrefix(q.Ret.Type(), name) {
556+
return true
557+
}
558+
}
559+
}
560+
// Only check non-struct arg types. Struct args appear as
561+
// the struct name in the function signature, not field types.
562+
if !q.Arg.EmitStruct() {
563+
for _, f := range q.Arg.Pairs() {
564+
if hasPrefixIgnoringSliceAndPointerPrefix(f.Type, name) {
565+
return true
566+
}
567+
}
568+
}
569+
}
570+
return false
571+
}
572+
}
573+
509574
func replaceConflictedArg(imports [][]ImportSpec, queries []Query) []Query {
510575
m := make(map[string]struct{})
511576
for _, is := range imports {

internal/codegen/golang/opts/options.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ type Options struct {
4545
OmitUnusedStructs bool `json:"omit_unused_structs,omitempty" yaml:"omit_unused_structs"`
4646
BuildTags string `json:"build_tags,omitempty" yaml:"build_tags"`
4747
Initialisms *[]string `json:"initialisms,omitempty" yaml:"initialisms"`
48+
EmitQueryBatch bool `json:"emit_query_batch,omitempty" yaml:"emit_query_batch"`
49+
OutputQueryBatchFileName string `json:"output_query_batch_file_name,omitempty" yaml:"output_query_batch_file_name"`
4850

4951
InitialismsMap map[string]struct{} `json:"-" yaml:"-"`
5052
}
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
{{define "queryBatchCodePgx"}}
2+
3+
// QueryBatch allows queuing multiple queries to be executed in a single
4+
// round-trip using pgx v5's QueuedQuery callback pattern. Each Queue* method
5+
// calls pgx.Batch.Queue and registers a result callback (QueryRow, Query, or
6+
// Exec) that is invoked when ExecuteBatch processes the batch results.
7+
// For :exec queries, no callback is needed - errors propagate via ExecuteBatch.
8+
//
9+
// The Batch field is exported to allow interoperability: callers can mix
10+
// generated Queue* calls with custom pgx batch operations on the same
11+
// underlying pgx.Batch.
12+
type QueryBatch struct {
13+
Batch *pgx.Batch
14+
}
15+
16+
// NewQueryBatch creates a new QueryBatch.
17+
func NewQueryBatch() *QueryBatch {
18+
return &QueryBatch{
19+
Batch: &pgx.Batch{},
20+
}
21+
}
22+
23+
// ExecuteBatch sends all queued queries and closes the batch.
24+
func (q *Queries) ExecuteBatch(ctx context.Context, {{if $.EmitMethodsWithDBArgument}}db DBTX, {{end}}batch *QueryBatch) error {
25+
return {{if $.EmitMethodsWithDBArgument}}db{{else}}q.db{{end}}.SendBatch(ctx, batch.Batch).Close()
26+
}
27+
28+
{{range .GoQueries}}
29+
{{if and (ne .Cmd ":copyfrom") (ne (hasPrefix .Cmd ":batch") true)}}
30+
{{if eq .Cmd ":one"}}
31+
// Queue{{.MethodName}} queues {{.MethodName}} for batch execution.
32+
// The callback fn is called when ExecuteBatch is called. The second parameter
33+
// is false if the row was not found (no error is returned in this case).
34+
func (b *QueryBatch) Queue{{.MethodName}}({{.Arg.Pair}}{{if .Arg.Pair}}, {{end}}fn func({{.Ret.DefineType}}, bool) error) {
35+
b.Batch.Queue({{.ConstantName}}, {{.Arg.Params}}).QueryRow(func(row pgx.Row) error {
36+
var {{.Ret.Name}} {{.Ret.Type}}
37+
err := row.Scan({{.Ret.Scan}})
38+
if err != nil {
39+
if errors.Is(err, pgx.ErrNoRows) {
40+
return fn({{.Ret.ReturnName}}, false)
41+
}
42+
return err
43+
}
44+
return fn({{.Ret.ReturnName}}, true)
45+
})
46+
}
47+
{{end}}
48+
49+
{{if eq .Cmd ":many"}}
50+
// Queue{{.MethodName}} queues {{.MethodName}} for batch execution.
51+
// The callback fn is called with the results when ExecuteBatch is called.
52+
func (b *QueryBatch) Queue{{.MethodName}}({{.Arg.Pair}}{{if .Arg.Pair}}, {{end}}fn func([]{{.Ret.DefineType}}) error) {
53+
b.Batch.Queue({{.ConstantName}}, {{.Arg.Params}}).Query(func(rows pgx.Rows) error {
54+
defer rows.Close()
55+
{{- if $.EmitEmptySlices}}
56+
items := []{{.Ret.DefineType}}{}
57+
{{else}}
58+
var items []{{.Ret.DefineType}}
59+
{{end -}}
60+
for rows.Next() {
61+
var {{.Ret.Name}} {{.Ret.Type}}
62+
if err := rows.Scan({{.Ret.Scan}}); err != nil {
63+
return err
64+
}
65+
items = append(items, {{.Ret.ReturnName}})
66+
}
67+
if err := rows.Err(); err != nil {
68+
return err
69+
}
70+
return fn(items)
71+
})
72+
}
73+
{{end}}
74+
75+
{{if eq .Cmd ":exec"}}
76+
// Queue{{.MethodName}} queues {{.MethodName}} for batch execution.
77+
func (b *QueryBatch) Queue{{.MethodName}}({{.Arg.Pair}}) {
78+
b.Batch.Queue({{.ConstantName}}, {{.Arg.Params}})
79+
}
80+
{{end}}
81+
82+
{{if eq .Cmd ":execrows"}}
83+
// Queue{{.MethodName}} queues {{.MethodName}} for batch execution.
84+
// The callback fn is called with the number of rows affected when ExecuteBatch is called.
85+
func (b *QueryBatch) Queue{{.MethodName}}({{.Arg.Pair}}{{if .Arg.Pair}}, {{end}}fn func(int64) error) {
86+
b.Batch.Queue({{.ConstantName}}, {{.Arg.Params}}).Exec(func(ct pgconn.CommandTag) error {
87+
return fn(ct.RowsAffected())
88+
})
89+
}
90+
{{end}}
91+
92+
{{if eq .Cmd ":execresult"}}
93+
// Queue{{.MethodName}} queues {{.MethodName}} for batch execution.
94+
// The callback fn is called with the command tag when ExecuteBatch is called.
95+
func (b *QueryBatch) Queue{{.MethodName}}({{.Arg.Pair}}{{if .Arg.Pair}}, {{end}}fn func(pgconn.CommandTag) error) {
96+
b.Batch.Queue({{.ConstantName}}, {{.Arg.Params}}).Exec(func(ct pgconn.CommandTag) error {
97+
return fn(ct)
98+
})
99+
}
100+
{{end}}
101+
{{end}}
102+
{{end}}
103+
{{end}}

0 commit comments

Comments
 (0)