Skip to content

engine-plugins.md

Latest commit

 

History

History
175 lines (122 loc) · 5.91 KB

engine-plugins.md

File metadata and controls

175 lines (122 loc) · 5.91 KB

External Database Engines (Engine Plugins)

Engine plugins let you use sqlc with databases that are not built-in. You can add support for other SQL-compatible systems (e.g. CockroachDB, TiDB, or custom engines) by implementing a small external program that parses SQL and returns parameters and result columns.

Why use an engine plugin?

  • Use sqlc with a database that doesn’t have native support.
  • Reuse an existing SQL parser or dialect in a separate binary.
  • Keep engine-specific logic outside the sqlc core.

Data returned by the engine plugin (SQL text, parameters, columns) is passed through to codegen plugins without an extra compiler/AST step. The plugin is the single place that defines how queries are interpreted for that engine.

Overview

An engine plugin is an external process that implements one RPC:

  • Parse — accepts the query text and either schema SQL or connection parameters, and returns processed SQL, parameter list, and result columns.

Process plugins (e.g. written in Go) talk to sqlc over stdin/stdout using Protocol Buffers. The schema is defined in engine/engine.proto.

Compatibility

For Go plugins, compatibility is enforced at compile time by importing the engine package:

import "github.com/sqlc-dev/sqlc/pkg/engine"
  • If the plugin builds, it matches this version of the engine API.
  • If the API changes in a breaking way, the plugin stops compiling until it’s updated.

No version handshake is required; the proto schema defines the contract.

Configuration

sqlc.yaml

version: "2"

engines:
  - name: mydb
    process:
      cmd: sqlc-engine-mydb
    env:
      - MYDB_DSN

sql:
  - engine: mydb
    schema: "schema.sql"
    queries: "queries.sql"
    codegen:
      - plugin: go
        out: db

Engine options

Field Description
name Engine name used in sql[].engine
process.cmd Command to run (PATH or absolute path)
env Environment variables passed to the plugin

Implementing an engine plugin (Go)

1. Dependencies and entrypoint

package main

import "github.com/sqlc-dev/sqlc/pkg/engine"

func main() {
    engine.Run(engine.Handler{
        PluginName:    "mydb",
        PluginVersion: "1.0.0",
        Parse:         handleParse,
    })
}

The engine API exposes only Parse. There are no separate methods for catalog, keywords, comment syntax, or dialect.

2. Parse

Request

  • sql — The query text to parse.
  • schema_source — One of:
    • schema_sql: schema as in a schema.sql file (used for schema-based parsing).
    • connection_params: DSN and options for database-only mode.

Response

  • sql — Processed query text. Often the same as input; with a schema you may expand * into explicit columns.
  • parameters — List of parameters (position/name, type, nullable, array, etc.).
  • columns — List of result columns (name, type, nullable, table/schema if known).

Example handler:

func handleParse(req *engine.ParseRequest) (*engine.ParseResponse, error) {
    sql := req.GetSql()

    var schema *SchemaInfo
    if s := req.GetSchemaSql(); s != "" {
        schema = parseSchema(s)
    }
    // Or use req.GetConnectionParams() for database-only mode.

    parameters := extractParameters(sql)
    columns := extractColumns(sql, schema)
    processedSQL := processSQL(sql, schema) // e.g. expand SELECT *

    return &engine.ParseResponse{
        Sql:        processedSQL,
        Parameters: parameters,
        Columns:    columns,
    }, nil
}

Parameter and column types use the Parameter and Column messages in engine.proto (name, position, data_type, nullable, is_array, array_dims; for columns, table_name and schema_name are optional).

Support for sqlc placeholders (sqlc.arg(), sqlc.narg(), sqlc.slice(), sqlc.embed()) is up to the plugin: it can parse and map them into parameters (and schema usage) as needed.

3. Build and run

go build -o sqlc-engine-mydb .
# Ensure sqlc-engine-mydb is on PATH or use an absolute path in process.cmd

Protocol

Process plugins use Protocol Buffers on stdin/stdout:

sqlc  →  stdin (protobuf)  →  plugin  →  stdout (protobuf)  →  sqlc

Invocation:

sqlc-engine-mydb parse   # stdin: ParseRequest, stdout: ParseResponse

The definition lives in engine/engine.proto (and generated Go in pkg/engine).

Example

A minimal engine that parses SQLite-style SQL and expands * using a schema is in this repository under examples/plugin-based-codegen/plugins/sqlc-engine-sqlite3/. It pairs with the Rust codegen example in the same plugin-based-codegen sample.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  sqlc generate                                                  │
│  1. Read sqlc.yaml, find engine for this sql block              │
│  2. Call plugin: parse (sql + schema_sql or connection_params)   │
│  3. Use returned sql, parameters, columns in codegen             │
└─────────────────────────────────────────────────────────────────┘

    sqlc                          sqlc-engine-mydb
      │──── spawn, args: ["parse"] ──────────────────────────────► │
      │──── stdin: ParseRequest{sql, schema_sql|connection_params} ► │
      │◄─── stdout: ParseResponse{sql, parameters, columns} ─────── │

See also