Add database engine plugins (external engines)

.gitignore

@@ -9,3 +9,4 @@ __pycache__
 .devenv*
 devenv.local.nix
 
+/bin/sqlc

README.md

@@ -29,6 +29,14 @@ Check out [an interactive example](https://play.sqlc.dev/) to see it in action,
 
 Additional languages can be added via [plugins](https://docs.sqlc.dev/en/latest/reference/language-support.html#community-language-support).
 
+## Supported database engines
+
+- PostgreSQL
+- MySQL
+- SQLite
+
+Additional database engines can be added via [engine plugins](https://docs.sqlc.dev/en/latest/howto/engine-plugins.html).
+
 ## Sponsors
 
 Development is possible thanks to our sponsors. If you would like to support sqlc,

docs/howto/engine-plugins.md

@@ -0,0 +1,175 @@
+# External Database Engines (Engine Plugins)
+
+Engine plugins let you use sqlc with databases that are not built-in. You can add support for other SQL-compatible systems (e.g. CockroachDB, TiDB, or custom engines) by implementing a small external program that parses SQL and returns parameters and result columns.
+
+## Why use an engine plugin?
+
+- Use sqlc with a database that doesn’t have native support.
+- Reuse an existing SQL parser or dialect in a separate binary.
+- Keep engine-specific logic outside the sqlc core.
+
+Data returned by the engine plugin (SQL text, parameters, columns) is passed through to [codegen plugins](../guides/plugins.md) without an extra compiler/AST step. The plugin is the single place that defines how queries are interpreted for that engine.
+
+## Overview
+
+An engine plugin is an external process that implements one RPC:
+
+- **Parse** — accepts the query text and either schema SQL or connection parameters, and returns processed SQL, parameter list, and result columns.
+
+Process plugins (e.g. written in Go) talk to sqlc over **stdin/stdout** using **Protocol Buffers**. The schema is defined in `engine/engine.proto`.
+
+## Compatibility
+
+For Go plugins, compatibility is enforced at **compile time** by importing the engine package:
+
+```go
+import "github.com/sqlc-dev/sqlc/pkg/engine"
+```
+
+- If the plugin builds, it matches this version of the engine API.
+- If the API changes in a breaking way, the plugin stops compiling until it’s updated.
+
+No version handshake is required; the proto schema defines the contract.
+
+## Configuration
+
+### sqlc.yaml
+
+```yaml
+version: "2"
+
+engines:
+  - name: mydb
+    process:
+      cmd: sqlc-engine-mydb
+    env:
+      - MYDB_DSN
+
+sql:
+  - engine: mydb
+    schema: "schema.sql"
+    queries: "queries.sql"
+    codegen:
+      - plugin: go
+        out: db
+```
+
+### Engine options
+
+| Field | Description |
+|-------|-------------|
+| `name` | Engine name used in `sql[].engine` |
+| `process.cmd` | Command to run (PATH or absolute path) |
+| `env` | Environment variables passed to the plugin |
+
+## Implementing an engine plugin (Go)
+
+### 1. Dependencies and entrypoint
+
+```go
+package main
+
+import "github.com/sqlc-dev/sqlc/pkg/engine"
+
+func main() {
+    engine.Run(engine.Handler{
+        PluginName:    "mydb",
+        PluginVersion: "1.0.0",
+        Parse:         handleParse,
+    })
+}
+```
+
+The engine API exposes only **Parse**. There are no separate methods for catalog, keywords, comment syntax, or dialect.
+
+### 2. Parse
+
+**Request**
+
+- `sql` — The query text to parse.
+- `schema_source` — One of:
+  - `schema_sql`: schema as in a schema.sql file (used for schema-based parsing).
+  - `connection_params`: DSN and options for database-only mode.
+
+**Response**
+
+- `sql` — Processed query text. Often the same as input; with a schema you may expand `*` into explicit columns.
+- `parameters` — List of parameters (position/name, type, nullable, array, etc.).
+- `columns` — List of result columns (name, type, nullable, table/schema if known).
+
+Example handler:
+
+```go
+func handleParse(req *engine.ParseRequest) (*engine.ParseResponse, error) {
+    sql := req.GetSql()
+
+    var schema *SchemaInfo
+    if s := req.GetSchemaSql(); s != "" {
+        schema = parseSchema(s)
+    }
+    // Or use req.GetConnectionParams() for database-only mode.
+
+    parameters := extractParameters(sql)
+    columns := extractColumns(sql, schema)
+    processedSQL := processSQL(sql, schema) // e.g. expand SELECT *
+
+    return &engine.ParseResponse{
+        Sql:        processedSQL,
+        Parameters: parameters,
+        Columns:    columns,
+    }, nil
+}
+```
+
+Parameter and column types use the `Parameter` and `Column` messages in `engine.proto` (name, position, data_type, nullable, is_array, array_dims; for columns, table_name and schema_name are optional).
+
+Support for sqlc placeholders (`sqlc.arg()`, `sqlc.narg()`, `sqlc.slice()`, `sqlc.embed()`) is up to the plugin: it can parse and map them into `parameters` (and schema usage) as needed.
+
+### 3. Build and run
+
+```bash
+go build -o sqlc-engine-mydb .
+# Ensure sqlc-engine-mydb is on PATH or use an absolute path in process.cmd
+```
+
+## Protocol
+
+Process plugins use Protocol Buffers on stdin/stdout:
+
+```
+sqlc  →  stdin (protobuf)  →  plugin  →  stdout (protobuf)  →  sqlc
+```
+
+Invocation:
+
+```bash
+sqlc-engine-mydb parse   # stdin: ParseRequest, stdout: ParseResponse
+```
+
+The definition lives in `engine/engine.proto` (and generated Go in `pkg/engine`).
+
+## Example
+
+A minimal engine that parses SQLite-style SQL and expands `*` using a schema is in this repository under `examples/plugin-based-codegen/plugins/sqlc-engine-sqlite3/`. It pairs with the Rust codegen example in the same `plugin-based-codegen` sample.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│  sqlc generate                                                  │
+│  1. Read sqlc.yaml, find engine for this sql block              │
+│  2. Call plugin: parse (sql + schema_sql or connection_params)   │
+│  3. Use returned sql, parameters, columns in codegen             │
+└─────────────────────────────────────────────────────────────────┘
+
+    sqlc                          sqlc-engine-mydb
+      │──── spawn, args: ["parse"] ──────────────────────────────► │
+      │──── stdin: ParseRequest{sql, schema_sql|connection_params} ► │
+      │◄─── stdout: ParseResponse{sql, parameters, columns} ─────── │
+```
+
+## See also
+
+- [Codegen plugins](../guides/plugins.md) — Custom code generators that consume engine output.
+- [Configuration reference](../reference/config.md)
+- Proto schema: `protos/engine/engine.proto`

docs/index.rst

@@ -66,6 +66,7 @@ code ever again.
    howto/embedding.md
    howto/overrides.md
    howto/rename.md
+   howto/engine-plugins.md
 
 .. toctree::
    :maxdepth: 3

internal/cmd/generate.go

@@ -350,6 +350,7 @@ func codegen(ctx context.Context, combo config.CombinedSettings, sql OutputPair,
 		case plug.Process != nil:
 			handler = &process.Runner{
 				Cmd:    plug.Process.Cmd,
+				Dir:    combo.Dir,
 				Env:    plug.Env,
 				Format: plug.Process.Format,
 			}

internal/cmd/process.go

@@ -69,6 +69,9 @@ func processQuerySets(ctx context.Context, rp ResultProcessor, conf *config.Conf
 
 		grp.Go(func() error {
 			combo := config.Combine(*conf, sql.SQL)
+			if dir != "" {
+				combo.Dir = dir
+			}
 			if sql.Plugin != nil {
 				combo.Codegen = *sql.Plugin
 			}

internal/cmd/vet.go

@@ -465,6 +465,9 @@ func (c *checker) DSN(dsn string) (string, error) {
 func (c *checker) checkSQL(ctx context.Context, s config.SQL) error {
 	// TODO: Create a separate function for this logic so we can
 	combo := config.Combine(*c.Conf, s)
+	if c.Dir != "" {
+		combo.Dir = c.Dir
+	}
 
 	// TODO: This feels like a hack that will bite us later
 	joined := make([]string, 0, len(s.Schema))

internal/compiler/engine.go

@@ -7,7 +7,9 @@ import (
 	"github.com/sqlc-dev/sqlc/internal/analyzer"
 	"github.com/sqlc-dev/sqlc/internal/config"
 	"github.com/sqlc-dev/sqlc/internal/dbmanager"
+	"github.com/sqlc-dev/sqlc/internal/engine"
 	"github.com/sqlc-dev/sqlc/internal/engine/dolphin"
+	"github.com/sqlc-dev/sqlc/internal/engine/plugin"
 	"github.com/sqlc-dev/sqlc/internal/engine/postgresql"
 	pganalyze "github.com/sqlc-dev/sqlc/internal/engine/postgresql/analyzer"
 	"github.com/sqlc-dev/sqlc/internal/engine/sqlite"
@@ -112,11 +114,49 @@ func NewCompiler(conf config.SQL, combo config.CombinedSettings, parserOpts opts
 			}
 		}
 	default:
-		return nil, fmt.Errorf("unknown engine: %s", conf.Engine)
+		// Check if this is a plugin engine
+		if enginePlugin, found := config.FindEnginePlugin(&combo.Global, string(conf.Engine)); found {
+			eng, err := createPluginEngine(enginePlugin, combo.Dir)
+			if err != nil {
+				return nil, err
+			}
+			c.parser = eng.Parser()
+			c.catalog = eng.Catalog()
+			sel := eng.Selector()
+			if sel != nil {
+				c.selector = &engineSelectorAdapter{sel}
+			} else {
+				c.selector = newDefaultSelector()
+			}
+		} else {
+			return nil, fmt.Errorf("unknown engine: %s\n\nTo use a custom database engine, add it to the 'engines' section of sqlc.yaml:\n\n  engines:\n    - name: %s\n      process:\n        cmd: sqlc-engine-%s\n\nThen install the plugin: go install github.com/example/sqlc-engine-%s@latest",
+				conf.Engine, conf.Engine, conf.Engine, conf.Engine)
+		}
 	}
 	return c, nil
 }
 
+// createPluginEngine creates an engine from an engine plugin configuration.
+func createPluginEngine(ep *config.EnginePlugin, dir string) (engine.Engine, error) {
+	switch {
+	case ep.Process != nil:
+		return plugin.NewPluginEngine(ep.Name, ep.Process.Cmd, dir, ep.Env), nil
+	case ep.WASM != nil:
+		return plugin.NewWASMPluginEngine(ep.Name, ep.WASM.URL, ep.WASM.SHA256, ep.Env), nil
+	default:
+		return nil, fmt.Errorf("engine plugin %s has no process or wasm configuration", ep.Name)
+	}
+}
+
+// engineSelectorAdapter adapts engine.Selector to the compiler's selector interface.
+type engineSelectorAdapter struct {
+	sel engine.Selector
+}
+
+func (a *engineSelectorAdapter) ColumnExpr(name string, column *Column) string {
+	return a.sel.ColumnExpr(name, column.DataType)
+}
+
 func (c *Compiler) Catalog() *catalog.Catalog {
 	return c.catalog
 }

internal/config/config.go

@@ -63,10 +63,43 @@ type Config struct {
 	SQL       []SQL                `json:"sql" yaml:"sql"`
 	Overrides Overrides            `json:"overrides,omitempty" yaml:"overrides"`
 	Plugins   []Plugin             `json:"plugins" yaml:"plugins"`
+	Engines   []EnginePlugin       `json:"engines" yaml:"engines"`
 	Rules     []Rule               `json:"rules" yaml:"rules"`
 	Options   map[string]yaml.Node `json:"options" yaml:"options"`
 }
 
+// EnginePlugin defines a custom database engine plugin.
+// Engine plugins allow external SQL parsers and database backends to be used with sqlc.
+type EnginePlugin struct {
+	// Name is the unique name for this engine (used in sql[].engine field)
+	Name string `json:"name" yaml:"name"`
+
+	// Env is a list of environment variable names to pass to the plugin
+	Env []string `json:"env" yaml:"env"`
+
+	// Process defines an engine plugin that runs as an external process
+	Process *EnginePluginProcess `json:"process" yaml:"process"`
+
+	// WASM defines an engine plugin that runs as a WASM module
+	WASM *EnginePluginWASM `json:"wasm" yaml:"wasm"`
+}
+
+// EnginePluginProcess defines a process-based engine plugin.
+type EnginePluginProcess struct {
+	// Cmd is the command to run (must be in PATH or an absolute path)
+	Cmd string `json:"cmd" yaml:"cmd"`
+}
+
+// EnginePluginWASM defines a WASM-based engine plugin.
+type EnginePluginWASM struct {
+	// URL is the URL to download the WASM module from
+	// Supports file:// and https:// schemes
+	URL string `json:"url" yaml:"url"`
+
+	// SHA256 is the expected SHA256 checksum of the WASM module
+	SHA256 string `json:"sha256" yaml:"sha256"`
+}
+
 type Server struct {
 	Name   string `json:"name,omitempty" yaml:"name"`
 	Engine Engine `json:"engine,omitempty" yaml:"engine"`
@@ -125,8 +158,8 @@ type SQL struct {
 // AnalyzerDatabase represents the database analyzer setting.
 // It can be a boolean (true/false) or the string "only" for database-only mode.
 type AnalyzerDatabase struct {
-	value   *bool  // nil means not set, true/false for boolean values
-	isOnly  bool   // true when set to "only"
+	value  *bool // nil means not set, true/false for boolean values
+	isOnly bool  // true when set to "only"
 }
 
 // IsEnabled returns true if the database analyzer should be used.
@@ -228,6 +261,14 @@ var ErrPluginNoType = errors.New("plugin: field `process` or `wasm` required")
 var ErrPluginBothTypes = errors.New("plugin: `process` and `wasm` cannot both be defined")
 var ErrPluginProcessNoCmd = errors.New("plugin: missing process command")
 
+var ErrEnginePluginNoName = errors.New("engine plugin: missing name")
+var ErrEnginePluginBuiltin = errors.New("engine plugin: cannot override built-in engine")
+var ErrEnginePluginExists = errors.New("engine plugin: a plugin with that name already exists")
+var ErrEnginePluginNoType = errors.New("engine plugin: field `process` or `wasm` required")
+var ErrEnginePluginBothTypes = errors.New("engine plugin: `process` and `wasm` cannot both be defined")
+var ErrEnginePluginProcessNoCmd = errors.New("engine plugin: missing process command")
+var ErrEnginePluginWASMNoURL = errors.New("engine plugin: missing wasm url")
+
 var ErrInvalidDatabase = errors.New("database must be managed or have a non-empty URI")
 var ErrManagedDatabaseNoProject = errors.New(`managed databases require a cloud project
 
@@ -285,6 +326,9 @@ type CombinedSettings struct {
 
 	// TODO: Combine these into a more usable type
 	Codegen Codegen
+
+	// Dir is the directory containing the config file (for resolving relative paths)
+	Dir string
 }
 
 func Combine(conf Config, pkg SQL) CombinedSettings {