|
1 | | -# pg-schema-diff |
2 | | -[](https://github.com/stripe/pg-schema-diff/actions/workflows/run-tests.yaml) |
3 | | -[](https://goreportcard.com/report/github.com/stripe/pg-schema-diff) |
4 | | -[](https://pkg.go.dev/github.com/stripe/pg-schema-diff) |
5 | | - |
| 1 | +# @pg-nano/pg-schema-diff |
6 | 2 |
|
7 | | -Computes the diff(erences) between Postgres database schemas and generates the SQL required to get your database schema from point A to B with |
8 | | -minimal downtime & locks. This enables you to take your database and migrate it to any desired schema defined in plain DDL. |
| 3 | +A cross-platform distribution of the `pg-schema-diff` binary, tailored for use with [pg-nano](https://github.com/pg-nano/pg-nano). |
9 | 4 |
|
10 | | -The tooling attempts to use native postgres migration operations to perform online migrations and avoid locking wherever possible. Not all migrations will |
11 | | -be lock-free and some might require downtime, but the hazards system will warn you ahead of time when that's the case. |
12 | | -Stateful online migration techniques, like shadow tables, aren't yet supported. |
| 5 | +## Installation |
13 | 6 |
|
14 | | -### Online index Replacement |
15 | | -Your project's diff: |
16 | | -``` |
17 | | -$ git diff |
18 | | -diff --git a/schema/schema.sql b/schema/schema.sql |
19 | | -index cc3a14b..cf4b32d 100644 |
20 | | ---- a/schema/schema.sql |
21 | | -+++ b/schema/schema.sql |
22 | | -@@ -2,5 +2,5 @@ CREATE TABLE foobar( |
23 | | - created_at timestamp, |
24 | | - message text |
25 | | - ); |
26 | | --CREATE INDEX message_idx ON foobar(message); |
27 | | -+CREATE INDEX message_idx ON foobar(message, created_at); |
28 | | -``` |
29 | | -The generated plan (*queries using `message_idx` will always have an index backing them, even while the new index is being built*): |
30 | | -``` |
31 | | -$ pg-schema-diff plan --dsn "postgres://postgres:postgres@localhost:5432/postgres" --schema-dir ./schema |
32 | | -################################ Generated plan ################################ |
33 | | -1. ALTER INDEX "message_idx" RENAME TO "pgschemadiff_tmpidx_message_idx_IiaKzkvPQtyA7ob9piVqiQ"; |
34 | | - -- Statement Timeout: 3s |
35 | | -
|
36 | | -2. CREATE INDEX CONCURRENTLY message_idx ON public.foobar USING btree (message, created_at); |
37 | | - -- Statement Timeout: 20m0s |
38 | | - -- Lock Timeout: 3s |
39 | | - -- Hazard INDEX_BUILD: This might affect database performance. Concurrent index builds require a non-trivial amount of CPU, potentially affecting database performance. They also can take a while but do not lock out writes. |
40 | | -
|
41 | | -3. DROP INDEX CONCURRENTLY "pgschemadiff_tmpidx_message_idx_IiaKzkvPQtyA7ob9piVqiQ"; |
42 | | - -- Statement Timeout: 20m0s |
43 | | - -- Lock Timeout: 3s |
44 | | - -- Hazard INDEX_DROPPED: Dropping this index means queries that use this index might perform worse because they will no longer will be able to leverage it. |
45 | | -``` |
46 | | -### Online `NOT NULL` constraint creation |
47 | | -Your project's diff: |
48 | | -``` |
49 | | -diff --git a/schema/schema.sql b/schema/schema.sql |
50 | | -index cc3a14b..5a1cec2 100644 |
51 | | ---- a/schema/schema.sql |
52 | | -+++ b/schema/schema.sql |
53 | | -@@ -1,5 +1,5 @@ |
54 | | - CREATE TABLE foobar( |
55 | | -- created_at timestamp, |
56 | | -+ created_at timestamp NOT NULL, |
57 | | - message text |
58 | | - ); |
59 | | - CREATE INDEX message_idx ON foobar(message); |
60 | | -``` |
61 | | -The generated plan (*leverages check constraints to eliminate the need for a long-lived access-exclusive lock on the table*): |
62 | | -``` |
63 | | -$ pg-schema-diff plan --dsn "postgres://postgres:postgres@localhost:5432/postgres" --schema-dir ./schema |
64 | | -################################ Generated plan ################################ |
65 | | -1. ALTER TABLE "public"."foobar" ADD CONSTRAINT "pgschemadiff_tmpnn_BCOxMXqAQwaXlKPCRXoMMg" CHECK("created_at" IS NOT NULL) NOT VALID; |
66 | | - -- Statement Timeout: 3s |
67 | | -
|
68 | | -2. ALTER TABLE "public"."foobar" VALIDATE CONSTRAINT "pgschemadiff_tmpnn_BCOxMXqAQwaXlKPCRXoMMg"; |
69 | | - -- Statement Timeout: 3s |
70 | | -
|
71 | | -3. ALTER TABLE "public"."foobar" ALTER COLUMN "created_at" SET NOT NULL; |
72 | | - -- Statement Timeout: 3s |
73 | | -
|
74 | | -4. ALTER TABLE "public"."foobar" DROP CONSTRAINT "pgschemadiff_tmpnn_BCOxMXqAQwaXlKPCRXoMMg"; |
75 | | - -- Statement Timeout: 3s |
76 | | -``` |
77 | | - |
78 | | -# Key features |
79 | | -* Declarative schema migrations |
80 | | -* The use of postgres native operations for zero-downtime migrations wherever possible: |
81 | | - * Concurrent index builds |
82 | | - * Online index replacement: If some index is changed, the new version will be built before the old version is dropped, preventing a window where no index is backing queries |
83 | | - * Online constraint builds: Constraints (check, foreign key) are added as `INVALID` before being validated, eliminating the need |
84 | | - for a long access-exclusive lock on the table |
85 | | - * Online `NOT NULL` constraint creation using check constraints to eliminate the need for an access-exclusive lock on the table |
86 | | - * Prioritized index builds: Building new indexes is always prioritized over deleting old indexes |
87 | | -* A comprehensive set of features to ensure the safety of planned migrations: |
88 | | - * Operators warned of dangerous operations. |
89 | | - * Migration plans are validated first against a temporary database exactly as they would be performed against the real database. |
90 | | -* Strong support of partitions |
91 | | -# Install |
92 | | -## CLI |
93 | 7 | ```bash |
94 | | -go install github.com/stripe/pg-schema-diff/cmd/pg-schema-diff |
| 8 | +pnpm add @pg-nano/pg-schema-diff |
95 | 9 | ``` |
96 | 10 |
|
97 | | -## Library |
98 | | -```bash |
99 | | -go get -u github.com/stripe/pg-schema-diff |
100 | | -``` |
101 | | -# Using CLI |
102 | | -## 1. Apply schema to fresh database |
103 | | -Create a directory to hold your schema files. Then, generate sql files and place them into a schema dir. |
104 | | -```bash |
105 | | -mkdir schema |
106 | | -echo "CREATE TABLE foobar (id int);" > schema/foobar.sql |
107 | | -echo "CREATE TABLE bar (id varchar(255), message TEXT NOT NULL);" > schema/bar.sql |
108 | | -``` |
| 11 | +## Usage |
109 | 12 |
|
110 | | -Apply the schema to a fresh database. [The connection string spec can be found here](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING). |
111 | | -Setting the `PGPASSWORD` env var will override any password set in the connection string and is recommended. |
112 | 13 | ```bash |
113 | | -pg-schema-diff apply --dsn "postgres://postgres:postgres@localhost:5432/postgres" --schema-dir schema |
| 14 | +pnpm pg-schema-diff --help |
114 | 15 | ``` |
115 | 16 |
|
116 | | -## 2. Updating schema |
117 | | -Update the SQL file(s) |
118 | | -```bash |
119 | | -echo "CREATE INDEX message_idx ON bar(message)" >> schema/bar.sql |
120 | | -``` |
121 | | - |
122 | | -Apply the schema. Any hazards in the generated plan must be approved |
123 | | -```bash |
124 | | -pg-schema-diff apply --dsn "postgres://postgres:postgres@localhost:5432/postgres" --schema-dir schema --allow-hazards INDEX_BUILD |
125 | | -``` |
126 | | - |
127 | | -# Using Library |
128 | | -Docs to use the library can be found [here](https://pkg.go.dev/github.com/stripe/pg-schema-diff). Check out [the CLI](https://github.com/stripe/pg-schema-diff/tree/main/cmd/pg-schema-diff) |
129 | | -for an example implementation with the library |
130 | | - |
131 | | -## 1. Generating plan |
132 | | -```go |
133 | | -// The tempDbFactory is used in plan generation to extract the new schema and validate the plan |
134 | | -tempDbFactory, err := tempdb.NewOnInstanceFactory(ctx, func(ctx context.Context, dbName string) (*sql.DB, error) { |
135 | | - copiedConfig := connConfig.Copy() |
136 | | - copiedConfig.Database = dbName |
137 | | - return openDbWithPgxConfig(copiedConfig) |
138 | | -}) |
139 | | -if err != nil { |
140 | | - panic("Generating the TempDbFactory failed") |
141 | | -} |
142 | | -defer tempDbFactory.Close() |
143 | | -// Generate the migration plan |
144 | | -plan, err := diff.Generate(ctx, connPool, diff.DDLSchemaSource(ddl), |
145 | | - diff.WithTempDbFactory(tempDbFactory), |
146 | | - diff.WithDataPackNewTables(), |
147 | | -) |
148 | | -if err != nil { |
149 | | - panic("Generating the plan failed") |
150 | | -} |
151 | | -``` |
| 17 | +You also have the option of spawning a child process to run the command: |
152 | 18 |
|
153 | | -## 2. Applying plan |
154 | | -We leave plan application up to the user. For example, you might want to take out a session-level advisory lock if you are |
155 | | -concerned about concurrent migrations on your database. You might also want a second user to approve the plan |
156 | | -before applying it. |
| 19 | +```js |
| 20 | +import spawn from "tinyspawn"; |
157 | 21 |
|
158 | | -Example apply: |
159 | | -```go |
160 | | -for _, stmt := range plan.Statements { |
161 | | - if _, err := conn.ExecContext(ctx, fmt.Sprintf("SET SESSION statement_timeout = %d", stmt.Timeout.Milliseconds())); err != nil { |
162 | | - panic(fmt.Sprintf("setting statement timeout: %s", err)) |
163 | | - } |
164 | | - if _, err := conn.ExecContext(ctx, fmt.Sprintf("SET SESSION lock_timeout = %d", stmt.LockTimeout.Milliseconds())); err != nil { |
165 | | - panic(fmt.Sprintf("setting lock timeout: %s", err)) |
166 | | - } |
167 | | - if _, err := conn.ExecContext(ctx, stmt.ToSQL()); err != nil { |
168 | | - panic(fmt.Sprintf("executing migration statement. the database maybe be in a dirty state: %s: %s", stmt, err)) |
169 | | - } |
170 | | -} |
| 22 | +await spawn("./node_modules/.bin/pg-schema-diff --help", { |
| 23 | + stdio: "inherit", |
| 24 | +}); |
171 | 25 | ``` |
172 | 26 |
|
173 | | -# Supported Postgres versions |
174 | | -Supported: 14, 15, 16 |
175 | | -Unsupported: <= 13 are not supported. Use at your own risk. |
| 27 | +## Thanks |
176 | 28 |
|
177 | | -# Unsupported migrations |
178 | | -An abridged list of unsupported migrations: |
179 | | -- Views (Planned) |
180 | | -- Privileges (Planned) |
181 | | -- Types (Only enums are currently supported) |
182 | | -- Renaming. The diffing library relies on names to identify the old and new versions of a table, index, etc. If you rename |
183 | | -an object, it will be treated as a drop and an add |
| 29 | +This package wouldn't exist without these tools: |
184 | 30 |
|
185 | | -# Contributing |
186 | | -This project is in its early stages. We appreciate all the feature/bug requests we receive, but we have limited cycles |
187 | | -to review direct code contributions at this time. See [Contributing](CONTRIBUTING.md) to learn more. |
| 31 | +- [stripe/pg-schema-diff](https://github.com/stripe/pg-schema-diff) |
| 32 | +- [goreleaser](https://github.com/goreleaser/goreleaser) |
| 33 | +- [prantlf/grab-github-release](https://github.com/prantlf/grab-github-release) |
0 commit comments