Skip to content

Commit 7b59651

Browse files
committed
save
1 parent 8495be9 commit 7b59651

10 files changed

Lines changed: 39 additions & 38 deletions

File tree

docs/features/sharding/2pc.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ ALTER SYSTEM SET max_prepared_transactions TO 1000;
3131
Alternatively, if you're running on managed Postgres (e.g., AWS RDS), this parameter can usually be set through your cloud admin panel.
3232

3333
!!! note
34-
This parameter can only be enabled on server start. Once you change it, make sure to restart your Postgres servers.
34+
Changes to this parameter require a server restart to take effect.
3535

3636
Once prepared transactions are enabled in Postgres, two-phase commit can be enabled in [`pgdog.toml`](../../configuration/pgdog.toml/general.md):
3737

docs/features/sharding/cross-shard-queries/copy.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ The cost of parsing and sharding the CSV stream in PgDog is negligibly small.
3838

3939
## COPY out
4040

41-
All `COPY [...] TO STDOUT` statements are treated as cross-shard and are executed on all shards concurrently. The rows are streamed directly, without buffering or sorting, which allows to read large amounts of data from all shards quickly.
41+
All `COPY [...] TO STDOUT` statements are treated as cross-shard and are executed on all shards concurrently. The rows are streamed directly, without buffering or sorting, which allows reading large amounts of data from all shards quickly.
4242

4343
PgDog doesn't currently support routing `COPY` statements based on its query. For example, the following statement will be sent to all shards even if it contains a sharding key:
4444

docs/features/sharding/cross-shard-queries/ddl.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,11 @@ icon: material/table-cog
44

55
# CREATE, ALTER, DROP
66

7-
`CREATE`, `ALTER` and `DROP`, also known as **D**ata **D**efinition **L**anguage (DDL), are by design, cross-shard statements. When a client sends over a DDL command, PgDog will send it to all shards in parallel, ensuring the table, index, view and sequence definitions are identical across the database cluster.
7+
`CREATE`, `ALTER` and `DROP`, also known as **D**ata **D**efinition **L**anguage (DDL), are, by design, cross-shard statements. When a client sends over a DDL command, PgDog will send it to all shards in parallel, ensuring the table, index, view and sequence definitions are identical across the database cluster.
88

99
## Atomicity
1010

11-
DDL statements should be atomic across all shards. This is to protect against a single shard failure to create a table or index, which could result in an inconsistent schema. PgDog can use [two-phase commit](../2pc.md) to ensure this is the case, however that means that all DDL statements must be executed inside a transaction, for example:
11+
DDL statements should be atomic across all shards. This is to protect against a single shard failing to create a table or index, which could result in an inconsistent schema. PgDog can use [two-phase commit](../2pc.md) to ensure this is the case, however that means that all DDL statements must be executed inside a transaction, for example:
1212

1313
```postgresql
1414
BEGIN;

docs/features/sharding/cross-shard-queries/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ Just like with [direct-to-shard](../query-routing.md) queries, each SQL command
2121
- [`SELECT`](select.md)
2222
- [`INSERT`](insert.md)
2323
- [`UPDATE`, `DELETE`](update.md)
24-
- [`CREATE`, `ALTER`, `DROP`](ddl.md)(and other DDL statements)
24+
- [`CREATE`, `ALTER`, `DROP`](ddl.md) (and other DDL statements)
2525
- [`COPY`](copy.md)
2626

2727

@@ -62,7 +62,7 @@ If you don't want PgDog to route cross-shard queries, e.g., because you have a [
6262
cross_shard_disabled = true
6363
```
6464

65-
If this feature is enabled, and a query doesn't have a sharding key, instead of executing the query, PgDog will return an error and abort the transaction.
65+
When this setting is enabled and a query doesn't have a sharding key, instead of executing the query, PgDog will return an error and abort the transaction.
6666

6767
## Read more
6868

docs/features/sharding/cross-shard-queries/insert.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,15 @@ If an `INSERT` statement specifies only one sharding key, it's sent [directly](.
1010

1111
Cross-shard `INSERT` statements fall into one of three categories, each one handled differently:
1212

13-
1. `INSERT` targetting [omnisharded tables](#omnisharded-tables)
14-
2. `INSERT` targetting a [sharded table](#sharded-insert), with no sharding key specified
13+
1. `INSERT` targeting [omnisharded tables](#omnisharded-tables)
14+
2. `INSERT` targeting a [sharded table](#sharded-tables), with no sharding key specified
1515
3. `INSERT` with [multiple tuples](#multiple-tuples), each destined for a different shard
1616

1717
By design, applications using PgDog don't need to concern themselves with this and can use the database normally. However, there are some trade-offs when using cross-shard queries, documented below.
1818

1919
## Omnisharded tables
2020

21-
Queries that target [omnisharded](../omnishards.md) tables, the statement is sent to all shards concurrently. This ensures that the data is identical on all shards, for example:
21+
For queries that target [omnisharded](../omnishards.md) tables, the statement is sent to all shards concurrently. This ensures that the data is identical on all shards, for example:
2222

2323
```postgresql
2424
INSERT INTO request_logs
@@ -43,7 +43,7 @@ VALUES
4343
COMMIT;
4444
```
4545

46-
This gives you a much higher chance of recording rows on all shards, since you will know if your statement violated some contstraint (e.g., unique index or `NOT NULL` check) before committing the transaction.
46+
This gives you a much higher chance of recording rows on all shards, since you will know if your statement violated some constraint (e.g., unique index or `NOT NULL` check) before committing the transaction.
4747

4848
### Primary keys
4949

@@ -57,7 +57,7 @@ To use unique IDs as primary keys (or in any other column) in omnisharded tables
5757
INSERT INTO ip_logs
5858
(id, client_ip, created_at)
5959
VALUES
60-
(pgdog.unqiue_id(), $1, now());
60+
(pgdog.unique_id(), $1, now());
6161
```
6262

6363
The function is evaluated inside PgDog which places the value it returns directly into the query. This works for all queries, including prepared statements.
@@ -82,21 +82,21 @@ This function can be used with any tables, not just omnisharded ones, or indepen
8282

8383
## Sharded tables
8484

85-
`INSERT` statements targetting sharded tables will commonly provide the sharding key. A notable exception to this rule is tables that shard on the primary key, which is often database-generated, e.g., using a sequence.
85+
`INSERT` statements targeting sharded tables will commonly provide the sharding key. A notable exception to this rule is tables that shard on the primary key, which is often database-generated, e.g., using a sequence.
8686

8787
The simplest way to work around this is to use the `pgdog.unique_id()` function to create a unique identifier on the fly, for example:
8888

8989
```postgresql
9090
INSERT INTO users
9191
(id, email, created_at)
9292
VALUES
93-
(pgdog.unqiue_id(), $1, $2)
93+
(pgdog.unique_id(), $1, $2)
9494
RETURNING *;
9595
```
9696

97-
If however, you prefer to use sequences instead, you can rely on [database-generated](../schema_management/primary_keys.md) primary keys.
97+
However, if you prefer to use sequences instead, you can rely on [database-generated](../schema_management/primary_keys.md) primary keys.
9898

99-
Statements that don't include the primary key in the `INSERT` tuple will be sent to one of the shards, using same round robin algorithm used for [omnisharded](#omnisharded-tables) tables. The shard will then generate the primary key value using PgDog's [sharded sequences](../schema_management/primary_keys.md#pgdognext_id_seq).
99+
Statements that don't include the primary key in the `INSERT` tuple will be sent to one of the shards, using the same round robin algorithm used for [omnisharded](#omnisharded-tables) tables. The shard will then generate the primary key value using PgDog's [sharded sequences](../schema_management/primary_keys.md#pgdognext_id_seq).
100100

101101
For example, assuming the table `users` is sharded on the primary key `id`, omitting it from the `INSERT` statement will send it to only one of the shards:
102102

@@ -153,7 +153,7 @@ INSERT INTO users (email, created_at) VALUES ($1, $2), ($3, $4);
153153
COMMIT; -- or ROLLBACK;
154154
```
155155

156-
If a transaction isn't started and a multi-tuple statement are sent by the application, PgDog will return an error and abort the request.
156+
If a transaction isn't started and a multi-tuple statement is sent by the application, PgDog will return an error and abort the request.
157157

158158
Requiring transactions ensures that if one of the `INSERT` statements fails, e.g., because of a unique constraint violation, the transaction can be rolled back, leaving the database in a consistent state.
159159

docs/features/sharding/cross-shard-queries/select.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ When PgDog receives a `SELECT` query with no (or multiple) sharding keys, it con
1313
If the result needs post-processing, e.g., to support [sorting](#sorting) or [aggregation](#aggregates), it will buffer the rows in memory and perform the necessary operations. Otherwise, PgDog will stream the rows directly to the client.
1414

1515
!!! note "Predicate push-down"
16-
PgDog pushes all filtering, sorting and aggregation statements to the database. If the query is correctly constructed, the shards will return very few rows, allowing to search vast quantities of data without causing out-of-memory errors or latency issues in the proxy.
16+
PgDog pushes all filtering, sorting and aggregation statements to the database. If the query is correctly constructed, the shards will return very few rows, allowing searches of vast quantities of data without causing out-of-memory errors or latency issues in the proxy.
1717

1818
## Supported features
1919

@@ -25,9 +25,9 @@ The SQL language allows for powerful data filtration and manipulation. While we
2525
| `ORDER BY` | :material-check: | Columns must be part of the returned tuples. See [sorting](#sorting). |
2626
| `DISTINCT` / `DISTINCT BY`| :material-check: | Columns must be part of the returned tuples. |
2727
| `GROUP BY` | :material-wrench: | Columns must be part of the returned tuples. See [aggregates](#aggregates). |
28-
| CTEs | :material-wrench: | CTE must refer to data located on the same shard. See [CTEs](#ctes) |
28+
| CTEs | :material-wrench: | CTE must refer to data located on the same shard. |
2929
| Window functions | :material-close: | Not currently supported. |
30-
| Subqueries | :material-wrench: | Subqueries must refer to data located on the same shard. See [subqueries](#subqueries). |
30+
| Subqueries | :material-wrench: | Subqueries must refer to data located on the same shard. |
3131

3232
## Sorting
3333

@@ -38,7 +38,7 @@ Currently, two forms of the `ORDER BY` SQL syntax are supported:
3838
| Syntax | Example | Notes |
3939
|-|-|-|
4040
| Order by column name | `ORDER BY id, email` | The column must be present in the returned tuples. |
41-
| Order by column position | `ORDER BY 1, 2` | The column is refernced by its position in the returned tuples. |
41+
| Order by column position | `ORDER BY 1, 2` | The column is referenced by its position in the returned tuples. |
4242

4343
Sorting by multiple columns is supported, including opposing sorting directions, for example:
4444

@@ -85,7 +85,7 @@ FROM users ORDER BY
8585

8686
ORMs like SQLAlchemy or ActiveRecord, more often than not, will write queries that work with PgDog out of the box. This is because they tend to fetch entire rows and use fully-qualified names in all parts of the statement, including the `ORDER BY` clause.
8787

88-
For example, this is how a [`first`](https://apidock.com/rails/ActiveRecord/FinderMethods/first) Rails/ActiveRecord query looks like:
88+
For example, this is what a [`first`](https://apidock.com/rails/ActiveRecord/FinderMethods/first) Rails/ActiveRecord query looks like:
8989

9090
```postgresql
9191
SELECT * FROM users ORDER BY users.id LIMIT 1
@@ -104,7 +104,7 @@ If an aggregate function is supported (see list of supported functions below), t
104104
| `count`, `count(*)` | :material-check: | Works for most [data types](#supported-data-types). |
105105
| `max`, `min`, `avg`, `sum` | :material-check: | Works for most [data types](#supported-data-types). |
106106
| `stddev`, `variance` | :material-check: | Works for most [data types](#supported-data-types).|
107-
| `percentile_disc`, `percentile_cont` | :material-close: | Doesn't work currently and very expensive to calculate on large datasets. |
107+
| `percentile_disc`, `percentile_cont` | :material-close: | Not currently supported and very expensive to calculate on large datasets. |
108108
| `*_agg` | :material-close: | Not currently supported. |
109109
| `json_*` | :material-close: | Not currently supported. |
110110

@@ -126,7 +126,7 @@ The `HAVING` clause requires additional filtering of the results and is not curr
126126

127127
For some aggregate functions to work as expected, each shard may need to return columns and intermediate calculations not present in the original query.
128128

129-
For example, to get an average of a column, we need to fetch the row `count`, multiply it by the `avg` the column on each shard, and divide it by the total `count` of rows on all shards.
129+
For example, to get an average of a column, we need to fetch the row `count`, multiply it by the `avg` of the column on each shard, and divide it by the total `count` of rows on all shards.
130130

131131
If the `count` function isn't present in the query, PgDog will automatically rewrite the query to add it. This allows queries, like the following example, to just work without modifications:
132132

docs/features/sharding/cross-shard-queries/update.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -41,19 +41,19 @@ When a client sends such a statement, PgDog will rewrite it into three statement
4141

4242
1. `SELECT` statement to fetch the row from the database
4343
2. `INSERT` statement to create that row on the new shard, with updated values
44-
2. `DELETE` statement to remove the now stale row from the old shard
44+
3. `DELETE` statement to remove the now stale row from the old shard
4545

4646
The entire exchange looks like this:
4747

4848
```postgresql
4949
-- Old `id` value.
5050
SELECT * FROM users WHERE id = $1;
5151
52-
-- New `id` value, with columns retrieved by the previous query.
52+
-- New `id` value, with columns retrieved by the previous query.
5353
INSERT INTO users (id, email, created_at)
5454
VALUES ($1, $2, $3);
5555
56-
-- Old `id` value.
56+
-- Old `id` value.
5757
DELETE FROM users WHERE id = $1;
5858
```
5959

@@ -81,13 +81,13 @@ shard_key = "rewrite"
8181

8282
### Updating multiple rows
8383

84-
While multi-row updates are supported, changing multiple rows' sharding keys is not. If the `UPDATE` statement `WHERE` clause returns more than one row, PgDog will abort the transaction and return an error.
84+
While multi-row updates are supported, changing multiple rows' sharding keys is not. If the `UPDATE` statement's `WHERE` clause matches more than one row, PgDog will abort the transaction and return an error.
8585

8686
This check happens early in the transaction and will not create partial updates to the database.
8787

8888
### Efficiency
8989

90-
It's common practice for ORMs, like ActiveRecord or SQLAlchemy to mutate entire rows when saving records. Take the following example:
90+
It's common practice for ORMs, like ActiveRecord or SQLAlchemy, to mutate entire rows when saving records. Take the following example:
9191

9292
=== "ActiveRecord"
9393
```ruby
@@ -106,6 +106,7 @@ It's common practice for ORMs, like ActiveRecord or SQLAlchemy to mutate entire
106106
While the sharding key (`id`) is technically updated in the query, its value doesn't change. To avoid unnecessary overhead, PgDog performs multiple checks on the new row before performing the [three statement](#how-it-works) sharding key update flow:
107107

108108
1. If the value of the sharding key parameter (`SET id = $1`) is the same as it is in the `WHERE` clause (`WHERE id = $1`), the query is routed directly to the shard without modifications.
109-
2. If the values are different, or the `WHERE` clause doesn't specify the sharding key, PgDog will check the value of the key returned by the first `SELECT` statement in the flow. If both keys map to the _same_ shard, will send the query directly to that shard, without modifications.
109+
2. If the values are different, or the `WHERE` clause doesn't specify the sharding key, PgDog will check the value of the key returned by the first `SELECT` statement in the flow. If both keys map to the _same_ shard, PgDog will send the query directly to that shard, without modifications.
110+
110111

111-
Updating the sharding key isn't a frequent operation and both of these mechanisms ensure that the more expensive algorithm isn't triggered unnecessarily.
112+
Updating the sharding key isn't a frequent operation and both of these mechanisms ensure that the more expensive algorithm isn't triggered unnecessarily.

docs/features/sharding/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Sharding splits a PostgreSQL database and all its tables and indices between mul
88

99
## Intro to sharding
1010

11-
If you're not familiar with database sharding fundamentals, take a look at the [sharding basics](basics.md). Even if you're a seasoned database expert, it's good to have a refresher to confirm your understanding matches with our implementation.
11+
If you're not familiar with database sharding fundamentals, take a look at the [sharding basics](basics.md). Even if you're a seasoned database expert, it's good to have a refresher to confirm your understanding matches our implementation.
1212

1313
[**→ Sharding basics**](basics.md)
1414

@@ -41,6 +41,6 @@ Resharding takes a database cluster with _N_ shards (where _N_ can be 1, for uns
4141

4242
### Schema management
4343

44-
PgDog makes sure that database schema is identical on all shards. It also has support for in-database primary key generation.
44+
PgDog makes sure that the database schema is identical on all shards. It also has support for in-database primary key generation.
4545

4646
[**→ Schema management**](schema_management/index.md)

docs/features/sharding/supported-queries.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ icon: material/check-bold
33
---
44
# Supported queries
55

6-
[Automatic routing](query-routing.md) in PgDog works by parsing queries and extracting the sharding key. SQL is a complex language and we are aiming to support as many queries as possible. As the development moves forward, this page will be updated with latest features.
6+
[Automatic routing](query-routing.md) in PgDog works by parsing queries and extracting the sharding key. SQL is a complex language and we are aiming to support as many queries as possible. As the development moves forward, this page will be updated with the latest features.
77

88
Postgres has 3 kinds of queries, each handled a little bit differently in a sharded context:
99

@@ -52,7 +52,7 @@ Queries that don't specify a sharding key or specify more than one will be route
5252

5353
### `UPDATE` and `DELETE`
5454

55-
Both `UPDATE` and `DELETE` queries work the same way as `SELECT` queries. The `WHERE` clause and any CTEs are checked for a sharding key, using any of the supported patterns and if a key is found, the query is routed to directly to one shard. Statements without a key are sent to all shards, in parallel.
55+
Both `UPDATE` and `DELETE` queries work the same way as `SELECT` queries. The `WHERE` clause and any CTEs are checked for a sharding key, using any of the supported patterns, and if a key is found, the query is routed directly to one shard. Statements without a key are sent to all shards, in parallel.
5656

5757
#### Examples
5858

0 commit comments

Comments
 (0)