You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/features/sharding/cross-shard-queries/copy.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,7 +38,7 @@ The cost of parsing and sharding the CSV stream in PgDog is negligibly small.
38
38
39
39
## COPY out
40
40
41
-
All `COPY [...] TO STDOUT` statements are treated as cross-shard and are executed on all shards concurrently. The rows are streamed directly, without buffering or sorting, which allows to read large amounts of data from all shards quickly.
41
+
All `COPY [...] TO STDOUT` statements are treated as cross-shard and are executed on all shards concurrently. The rows are streamed directly, without buffering or sorting, which allows reading large amounts of data from all shards quickly.
42
42
43
43
PgDog doesn't currently support routing `COPY` statements based on its query. For example, the following statement will be sent to all shards even if it contains a sharding key:
Copy file name to clipboardExpand all lines: docs/features/sharding/cross-shard-queries/ddl.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,11 +4,11 @@ icon: material/table-cog
4
4
5
5
# CREATE, ALTER, DROP
6
6
7
-
`CREATE`, `ALTER` and `DROP`, also known as **D**ata **D**efinition **L**anguage (DDL), are by design, cross-shard statements. When a client sends over a DDL command, PgDog will send it to all shards in parallel, ensuring the table, index, view and sequence definitions are identical across the database cluster.
7
+
`CREATE`, `ALTER` and `DROP`, also known as **D**ata **D**efinition **L**anguage (DDL), are, by design, cross-shard statements. When a client sends over a DDL command, PgDog will send it to all shards in parallel, ensuring the table, index, view and sequence definitions are identical across the database cluster.
8
8
9
9
## Atomicity
10
10
11
-
DDL statements should be atomic across all shards. This is to protect against a single shard failure to create a table or index, which could result in an inconsistent schema. PgDog can use [two-phase commit](../2pc.md) to ensure this is the case, however that means that all DDL statements must be executed inside a transaction, for example:
11
+
DDL statements should be atomic across all shards. This is to protect against a single shard failing to create a table or index, which could result in an inconsistent schema. PgDog can use [two-phase commit](../2pc.md) to ensure this is the case, however that means that all DDL statements must be executed inside a transaction, for example:
Copy file name to clipboardExpand all lines: docs/features/sharding/cross-shard-queries/index.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ Just like with [direct-to-shard](../query-routing.md) queries, each SQL command
21
21
-[`SELECT`](select.md)
22
22
-[`INSERT`](insert.md)
23
23
-[`UPDATE`, `DELETE`](update.md)
24
-
-[`CREATE`, `ALTER`, `DROP`](ddl.md)(and other DDL statements)
24
+
-[`CREATE`, `ALTER`, `DROP`](ddl.md)(and other DDL statements)
25
25
-[`COPY`](copy.md)
26
26
27
27
@@ -62,7 +62,7 @@ If you don't want PgDog to route cross-shard queries, e.g., because you have a [
62
62
cross_shard_disabled = true
63
63
```
64
64
65
-
If this feature is enabled, and a query doesn't have a sharding key, instead of executing the query, PgDog will return an error and abort the transaction.
65
+
When this setting is enabled and a query doesn't have a sharding key, instead of executing the query, PgDog will return an error and abort the transaction.
2.`INSERT`targeting a [sharded table](#sharded-tables), with no sharding key specified
15
15
3.`INSERT` with [multiple tuples](#multiple-tuples), each destined for a different shard
16
16
17
17
By design, applications using PgDog don't need to concern themselves with this and can use the database normally. However, there are some trade-offs when using cross-shard queries, documented below.
18
18
19
19
## Omnisharded tables
20
20
21
-
Queries that target [omnisharded](../omnishards.md) tables, the statement is sent to all shards concurrently. This ensures that the data is identical on all shards, for example:
21
+
For queries that target [omnisharded](../omnishards.md) tables, the statement is sent to all shards concurrently. This ensures that the data is identical on all shards, for example:
22
22
23
23
```postgresql
24
24
INSERT INTO request_logs
@@ -43,7 +43,7 @@ VALUES
43
43
COMMIT;
44
44
```
45
45
46
-
This gives you a much higher chance of recording rows on all shards, since you will know if your statement violated some contstraint (e.g., unique index or `NOT NULL` check) before committing the transaction.
46
+
This gives you a much higher chance of recording rows on all shards, since you will know if your statement violated some constraint (e.g., unique index or `NOT NULL` check) before committing the transaction.
47
47
48
48
### Primary keys
49
49
@@ -57,7 +57,7 @@ To use unique IDs as primary keys (or in any other column) in omnisharded tables
57
57
INSERT INTO ip_logs
58
58
(id, client_ip, created_at)
59
59
VALUES
60
-
(pgdog.unqiue_id(), $1, now());
60
+
(pgdog.unique_id(), $1, now());
61
61
```
62
62
63
63
The function is evaluated inside PgDog which places the value it returns directly into the query. This works for all queries, including prepared statements.
@@ -82,21 +82,21 @@ This function can be used with any tables, not just omnisharded ones, or indepen
82
82
83
83
## Sharded tables
84
84
85
-
`INSERT` statements targetting sharded tables will commonly provide the sharding key. A notable exception to this rule is tables that shard on the primary key, which is often database-generated, e.g., using a sequence.
85
+
`INSERT` statements targeting sharded tables will commonly provide the sharding key. A notable exception to this rule is tables that shard on the primary key, which is often database-generated, e.g., using a sequence.
86
86
87
87
The simplest way to work around this is to use the `pgdog.unique_id()` function to create a unique identifier on the fly, for example:
88
88
89
89
```postgresql
90
90
INSERT INTO users
91
91
(id, email, created_at)
92
92
VALUES
93
-
(pgdog.unqiue_id(), $1, $2)
93
+
(pgdog.unique_id(), $1, $2)
94
94
RETURNING *;
95
95
```
96
96
97
-
If however, you prefer to use sequences instead, you can rely on [database-generated](../schema_management/primary_keys.md) primary keys.
97
+
However, if you prefer to use sequences instead, you can rely on [database-generated](../schema_management/primary_keys.md) primary keys.
98
98
99
-
Statements that don't include the primary key in the `INSERT` tuple will be sent to one of the shards, using same round robin algorithm used for [omnisharded](#omnisharded-tables) tables. The shard will then generate the primary key value using PgDog's [sharded sequences](../schema_management/primary_keys.md#pgdognext_id_seq).
99
+
Statements that don't include the primary key in the `INSERT` tuple will be sent to one of the shards, using the same round robin algorithm used for [omnisharded](#omnisharded-tables) tables. The shard will then generate the primary key value using PgDog's [sharded sequences](../schema_management/primary_keys.md#pgdognext_id_seq).
100
100
101
101
For example, assuming the table `users` is sharded on the primary key `id`, omitting it from the `INSERT` statement will send it to only one of the shards:
If a transaction isn't started and a multi-tuple statement are sent by the application, PgDog will return an error and abort the request.
156
+
If a transaction isn't started and a multi-tuple statement is sent by the application, PgDog will return an error and abort the request.
157
157
158
158
Requiring transactions ensures that if one of the `INSERT` statements fails, e.g., because of a unique constraint violation, the transaction can be rolled back, leaving the database in a consistent state.
Copy file name to clipboardExpand all lines: docs/features/sharding/cross-shard-queries/select.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,7 +13,7 @@ When PgDog receives a `SELECT` query with no (or multiple) sharding keys, it con
13
13
If the result needs post-processing, e.g., to support [sorting](#sorting) or [aggregation](#aggregates), it will buffer the rows in memory and perform the necessary operations. Otherwise, PgDog will stream the rows directly to the client.
14
14
15
15
!!! note "Predicate push-down"
16
-
PgDog pushes all filtering, sorting and aggregation statements to the database. If the query is correctly constructed, the shards will return very few rows, allowing to search vast quantities of data without causing out-of-memory errors or latency issues in the proxy.
16
+
PgDog pushes all filtering, sorting and aggregation statements to the database. If the query is correctly constructed, the shards will return very few rows, allowing searches of vast quantities of data without causing out-of-memory errors or latency issues in the proxy.
17
17
18
18
## Supported features
19
19
@@ -25,9 +25,9 @@ The SQL language allows for powerful data filtration and manipulation. While we
25
25
|`ORDER BY`| :material-check: | Columns must be part of the returned tuples. See [sorting](#sorting). |
26
26
|`DISTINCT` / `DISTINCT BY`| :material-check: | Columns must be part of the returned tuples. |
27
27
|`GROUP BY`| :material-wrench: | Columns must be part of the returned tuples. See [aggregates](#aggregates). |
28
-
| CTEs | :material-wrench: | CTE must refer to data located on the same shard. See [CTEs](#ctes)|
28
+
| CTEs | :material-wrench: | CTE must refer to data located on the same shard. |
29
29
| Window functions | :material-close: | Not currently supported. |
30
-
| Subqueries | :material-wrench: | Subqueries must refer to data located on the same shard. See [subqueries](#subqueries). |
30
+
| Subqueries | :material-wrench: | Subqueries must refer to data located on the same shard. |
31
31
32
32
## Sorting
33
33
@@ -38,7 +38,7 @@ Currently, two forms of the `ORDER BY` SQL syntax are supported:
38
38
| Syntax | Example | Notes |
39
39
|-|-|-|
40
40
| Order by column name |`ORDER BY id, email`| The column must be present in the returned tuples. |
41
-
| Order by column position |`ORDER BY 1, 2`| The column is refernced by its position in the returned tuples. |
41
+
| Order by column position |`ORDER BY 1, 2`| The column is referenced by its position in the returned tuples. |
42
42
43
43
Sorting by multiple columns is supported, including opposing sorting directions, for example:
44
44
@@ -85,7 +85,7 @@ FROM users ORDER BY
85
85
86
86
ORMs like SQLAlchemy or ActiveRecord, more often than not, will write queries that work with PgDog out of the box. This is because they tend to fetch entire rows and use fully-qualified names in all parts of the statement, including the `ORDER BY` clause.
87
87
88
-
For example, this is how a [`first`](https://apidock.com/rails/ActiveRecord/FinderMethods/first) Rails/ActiveRecord query looks like:
88
+
For example, this is what a [`first`](https://apidock.com/rails/ActiveRecord/FinderMethods/first) Rails/ActiveRecord query looks like:
89
89
90
90
```postgresql
91
91
SELECT * FROM users ORDER BY users.id LIMIT 1
@@ -104,7 +104,7 @@ If an aggregate function is supported (see list of supported functions below), t
104
104
|`count`, `count(*)`| :material-check: | Works for most [data types](#supported-data-types). |
105
105
|`max`, `min`, `avg`, `sum`| :material-check: | Works for most [data types](#supported-data-types). |
106
106
|`stddev`, `variance`| :material-check: | Works for most [data types](#supported-data-types).|
107
-
|`percentile_disc`, `percentile_cont`| :material-close: |Doesn't work currently and very expensive to calculate on large datasets. |
107
+
|`percentile_disc`, `percentile_cont`| :material-close: |Not currently supported and very expensive to calculate on large datasets. |
108
108
|`*_agg`| :material-close: | Not currently supported. |
109
109
|`json_*`| :material-close: | Not currently supported. |
110
110
@@ -126,7 +126,7 @@ The `HAVING` clause requires additional filtering of the results and is not curr
126
126
127
127
For some aggregate functions to work as expected, each shard may need to return columns and intermediate calculations not present in the original query.
128
128
129
-
For example, to get an average of a column, we need to fetch the row `count`, multiply it by the `avg` the column on each shard, and divide it by the total `count` of rows on all shards.
129
+
For example, to get an average of a column, we need to fetch the row `count`, multiply it by the `avg`of the column on each shard, and divide it by the total `count` of rows on all shards.
130
130
131
131
If the `count` function isn't present in the query, PgDog will automatically rewrite the query to add it. This allows queries, like the following example, to just work without modifications:
Copy file name to clipboardExpand all lines: docs/features/sharding/cross-shard-queries/update.md
+8-7Lines changed: 8 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,19 +41,19 @@ When a client sends such a statement, PgDog will rewrite it into three statement
41
41
42
42
1.`SELECT` statement to fetch the row from the database
43
43
2.`INSERT` statement to create that row on the new shard, with updated values
44
-
2.`DELETE` statement to remove the now stale row from the old shard
44
+
3.`DELETE` statement to remove the now stale row from the old shard
45
45
46
46
The entire exchange looks like this:
47
47
48
48
```postgresql
49
49
-- Old `id` value.
50
50
SELECT * FROM users WHERE id = $1;
51
51
52
-
-- New `id` value, with columns retrieved by the previous query.
52
+
-- New `id` value, with columns retrieved by the previous query.
53
53
INSERT INTO users (id, email, created_at)
54
54
VALUES ($1, $2, $3);
55
55
56
-
-- Old `id` value.
56
+
-- Old `id` value.
57
57
DELETE FROM users WHERE id = $1;
58
58
```
59
59
@@ -81,13 +81,13 @@ shard_key = "rewrite"
81
81
82
82
### Updating multiple rows
83
83
84
-
While multi-row updates are supported, changing multiple rows' sharding keys is not. If the `UPDATE` statement `WHERE` clause returns more than one row, PgDog will abort the transaction and return an error.
84
+
While multi-row updates are supported, changing multiple rows' sharding keys is not. If the `UPDATE` statement's`WHERE` clause matches more than one row, PgDog will abort the transaction and return an error.
85
85
86
86
This check happens early in the transaction and will not create partial updates to the database.
87
87
88
88
### Efficiency
89
89
90
-
It's common practice for ORMs, like ActiveRecord or SQLAlchemy to mutate entire rows when saving records. Take the following example:
90
+
It's common practice for ORMs, like ActiveRecord or SQLAlchemy, to mutate entire rows when saving records. Take the following example:
91
91
92
92
=== "ActiveRecord"
93
93
```ruby
@@ -106,6 +106,7 @@ It's common practice for ORMs, like ActiveRecord or SQLAlchemy to mutate entire
106
106
While the sharding key (`id`) is technically updated in the query, its value doesn't change. To avoid unnecessary overhead, PgDog performs multiple checks on the new row before performing the [three statement](#how-it-works) sharding key update flow:
107
107
108
108
1. If the value of the sharding key parameter (`SET id = $1`) is the same as it is in the `WHERE` clause (`WHERE id = $1`), the query is routed directly to the shard without modifications.
109
-
2. If the values are different, or the `WHERE` clause doesn't specify the sharding key, PgDog will check the value of the key returned by the first `SELECT` statement in the flow. If both keys map to the _same_ shard, will send the query directly to that shard, without modifications.
109
+
2. If the values are different, or the `WHERE` clause doesn't specify the sharding key, PgDog will check the value of the key returned by the first `SELECT` statement in the flow. If both keys map to the _same_ shard, PgDog will send the query directly to that shard, without modifications.
110
+
110
111
111
-
Updating the sharding key isn't a frequent operation and both of these mechanisms ensure that the more expensive algorithm isn't triggered unnecessarily.
112
+
Updating the sharding key isn't a frequent operation and both of these mechanisms ensure that the more expensive algorithm isn't triggered unnecessarily.
Copy file name to clipboardExpand all lines: docs/features/sharding/index.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ Sharding splits a PostgreSQL database and all its tables and indices between mul
8
8
9
9
## Intro to sharding
10
10
11
-
If you're not familiar with database sharding fundamentals, take a look at the [sharding basics](basics.md). Even if you're a seasoned database expert, it's good to have a refresher to confirm your understanding matches with our implementation.
11
+
If you're not familiar with database sharding fundamentals, take a look at the [sharding basics](basics.md). Even if you're a seasoned database expert, it's good to have a refresher to confirm your understanding matches our implementation.
12
12
13
13
[**→ Sharding basics**](basics.md)
14
14
@@ -41,6 +41,6 @@ Resharding takes a database cluster with _N_ shards (where _N_ can be 1, for uns
41
41
42
42
### Schema management
43
43
44
-
PgDog makes sure that database schema is identical on all shards. It also has support for in-database primary key generation.
44
+
PgDog makes sure that the database schema is identical on all shards. It also has support for in-database primary key generation.
Copy file name to clipboardExpand all lines: docs/features/sharding/supported-queries.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ icon: material/check-bold
3
3
---
4
4
# Supported queries
5
5
6
-
[Automatic routing](query-routing.md) in PgDog works by parsing queries and extracting the sharding key. SQL is a complex language and we are aiming to support as many queries as possible. As the development moves forward, this page will be updated with latest features.
6
+
[Automatic routing](query-routing.md) in PgDog works by parsing queries and extracting the sharding key. SQL is a complex language and we are aiming to support as many queries as possible. As the development moves forward, this page will be updated with the latest features.
7
7
8
8
Postgres has 3 kinds of queries, each handled a little bit differently in a sharded context:
9
9
@@ -52,7 +52,7 @@ Queries that don't specify a sharding key or specify more than one will be route
52
52
53
53
### `UPDATE` and `DELETE`
54
54
55
-
Both `UPDATE` and `DELETE` queries work the same way as `SELECT` queries. The `WHERE` clause and any CTEs are checked for a sharding key, using any of the supported patterns and if a key is found, the query is routed to directly to one shard. Statements without a key are sent to all shards, in parallel.
55
+
Both `UPDATE` and `DELETE` queries work the same way as `SELECT` queries. The `WHERE` clause and any CTEs are checked for a sharding key, using any of the supported patterns, and if a key is found, the query is routed directly to one shard. Statements without a key are sent to all shards, in parallel.
0 commit comments