Skip to content

Commit f22b609

Browse files
levkklevkk
committed
feat: document sharded sequences and new mirroring features
1 parent 54719e2 commit f22b609

5 files changed

Lines changed: 229 additions & 13 deletions

File tree

‎docs/configuration/pgdog.toml/mirroring.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,11 @@ Default: **none** (optional)
4141
The percentage of transactions to mirror, specified as a floating point number between 0.0 and 1.0. See [mirroring](../../features/mirroring.md) for more details. This overrides the [`mirror_exposure`](./general.md#mirror_exposure) setting.
4242

4343
Default: **none** (optional)
44+
45+
### `level`
46+
47+
The type of statements to mirror. Available options are:
48+
49+
- `ddl`
50+
- `dml`
51+
- `all` (default)

‎docs/features/mirroring.md‎

Lines changed: 47 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Mirroring in PgDog is asynchronous and should have minimal impact on production
1414
<img src="/images/mirroring.png" width="80%" height="auto" alt="Mirroring">
1515
</center>
1616

17-
## Configuration
17+
### Configuration
1818

1919
To use mirroring, first configure both the mirror and the production database in [`pgdog.toml`](../configuration/pgdog.toml/databases.md). Once both databases are running, add a `[[mirroring]]` section:
2020

@@ -43,7 +43,7 @@ Each client connected to the main database has its own queue, so concurrency sca
4343

4444
You can have as many mirror databases as you like. Queries will be sent to each one of them, in parallel. More mirrors will require more CPU and network resources, so make sure to allocate enough compute to PgDog in production.
4545

46-
## Mirror queue
46+
### Mirror queue
4747

4848
If the mirror database(s) can't keep up with production traffic, queries will back up in the queue. To make sure it doesn't overflow and cause out-of-memory errors, the size of the queue is limited:
4949

@@ -69,7 +69,7 @@ If the queue gets full, all subsequent mirrored transactions will be dropped unt
6969
!!! note "Replication"
7070
Since mirror queues can drop queries, it is not a replacement for Postgres replication and should be used for testing & benchmarking purposes only.
7171

72-
## Exposure
72+
### Exposure
7373

7474
It's possible to limit how much traffic mirror databases receive. This is useful when warming up databases from a snapshot or if the mirror databases are smaller than production and can't handle as many transactions.
7575

@@ -96,8 +96,51 @@ Acceptable values are between **0.0** (0%) and **1.0** (100%).
9696

9797
This is changeable at runtime, without restarting PgDog. When adding a mirror, it's a good idea to start slow, e.g., with only 0.1% exposure (`mirror_exposure = 0.01`), and gradually increase it over time.
9898

99-
## Realism
99+
### Realism
100100

101101
We try to make mirrored traffic as realistic as possible. For each statement inside a transaction, we record the timing between that statement and the next one.
102102

103103
When replaying traffic against a mirror, we pause between statements for the same amount of time. This helps reproduce lock contention experienced by production databases, on the mirrors.
104+
105+
### Filtering
106+
107+
It's possible to filter what kind of statements mirrors receive using configuration, for example:
108+
109+
=== "pgdog.toml"
110+
```toml
111+
[[mirroring]]
112+
source_db = "source"
113+
destination_db = "dest"
114+
level = "ddl"
115+
```
116+
=== "Helm chart"
117+
```yaml
118+
mirroring:
119+
- sourceDb: source
120+
destinationDb: dest
121+
level: ddl
122+
```
123+
124+
The `level` setting supports the following arguments:
125+
126+
| Argument | Description |
127+
|-|-|
128+
| `ddl` | Mirror only DDL statements like `CREATE`, `DROP`, etc. |
129+
| `dml` | Mirror all statements except DDL, e.g. `INSERT`, `UPDATE`, etc. |
130+
| `all` | Mirror all statements. This is the default. |
131+
132+
DDL-only mirroring is useful when maintaining long-running logical replicas, since the logical replication protocol doesn't support synchronizing schema changes.
133+
134+
#### Query parser
135+
136+
Filtering specific statements requires parsing queries. If your database setup doesn't have replicas or sharding, the query parser is typically disabled. Before using this feature, make sure to enable it in [`pgdog.toml`](../configuration/pgdog.toml/general.md#query_parser):
137+
138+
=== "pgdog.toml"
139+
```toml
140+
[general]
141+
query_parser = "on"
142+
```
143+
=== "Helm chart"
144+
```yaml
145+
queryParser: on
146+
```

‎docs/features/sharding/schema_management/migrations.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
---
22
icon: material/arrow-u-left-bottom
33
---
4+
45
# Schema migrations
56

67
PgDog expects that all shards have, roughly, the same tables. A notable exception to this rule is partitioned tables,

‎docs/features/sharding/sequences.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
---
2+
icon: material/numeric
3+
---
4+
5+
# Sharded sequences
6+
7+
!!! note "Unique IDs"
8+
Sharded sequences require a bit more configuration to get working. If you're looking
9+
for an easy way to generate cross-shard unique 64-bit integers, consider [Unique IDs](unique-ids.md).
10+
11+
!!! note "Experimental feature"
12+
This feature is new and experimental. Please report any issues you may run into and test it
13+
before deploying to production.
14+
15+
Sharded sequences are a way to generate monotonically increasing, globally unique 64-bit integers, without large gaps between numbers
16+
or using a timestamp-based approach that produces very large numbers.
17+
18+
They can be used for producing cross-shard unique primary keys in [sharded](query-routing.md#sharding-configuration) tables, directly inside the database.
19+
20+
## How it works
21+
22+
Sharded sequences combine two Postgres primitives:
23+
24+
1. A normal sequence (created with `CREATE SEQUENCE`)
25+
2. A hashing function, `satisfies_hash_partition`, used for number selection
26+
27+
The two are called inside a PL/pgSQL function that fetches numbers from a sequence until `satisfies_hash_partition` returns `true`, for the total number of shards in the cluster and the shard number it's being executed on:
28+
29+
```postgresql
30+
LOOP
31+
SELECT nextval('normal_seq'::regclass) INTO val;
32+
33+
IF satisfies_hash_partition(/* ... */, val) THEN
34+
RETURN val;
35+
END IF;
36+
37+
END;
38+
```
39+
40+
Since fetching values from a sequence is very quick, we are able to find the correct number without introducing significant latency to row creation. The Postgres hash function is also good at producing uniform outputs, so all shards will have similar, small gaps between generated numbers.
41+
42+
### Configuration
43+
44+
Sharded sequences can only be used to generate primary keys for _sharded_ tables. [Omnisharded](omnishards.md) tables cannot use database sequences since they aren't guaranteed to produce the same number on all shards.
45+
46+
To make sure this constraint is enforced, PgDog can inject [unique IDs](unique-ids.md) into omnisharded-targeted `INSERT` queries only:
47+
48+
=== "pgdog.toml"
49+
```toml
50+
[rewrite]
51+
primary_key = "rewrite_omni"
52+
```
53+
=== "Helm chart"
54+
```yaml
55+
rewrite:
56+
primaryKey: rewrite_omni
57+
```
58+
59+
This configuration setting is required to use sharded sequences, so make sure to set it before proceeding.
60+
61+
### Installation
62+
63+
To install and use sharded sequences, configure [rewrites](#configuration) to target omnisharded tables only, add all the shards to [`pgdog.toml`](../../configuration/pgdog.toml/databases.md) `[[databases]]` section, and run the following [admin database](../../administration/index.md) command:
64+
65+
=== "Admin database"
66+
```
67+
SETUP SCHEMA;
68+
```
69+
=== "CLI"
70+
Since PgDog is also a CLI application, you can run the same command as follows:
71+
72+
```
73+
$ pgdog setup --database <name>
74+
```
75+
76+
| Option | Description |
77+
|-|-|
78+
| `database` | Database `name` in `pgdog.toml`. |
79+
80+
This command will perform the following steps:
81+
82+
1. Install the [schema manager](schema_management/index.md) into all database shards along with the necessary PL/pgSQL functions
83+
2. Find all tables that contain `BIGINT PRIMARY KEY` columns (incl. `BIGSERIAL`) and change their default values to call the sharded sequence function
84+
85+
Once done, all subsequent `INSERT` statements that don't specify the primary key will automatically use the sharded sequence for their respective tables, for example:
86+
87+
=== "Queries"
88+
```postgresql
89+
-- Using DEFAULT explicitly.
90+
INSERT INTO users
91+
(id, email, tenant_id)
92+
VALUES
93+
(DEFAULT, 'admin@example.com', 5) RETURNING id;
94+
95+
-- Omitting the primary key.
96+
INSERT INTO users
97+
(email, tenant_id)
98+
VALUES
99+
('user@example.com', 5) RETURNING id;
100+
```
101+
=== "Output"
102+
```
103+
id
104+
----
105+
1
106+
(1 row)
107+
108+
id
109+
----
110+
5
111+
(1 row)
112+
```
113+
114+
The returned `id` will be globally unique and monotonically increasing.
115+
116+
### Migrations
117+
118+
The schema manager will only install the sharded sequence in tables currently present in the database. When adding new tables or primary keys, make sure to execute the following PL/pgSQL function
119+
as well:
120+
121+
```postgresql
122+
SELECT pgdog.install_sharded_sequence('schema_name', 'table_name', 'column_name');
123+
```
124+
125+
| Argument | Description |
126+
|-|-|
127+
| Schema name | The name of the schema where the table is being created. This is commonly the `public` schema, but can be any other as well. |
128+
| Table name | The name of the new or existing table with the primary key. |
129+
| Column name | The name of the primary key column. |
130+
131+
##### Example
132+
133+
The entire migration can be executed inside the same transaction:
134+
135+
```postgresql
136+
BEGIN;
137+
138+
CREATE TABLE public.users (
139+
id BIGINT PRIMARY KEY,
140+
email VARCHAR NOT NULL,
141+
created_at TIMESTAMPTZ
142+
);
143+
144+
SELECT pgdog.install_sharded_sequence('public', 'users', 'id');
145+
146+
COMMIT;
147+
```

‎docs/features/sharding/unique-ids.md‎

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,9 @@ icon: material/identifier
33
---
44
# Unique IDs
55

6-
To generate unique identifiers, regular PostgreSQL databases use [sequences](https://www.postgresql.org/docs/current/sql-createsequence.html). For example, `BIGSERIAL` and `SERIAL` columns get their values by calling:
6+
To generate unique identifiers, regular PostgreSQL databases use [sequences](https://www.postgresql.org/docs/current/sql-createsequence.html). For example, `BIGSERIAL` and `SERIAL` columns get their values by calling `SELECT nextval('users_id_seq')`.
77

8-
```postgresql
9-
SELECT nextval('sequence_name');
10-
```
11-
12-
This guarantees that these columns contain unique and monotonically increasing integers.
13-
14-
If your database is sharded, however, using sequences will create identical IDs for different rows on different shards. To address this, PgDog can generate unique 64-bit signed identifiers internally, based on the system clock.
8+
This guarantees that these columns contain unique and monotonically increasing integers. If your database is sharded, however, using regular sequences will create identical IDs for different rows on different shards. To address this, PgDog can generate unique 64-bit signed identifiers internally, based on the system clock.
159

1610
## How it works
1711

@@ -95,7 +89,7 @@ If you're migrating data from an existing database, you can ensure that all IDs
9589
unique_id_min = 5_000_000
9690
```
9791

98-
When set, all generated IDs are guaranteed to be larger than this value.
92+
When set, all generated IDs are guaranteed to be larger than this value. This feature however is normally not needed, since IDs generated by this function are very large.
9993

10094
## Limitations
10195

@@ -117,3 +111,26 @@ ID range is **69.73 years**, set to overflow on **August 3, 2095**. We expect da
117111
Since the identifiers are time-based, to ensure uniqueness, PgDog limits how many IDs can be generated per unit of time. This limit is currently **4,096** IDs per millisecond.
118112

119113
When it's reached, PgDog will pause ID generation until the clock ticks to the next millisecond. This gives it an effective ID generation rate of _4,096,000 / second / node_, which should be sufficient for most deployments.
114+
115+
## Compact IDs
116+
117+
The unique ID algorithm generates very large 64-bit integers. This is because the timestamp portion is located 22 bits off to the left (little-endian). Some applications pass those IDs directly to the JavaScript-written frontends, which cannot display those numbers accurately: JS doesn't support any numbers larger than 2^53 - 1.
118+
119+
For this reason, we added a more "compact", 53-bit unique ID generator function. It can be used by enabling it in [`pgdog.toml`](../../configuration/pgdog.toml/general.md):
120+
121+
```toml
122+
[general]
123+
unique_id_function = "compact"
124+
```
125+
126+
!!! warning "Switching to the compact generator"
127+
If you're currently using the `"standard"` unique ID generator (default), be careful switching because the IDs it will generate will be considerably smaller, breaking the monotonic guarantee
128+
and possibly causing unique index constraint errors.
129+
130+
### Limitations
131+
132+
Since the bit space in this function is smaller, and the timestamp granularity had to remain the same (ms), the space allocated to the node identifier
133+
and the internal sequence has been reduced accordingly.
134+
135+
For this reason, the compact function only has a generation rate of _64,000 / second / node_ and supports up to 64 total nodes
136+
in the same deployment.

0 commit comments

Comments
 (0)