Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion configuration/source-db/postgres-maintenance.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,20 @@

Inactive slots can be dropped using:

```bash
```sql
select slot_name, pg_drop_replication_slot(slot_name) from pg_replication_slots where active = false;

Check warning on line 31 in configuration/source-db/postgres-maintenance.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/postgres-maintenance.mdx#L31

Did you really mean 'slot_name'?
```

Postgres prevents active slots from being dropped. If it does happen (e.g. while a PowerSync instance is disconnected), PowerSync would automatically re-create the slot, and restart replication.

### WAL Slot Invalidation

Postgres can invalidate a replication slot when the amount of retained WAL data exceeds the `max_slot_wal_keep_size` limit. This is most likely to happen during a long-running [initial snapshot](/architecture/powersync-service#initial-replication-vs-incremental-replication). PowerSync must hold the slot open while copying your entire dataset, and WAL accumulates throughout that time.

If the slot is invalidated mid-snapshot, PowerSync detects the problem and stops with error [`PSYNC_S1146`](/debugging/error-codes#psync_s11xx-postgres-replication-issues) instead of finishing a snapshot that will fail. On the source database, increase `max_slot_wal_keep_size` and delete the existing replication slot. PowerSync creates a new slot and restarts the snapshot.

During a snapshot, PowerSync warns when less than 50% of the WAL budget remains. You may see this warning in the PowerSync dashboard, in the [Diagnostics API](/maintenance-ops/self-hosting/diagnostics) if you self-host, and in PowerSync Service logs. Increase `max_slot_wal_keep_size` or reduce snapshot work before the slot is invalidated. For how to choose a value, see [Managing & Monitoring Replication Lag](/maintenance-ops/production-readiness-guide#managing--monitoring-replication-lag).

### Maximum Replication Slots

Postgres is configured with a maximum number of replication slots per server. Since each PowerSync instance uses one replication slot for replication and an additional one while deploying a new Sync Streams/Rules version, the maximum number of PowerSync instances connected to one Postgres server is equal to the maximum number of replication slots, minus 1\.
Expand Down
29 changes: 28 additions & 1 deletion debugging/error-codes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

This reference documents PowerSync error codes organized by component, with troubleshooting suggestions for developers. Use the search bar to look up specific error codes (e.g., `PSYNC_R0001`).

# PSYNC_Rxxxx: Sync Rules issues

Check warning on line 9 in debugging/error-codes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

debugging/error-codes.mdx#L9

Did you really mean 'PSYNC_Rxxxx'?

- **PSYNC_R0001**:
Catch-all [Sync Rules](/sync/rules/overview) parsing error, if no more specific error is available
Expand All @@ -23,7 +23,7 @@

## PSYNC_R24xx: SQL security warnings

# PSYNC_Sxxxx: Service issues

Check warning on line 26 in debugging/error-codes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

debugging/error-codes.mdx#L26

Did you really mean 'PSYNC_Sxxxx'?

- **PSYNC_S0001**:
Internal assertion.
Expand Down Expand Up @@ -62,6 +62,11 @@

This may occur if there is very deep nesting in JSON or embedded documents.

- **PSYNC_S1005**:
Storage version not supported.

This could be caused by a downgrade to a version that does not support the current storage version.

## PSYNC_S11xx: Postgres replication issues

- **PSYNC_S1101**:
Expand Down Expand Up @@ -121,7 +126,7 @@
Create a publication using `WITH (publish = "insert, update, delete, truncate")` (the default).

- **PSYNC_S1143**:
Publication uses publish_via_partition_root.

Check warning on line 129 in debugging/error-codes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

debugging/error-codes.mdx#L129

Did you really mean 'publish_via_partition_root'?

- **PSYNC_S1144**:
Invalid Postgres server configuration for replication and sync bucket storage.
Expand All @@ -143,6 +148,17 @@
An alternative is to create explicit policies for the replication role. If you have done that,
you may ignore this warning.

- **PSYNC_S1146**:
Replication slot invalidated.

The replication slot was invalidated by PostgreSQL, typically because WAL retention exceeded `max_slot_wal_keep_size` during a long-running snapshot. Increase `max_slot_wal_keep_size` on the source database and delete the existing replication slot to recover. PowerSync will create a new slot and restart replication automatically.

Other causes: `rows_removed` (catalog rows needed by the slot were removed), `wal_level_insufficient`, or `idle_timeout`.

`idle_timeout` is a PostgreSQL 18+ slot invalidation, in this case increase `idle_replication_slot_timeout` instead of `max_slot_wal_keep_size`.

See [Managing & Monitoring Replication Lag](/maintenance-ops/production-readiness-guide#managing--monitoring-replication-lag) for guidance on sizing `max_slot_wal_keep_size`.

## PSYNC_S12xx: MySQL replication issues

## PSYNC_S13xx: MongoDB replication issues
Expand Down Expand Up @@ -200,7 +216,7 @@
The MongoDB Change Stream has been invalidated.

Possible causes:
- Some change stream documents do not have postImages.

Check warning on line 219 in debugging/error-codes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

debugging/error-codes.mdx#L219

Did you really mean 'postImages'?
- startAfter/resumeToken is not valid anymore.
- The replication connection has changed.
- The database has been dropped.
Expand Down Expand Up @@ -235,6 +251,17 @@
Possible causes:
- Older data has been cleaned up due to exceeding the retention period.

## PSYNC_S16xx: MSSQL replication issues

- **PSYNC_S1601**:
A replicated source table's capture instance has been dropped during a polling cycle.

Possible causes:
- CDC has been disabled for the table.
- The table has been dropped, which also drops the capture instance.

Replication for the table will only resume once CDC has been re-enabled for the table.

## PSYNC_S2xxx: Service API

- **PSYNC_S2001**:
Expand Down Expand Up @@ -264,15 +291,15 @@

Common causes:
1. **JWT signing key mismatch** (Supabase): The client is using tokens signed with a different key type (legacy vs. new JWT signing keys) than PowerSync expects. If you've migrated to new JWT signing keys, ensure users sign out and back in to get fresh tokens. See [Migrating from Legacy to New JWT Signing Keys](/installation/authentication-setup/supabase-auth#migrating-from-legacy-to-new-jwt-signing-keys).
2. **Missing or invalid key ID (kid)**: The token's kid header doesn't match any keys in PowerSync's keystore.

Check warning on line 294 in debugging/error-codes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

debugging/error-codes.mdx#L294

Did you really mean 'keystore'?
3. **Incorrect JWT secret or JWKS endpoint**: Verify your authentication configuration matches your auth provider's settings.

- **PSYNC_S2102**:
Could not verify the auth token signature.

Typical causes include:
1. Token kid is not found in the keystore.

Check warning on line 301 in debugging/error-codes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

debugging/error-codes.mdx#L301

Did you really mean 'keystore'?
2. Signature does not match the kid in the keystore.

Check warning on line 302 in debugging/error-codes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

debugging/error-codes.mdx#L302

Did you really mean 'keystore'?

- **PSYNC_S2103**:
Token has expired. Check the expiry date on the token.
Expand Down Expand Up @@ -303,7 +330,7 @@
- **PSYNC_S2203**:
IPs in this range are not supported.

Make sure to use a publically-accessible JWKS URI.
Make sure to use a publicly-accessible JWKS URI.

- **PSYNC_S2204**:
JWKS request failed.
Expand Down
18 changes: 9 additions & 9 deletions maintenance-ops/production-readiness-guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
```typescript App Entry Point
createRoot(document.getElementById("root")!,
{
onUncaughtError: Sentry.reactErrorHandler((error, errorInfo) => {

Check warning on line 57 in maintenance-ops/production-readiness-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

maintenance-ops/production-readiness-guide.mdx#L57

Did you really mean 'errorInfo'?
console.warn('Uncaught error', error, errorInfo.componentStack);
}),
// Callback called when React catches an error in an ErrorBoundary.
Expand Down Expand Up @@ -262,7 +262,7 @@

The WAL growth rate is expected to increase substantially during the initial replication of large datasets with high update frequency, particularly for tables included in the PowerSync publication.

During normal operation (after Sync Streams (or legacy Sync Rules) are deployed) the WAL growth rate is much smaller than the initial replication period, since the PowerSync Service can replicate ~5k operations per second, meaning the WAL lag is typically in the MB range as opposed to the GB range.
During normal operation (after Sync Streams/Sync Rules are deployed) the WAL growth rate is much smaller than the initial replication period, since the PowerSync Service can replicate ~5k operations per second, meaning the WAL lag is typically in the MB range as opposed to the GB range.

When deciding what to set the `max_slot_wal_keep_size` configuration parameter the following should be taken in account:
1. Database size - This impacts the time it takes to complete the initial replication from the source Postgres database.
Expand All @@ -271,7 +271,7 @@

To view the current replication slots that are being used by PowerSync you can run the following query:

```
```sql
SELECT slot_name,
plugin,
slot_type,
Expand All @@ -281,14 +281,14 @@
```

To view the current configured value of the `max_slot_wal_keep_size` you can run the following query:

```sql
SHOW max_slot_wal_keep_size
```
SELECT setting as max_slot_wal_keep_size
FROM pg_settings
WHERE name = 'max_slot_wal_keep_size'
```

It's recommended to check the current replication slot lag and `max_slot_wal_keep_size` when deploying Sync Streams/Sync Rules changes to your PowerSync Service instance, especially when you're working with large database volumes.
If you notice that the replication lag is greater than the current `max_slot_wal_keep_size` it's recommended to increase value of the `max_slot_wal_keep_size` on the connected source Postgres database to accommodate for the lag and to ensure the PowerSync Service can complete initial replication without further delays.
If the slot is invalidated mid-snapshot, PowerSync detects the problem and stops replication with error [`PSYNC_S1146`](/debugging/error-codes#psync_s11xx-postgres-replication-issues). On the source database, increase `max_slot_wal_keep_size` and delete the existing replication slot. PowerSync creates a new slot and restarts the snapshot.

During a snapshot, PowerSync warns when less than 50% of the WAL budget remains. You may see this warning in the PowerSync dashboard, in the [Diagnostics API](/maintenance-ops/self-hosting/diagnostics) if you self-host, and in PowerSync Service logs. Increase `max_slot_wal_keep_size` or reduce snapshot work before the slot is invalidated. Use the considerations above to set a high enough cap.

### Managing Replication Slots

Expand All @@ -311,7 +311,7 @@

The alternative to manually checking for inactive replication slots would be to configure the `idle_replication_slot_timeout` configuration parameter on the source Postgres database.

<Note>The `idle_replication_slot_timeout` [configuration parameter](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-IDLE-REPLICATION-SLOT-TIMEOUT) is only available from PostgresSQL 18 and above.</Note>
<Note>The `idle_replication_slot_timeout` [configuration parameter](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-IDLE-REPLICATION-SLOT-TIMEOUT) is only available from Postgres 18 onward.</Note>

Check warning on line 314 in maintenance-ops/production-readiness-guide.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

maintenance-ops/production-readiness-guide.mdx#L314

Did you really mean 'idle_replication_slot_timeout'?

The `idle_replication_slot_timeout` will invalidate replication slots that have remained inactive for longer than the value set for the `idle_replication_slot_timeout` parameter.

Expand Down
37 changes: 26 additions & 11 deletions maintenance-ops/self-hosting/diagnostics.mdx
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
---
title: "Diagnostics"
description: "How to use the PowerSync Service Diagnostics API"
description: "How to use the PowerSync Service Diagnostics API to inspect replication status, errors, and slot health."
---

All self-hosted PowerSync Service instances ship with a Diagnostics API.
This API provides the following diagnostic information:

- Connections → Connected backend source database and any active errors associated with the connection.
- Active Sync Streams / Sync Rules → Currently deployed Sync Streams (or legacy Sync Rules) and its status.
All self-hosted PowerSync Service instances ship with a Diagnostics API for inspecting replication state, surfacing errors, and monitoring source database health.

## CLI

Expand All @@ -17,27 +13,46 @@ If you have the [PowerSync CLI](/tools/cli) installed, use `powersync status` to
powersync status

# Extract a specific field
powersync status --output=json | jq '.connections[0]'
powersync status --output=json | jq '.data.active_sync_rules'
```

## Diagnostics API

# Configuration
### Configuration

1. To enable the Diagnostics API, specify an API token in your PowerSync YAML file:
1. Specify an API token in your PowerSync YAML file:

```yaml service.yaml
api:
tokens:
- YOUR_API_TOKEN
```
<Warning>Make sure to use a secure API token as part of this configuration</Warning>

<Warning>Use a secure, randomly generated API token.</Warning>

2. Restart the PowerSync Service.

3. Once configured, send an HTTP request to your PowerSync Service Diagnostics API endpoint. Include the API token set in step 1 as a Bearer token in the Authorization header.
3. Send a POST request to the diagnostics endpoint, passing the token as a Bearer token:

```shell
curl -X POST http://localhost:8080/api/admin/v1/diagnostics \
-H "Authorization: Bearer YOUR_API_TOKEN"
```

### Response structure

The response `data` object contains information about:

**`connections`** — whether PowerSync can reach the configured source database, and any connection-level errors.

**`active_sync_rules`** — the currently serving sync config (Sync Streams/Sync Rules). Shows which replication slot is in use, whether initial replication has completed, which tables are being replicated, and any replication lag or errors.

**`deploying_sync_rules`** — only present while a new sync config is being deployed and the initial replication is in progress. PowerSync runs this process in parallel so clients continue to be served by the existing active config. Once initial replication completes, this section disappears and `active_sync_rules` updates. Errors during initial replication surface here rather than in `active_sync_rules`.

From version 1.20.5 of the PowerSync Service, each connection entry under `active_sync_rules` includes WAL information so you can see how much budget remains before Postgres could invalidate the replication slot:

- `wal_status` — slot status from `pg_replication_slots` (Postgres 13+)
- `safe_wal_size` — bytes remaining before potential invalidation
- `max_slot_wal_keep_size` — configured limit in bytes

This information is especially useful when you deploy a new sync config against a large database. When less than 50% of the WAL budget remains during a snapshot, PowerSync adds a warning to the response. If the slot is invalidated, the response includes error [`PSYNC_S1146`](/debugging/error-codes#psync_s11xx-postgres-replication-issues).