Skip to content

Commit deb512f

Browse files
danielgerlagdanielgerlag
committed
mysql
Signed-off-by: Daniel Gerlag <daniel@gerlag.ca>
1 parent 9ac0d3a commit deb512f

4 files changed

Lines changed: 378 additions & 0 deletions

File tree

  • docs/content/drasi-server/how-to-guides/configuration

‎docs/content/drasi-server/how-to-guides/configuration/configure-bootstrap-providers/_index.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,15 @@ description: "Load initial data before streaming begins"
3232
</div>
3333
</div>
3434
</a> -->
35+
<a href="configure-mysql-bootstrap-provider/">
36+
<div class="unified-card unified-card--howto">
37+
<div class="unified-card-icon"><i class="fas fa-database"></i></div>
38+
<div class="unified-card-content">
39+
<h3 class="unified-card-title">MySQL</h3>
40+
<p class="unified-card-summary">Bootstrap from a MySQL snapshot (MySQL sources only)</p>
41+
</div>
42+
</div>
43+
</a>
3544
<a href="configure-postgres-bootstrap-provider/">
3645
<div class="unified-card unified-card--howto">
3746
<div class="unified-card-icon"><i class="fas fa-database"></i></div>

‎docs/content/drasi-server/how-to-guides/configuration/configure-bootstrap-providers/configure-mysql-bootstrap-provider/_index.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
---
2+
type: "docs"
3+
title: "Configure MySQL Bootstrap Provider"
4+
linkTitle: "MySQL"
5+
weight: 45
6+
description: "Bootstrap queries from a MySQL snapshot"
7+
related:
8+
howto:
9+
- title: "Configure Sources"
10+
url: "/drasi-server/how-to-guides/configuration/configure-sources/"
11+
- title: "Configure MySQL Source"
12+
url: "/drasi-server/how-to-guides/configuration/configure-sources/configure-mysql-source/"
13+
- title: "Configure Bootstrap Providers"
14+
url: "/drasi-server/how-to-guides/configuration/configure-bootstrap-providers/"
15+
---
16+
17+
The **MySQL bootstrap provider** loads initial state from a MySQL database so queries start with a complete snapshot before streaming begins.
18+
19+
## When to use MySQL bootstrap
20+
21+
- You need historical/current state from MySQL when a query starts.
22+
- Your query depends on existing rows (aggregations, joins, thresholds).
23+
- You want snapshot + binlog streaming from the same database.
24+
25+
## Prerequisites
26+
27+
- Use with a **MySQL source** (`sources[].kind: mysql`).
28+
- The database user must have **SELECT** permission on the configured tables.
29+
- The `tables` list is **required** — you must explicitly specify which tables to bootstrap.
30+
31+
## Quick example (Drasi Server config)
32+
33+
In Drasi Server config, bootstrap provider keys are **camelCase**, and the discriminator field is `bootstrapProvider.kind`.
34+
35+
```yaml
36+
sources:
37+
- kind: mysql
38+
id: orders-db
39+
autoStart: true
40+
41+
host: ${MYSQL_HOST:-localhost}
42+
port: ${MYSQL_PORT:-3306}
43+
database: ${MYSQL_DATABASE:-mydb}
44+
user: ${MYSQL_USER:-drasi_user}
45+
password: ${MYSQL_PASSWORD}
46+
47+
tables:
48+
- orders
49+
- customers
50+
51+
bootstrapProvider:
52+
kind: mysql
53+
```
54+
55+
## Configuration reference
56+
57+
| Field | Type | Required | Description |
58+
|---|---|---:|---|
59+
| `kind` | string | Yes | Must be `mysql`. |
60+
| `host` | string | No | MySQL host (default: `localhost`). |
61+
| `port` | integer | No | MySQL port (default: `3306`). |
62+
| `database` | string | Yes | Database name to connect to. |
63+
| `user` | string | Yes | Database user with SELECT permission. |
64+
| `password` | string | No | Password (default: `""`). |
65+
| `tables` | string[] | Yes | Table allow-list for bootstrapping. Must contain at least one table. |
66+
| `tableKeys` | array | No | Override key columns per table (see below). |
67+
68+
### tableKeys
69+
70+
Use `tableKeys` to define key columns for element ID generation when primary keys are missing.
71+
72+
```yaml
73+
bootstrapProvider:
74+
kind: mysql
75+
tableKeys:
76+
- table: order_items
77+
keyColumns: [order_id, product_id]
78+
```
79+
80+
## Notes
81+
82+
- Drasi Server only allows `kind: mysql` when the source is also `kind: mysql`.
83+
- The `tables` list acts as a security allow-list — only tables explicitly listed will be bootstrapped.
84+
- Table names must use only letters, numbers, and underscores.
85+
86+
## Documentation resources
87+
88+
<div class="card-grid card-grid--2">
89+
<a href="https://github.com/drasi-project/drasi-core/blob/main/components/bootstrappers/mysql/README.md" target="_blank" rel="noopener">
90+
<div class="unified-card unified-card--tutorials">
91+
<div class="unified-card-icon"><i class="fab fa-github"></i></div>
92+
<div class="unified-card-content">
93+
<h3 class="unified-card-title">MySQL Bootstrap README</h3>
94+
<p class="unified-card-summary">Implementation notes and behavior details</p>
95+
</div>
96+
</div>
97+
</a>
98+
<a href="https://crates.io/crates/drasi-bootstrap-mysql" target="_blank" rel="noopener">
99+
<div class="unified-card unified-card--howto">
100+
<div class="unified-card-icon"><i class="fas fa-box"></i></div>
101+
<div class="unified-card-content">
102+
<h3 class="unified-card-title">drasi-bootstrap-mysql on crates.io</h3>
103+
<p class="unified-card-summary">Package info and release history</p>
104+
</div>
105+
</div>
106+
</a>
107+
</div>

‎docs/content/drasi-server/how-to-guides/configuration/configure-sources/_index.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,15 @@ description: "Connect Drasi Server to databases, APIs, and data streams"
4848
</div>
4949
</div>
5050
</a> -->
51+
<a href="configure-mysql-source/">
52+
<div class="unified-card unified-card--howto">
53+
<div class="unified-card-icon"><i class="fas fa-database"></i></div>
54+
<div class="unified-card-content">
55+
<h3 class="unified-card-title">MySQL</h3>
56+
<p class="unified-card-summary">Stream changes from MySQL using binlog replication</p>
57+
</div>
58+
</div>
59+
</a>
5160
<a href="configure-postgresql-source/">
5261
<div class="unified-card unified-card--howto">
5362
<div class="unified-card-icon"><i class="fas fa-database"></i></div>

‎docs/content/drasi-server/how-to-guides/configuration/configure-sources/configure-mysql-source/_index.md‎

Lines changed: 253 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,253 @@
1+
---
2+
type: "docs"
3+
title: "Configure MySQL Source"
4+
linkTitle: "MySQL"
5+
weight: 55
6+
description: "Stream changes from MySQL using binlog replication"
7+
related:
8+
concepts:
9+
- title: "Sources"
10+
url: "/concepts/sources/"
11+
howto:
12+
- title: "Configure Bootstrap Providers"
13+
url: "/drasi-server/how-to-guides/configuration/configure-bootstrap-providers/"
14+
reference:
15+
- title: "Configuration Reference"
16+
url: "/drasi-server/reference/configuration/"
17+
---
18+
19+
The MySQL {{< term "Source" >}} streams row-level changes from a MySQL database using **binary log (binlog) replication**.
20+
21+
## When to use the MySQL source
22+
23+
- Keep Drasi queries continuously updated from a system-of-record MySQL database.
24+
- Drive reactions from database changes (alerts, notifications, downstream sync, cache/materialized-view updates).
25+
- Build reactive services that need transactional ordering of changes.
26+
27+
## Prerequisites
28+
29+
- MySQL **5.7+** or **8.0+**.
30+
- Binary logging enabled with row-based format:
31+
- `binlog_format = ROW`
32+
- `binlog_row_image = FULL`
33+
- `binlog_row_metadata = FULL`
34+
- A database user with **REPLICATION SLAVE**, **REPLICATION CLIENT**, and **SELECT** permissions.
35+
36+
## How it connects
37+
38+
This source **connects outbound** from Drasi Server to MySQL over the MySQL protocol; it does not open an inbound port.
39+
40+
## Quick example (Drasi Server config)
41+
42+
Drasi Server source configuration uses **camelCase** keys.
43+
44+
```yaml
45+
sources:
46+
- kind: mysql
47+
id: orders-db
48+
autoStart: true
49+
50+
host: ${MYSQL_HOST:-localhost}
51+
port: ${MYSQL_PORT:-3306}
52+
database: ${MYSQL_DATABASE:-mydb}
53+
user: ${MYSQL_USER:-drasi_user}
54+
password: ${MYSQL_PASSWORD}
55+
56+
# Tables to monitor
57+
tables:
58+
- orders
59+
- customers
60+
61+
# Optional: override key columns for tables without primary keys
62+
tableKeys:
63+
- table: order_items
64+
keyColumns: [order_id, product_id]
65+
66+
# Optional: where to start reading the binlog
67+
startPosition: from_end
68+
69+
# Optional: preload initial state for newly-subscribed queries
70+
bootstrapProvider:
71+
kind: mysql
72+
```
73+
74+
## Configure MySQL
75+
76+
### 1) Enable binary logging with row format
77+
78+
Add to your MySQL configuration (`my.cnf` or `my.ini`) and restart MySQL:
79+
80+
```ini
81+
[mysqld]
82+
log-bin = mysql-bin
83+
binlog_format = ROW
84+
binlog_row_image = FULL
85+
binlog_row_metadata = FULL
86+
server-id = 1
87+
```
88+
89+
### 2) Create a replication user
90+
91+
```sql
92+
CREATE USER 'drasi_user'@'%' IDENTIFIED BY 'your-password';
93+
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'drasi_user'@'%';
94+
GRANT SELECT ON mydb.* TO 'drasi_user'@'%';
95+
FLUSH PRIVILEGES;
96+
```
97+
98+
### 3) (Recommended) Ensure tables have primary keys
99+
100+
For reliable change tracking, tables should have primary keys.
101+
If replicating tables without primary keys, configure `tableKeys` (below).
102+
103+
## Data mapping
104+
105+
- Each changed row becomes a Drasi graph {{< term "Node" >}}.
106+
- **Label**: table name (for example `orders`).
107+
- **Properties**: columns become node properties.
108+
- **Element ID**: `table:key` (for example `orders:123`).
109+
110+
If no key columns can be resolved for a row, the source logs a warning and falls back to a generated UUID: `table:uuid`.
111+
112+
## Configuration reference (Drasi Server)
113+
114+
| Field | Type | Default | Description |
115+
|---|---:|---:|---|
116+
| `kind` | string | required | Must be `mysql`. |
117+
| `id` | string | required | Unique source identifier. |
118+
| `autoStart` | boolean | `true` | Whether Drasi Server starts the source on startup. |
119+
| `bootstrapProvider` | object | none | Optional bootstrap provider for initial state. For MySQL bootstrap, use `{ kind: mysql }`. |
120+
| `host` | string | `localhost` | MySQL host. |
121+
| `port` | integer | `3306` | MySQL port. |
122+
| `database` | string | required | Database name. |
123+
| `user` | string | required | Database user (must have replication permission). |
124+
| `password` | string | `""` | Password. |
125+
| `tables` | string[] | `[]` | List of tables to monitor for changes. |
126+
| `sslMode` | string | `disabled` | SSL mode: `disabled`, `if_available`, `require`, `require_verify_ca`, `require_verify_full`. |
127+
| `tableKeys` | array | `[]` | Override key columns per table (see below). |
128+
| `startPosition` | string | `from_end` | Where to start the binlog stream: `from_start`, `from_end`, `from_position`, `from_gtid`. |
129+
| `serverId` | integer | auto-generated | MySQL server ID for the replication connection. Auto-generated from source instance ID if not specified. |
130+
| `heartbeatIntervalSeconds` | integer | `30` | Heartbeat interval in seconds for the replication connection. |
131+
132+
Fields marked with support Drasi Server config references like `${ENV_VAR}` / `${ENV_VAR:-default}`.
133+
134+
### tableKeys
135+
136+
Use `tableKeys` to define key columns for element ID generation when primary keys are missing or not suitable.
137+
138+
```yaml
139+
tableKeys:
140+
- table: order_items
141+
keyColumns: [order_id, product_id]
142+
```
143+
144+
Notes:
145+
- Key columns are joined with `_` in the element ID (for example `order_items:1001_5`).
146+
147+
### startPosition
148+
149+
Controls where binlog replication begins when the source starts for the first time.
150+
151+
| Value | Description |
152+
|---|---|
153+
| `from_end` | Start from the current end of the binlog (default). Only captures new changes. |
154+
| `from_start` | Replay the binlog from the beginning. |
155+
| `from_position` | Start from a specific binlog file and position. |
156+
| `from_gtid` | Start from a specific GTID set. |
157+
158+
## Performance tuning notes
159+
160+
- Only list the tables you need in `tables`; this reduces processing overhead.
161+
- The `heartbeatIntervalSeconds` prevents idle connection timeouts and allows the source to detect disconnections faster.
162+
- Monitor binlog retention to ensure the source can reconnect after brief outages.
163+
164+
## Verifying it works
165+
166+
After starting Drasi Server with your MySQL source, verify the connection:
167+
168+
### 1. Check source status
169+
170+
```bash
171+
curl http://localhost:8080/api/v1/sources/orders-db
172+
```
173+
174+
Expected response includes `"status": "running"`.
175+
176+
### 2. Make a test change in MySQL
177+
178+
```sql
179+
INSERT INTO orders (id, customer_id, total, status)
180+
VALUES (999, 1, 100.00, 'pending');
181+
```
182+
183+
### 3. Verify the change was captured
184+
185+
If you have a log reaction configured:
186+
187+
```
188+
[console-output] Query 'my-query' (1 items):
189+
[console-output] [ADD] {"id":"999","customer_id":"1","total":"100.00","status":"pending"}
190+
```
191+
192+
Or query results via API:
193+
194+
```bash
195+
curl http://localhost:8080/api/v1/queries/my-query/results
196+
```
197+
198+
### 4. Check binlog status in MySQL
199+
200+
```sql
201+
SHOW BINARY LOG STATUS;
202+
SHOW REPLICAS;
203+
```
204+
205+
## Troubleshooting
206+
207+
| Error | Cause | Solution |
208+
|-------|-------|----------|
209+
| `binlog_format must be ROW` | Binlog format not set to ROW | Set `binlog_format = ROW` in `my.cnf` and restart |
210+
| `Access denied; you need the REPLICATION SLAVE privilege` | Missing permission | `GRANT REPLICATION SLAVE ON *.* TO 'drasi_user'@'%';` |
211+
| `Can't connect to MySQL server` | Wrong host/port | Verify `host` and `port` values, check firewall |
212+
| No changes captured | Table not in `tables` list | Add the table to the `tables` configuration |
213+
| Unstable element IDs | No primary key | Add primary key or configure `tableKeys` |
214+
| `binlog_row_image must be FULL` | Row image not complete | Set `binlog_row_image = FULL` in `my.cnf` and restart |
215+
216+
### MySQL Permissions Checklist
217+
218+
Your database user needs:
219+
220+
```sql
221+
-- Required permissions
222+
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'drasi_user'@'%';
223+
GRANT SELECT ON mydb.* TO 'drasi_user'@'%';
224+
FLUSH PRIVILEGES;
225+
```
226+
227+
## Known limitations
228+
229+
- Packets larger than 16 MB are not supported.
230+
- DDL changes (schema migrations) during streaming may require a source restart.
231+
232+
## Documentation resources
233+
234+
<div class="card-grid card-grid--2">
235+
<a href="https://github.com/drasi-project/drasi-core/blob/main/components/sources/mysql/README.md" target="_blank" rel="noopener">
236+
<div class="unified-card unified-card--tutorials">
237+
<div class="unified-card-icon"><i class="fab fa-github"></i></div>
238+
<div class="unified-card-content">
239+
<h3 class="unified-card-title">MySQL Source README</h3>
240+
<p class="unified-card-summary">Implementation notes, prerequisites, and behavior details</p>
241+
</div>
242+
</div>
243+
</a>
244+
<a href="https://crates.io/crates/drasi-source-mysql" target="_blank" rel="noopener">
245+
<div class="unified-card unified-card--howto">
246+
<div class="unified-card-icon"><i class="fas fa-box"></i></div>
247+
<div class="unified-card-content">
248+
<h3 class="unified-card-title">drasi-source-mysql on crates.io</h3>
249+
<p class="unified-card-summary">Package info and release history</p>
250+
</div>
251+
</div>
252+
</a>
253+
</div>

0 commit comments

Comments
 (0)