-
Notifications
You must be signed in to change notification settings - Fork 15
Add MySQL provider documentation and configuration #217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
danielgerlag
wants to merge
3
commits into
drasi-project:main
Choose a base branch
from
danielgerlag:mysql
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+362
−0
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
91 changes: 91 additions & 0 deletions
91
...tion/configure-bootstrap-providers/configure-mysql-bootstrap-provider/_index.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| --- | ||
| type: "docs" | ||
| title: "Configure MySQL Bootstrap Provider" | ||
| linkTitle: "MySQL" | ||
| weight: 45 | ||
| description: "Bootstrap queries from a MySQL snapshot" | ||
| related: | ||
| howto: | ||
| - title: "Configure Sources" | ||
| url: "/drasi-server/how-to-guides/configuration/configure-sources/" | ||
| - title: "Configure MySQL Source" | ||
| url: "/drasi-server/how-to-guides/configuration/configure-sources/configure-mysql-source/" | ||
| - title: "Configure Bootstrap Providers" | ||
| url: "/drasi-server/how-to-guides/configuration/configure-bootstrap-providers/" | ||
| --- | ||
|
|
||
| The **MySQL bootstrap provider** loads initial state from a MySQL database so queries start with a complete snapshot before streaming begins. | ||
|
|
||
| ## When to use MySQL bootstrap | ||
|
|
||
| - You need historical/current state from MySQL when a query starts. | ||
| - Your query depends on existing rows (aggregations, joins, thresholds). | ||
| - You want snapshot + binlog streaming from the same database. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Use with a **MySQL source** (`sources[].kind: mysql`). | ||
| - The database user must have **SELECT** permission on the configured tables. | ||
| - Configure `tables` (and optional `tableKeys`) on the MySQL source. | ||
|
|
||
| ## Quick example (Drasi Server config) | ||
|
|
||
| In Drasi Server config, bootstrap provider keys are **camelCase**, and the discriminator field is `bootstrapProvider.kind`. | ||
|
|
||
| ```yaml | ||
| sources: | ||
| - kind: mysql | ||
| id: orders-db | ||
| autoStart: true | ||
|
|
||
| host: ${MYSQL_HOST:-localhost} | ||
| port: ${MYSQL_PORT:-3306} | ||
| database: ${MYSQL_DATABASE:-mydb} | ||
| user: ${MYSQL_USER:-drasi_user} | ||
| password: ${MYSQL_PASSWORD} | ||
|
|
||
| tables: | ||
| - orders | ||
| - customers | ||
|
|
||
| bootstrapProvider: | ||
| kind: mysql | ||
| ``` | ||
|
|
||
| ## Configuration reference | ||
|
|
||
| `mysql` accepts **no additional fields**. | ||
|
|
||
| | Field | Type | Required | Description | | ||
| |---|---|---:|---| | ||
| | `kind` | string | Yes | Must be `mysql`. | | ||
|
Comment on lines
+59
to
+61
|
||
|
|
||
| ## Notes | ||
|
|
||
| - Drasi Server only allows `kind: mysql` when the source is also `kind: mysql`. | ||
| - Connection and table scope come from the MySQL source configuration (for example `host`, `database`, `tables`, and `tableKeys`). | ||
| - The `tables` list acts as a security allow-list — only tables explicitly listed will be bootstrapped. | ||
| - Table names must use only letters, numbers, and underscores. | ||
|
|
||
| ## Documentation resources | ||
|
|
||
| <div class="card-grid card-grid--2"> | ||
| <a href="https://github.com/drasi-project/drasi-core/blob/main/components/bootstrappers/mysql/README.md" target="_blank" rel="noopener"> | ||
| <div class="unified-card unified-card--tutorials"> | ||
| <div class="unified-card-icon"><i class="fab fa-github"></i></div> | ||
| <div class="unified-card-content"> | ||
| <h3 class="unified-card-title">MySQL Bootstrap README</h3> | ||
| <p class="unified-card-summary">Implementation notes and behavior details</p> | ||
| </div> | ||
| </div> | ||
| </a> | ||
| <a href="https://crates.io/crates/drasi-bootstrap-mysql" target="_blank" rel="noopener"> | ||
| <div class="unified-card unified-card--howto"> | ||
| <div class="unified-card-icon"><i class="fas fa-box"></i></div> | ||
| <div class="unified-card-content"> | ||
| <h3 class="unified-card-title">drasi-bootstrap-mysql on crates.io</h3> | ||
| <p class="unified-card-summary">Package info and release history</p> | ||
| </div> | ||
| </div> | ||
| </a> | ||
| </div> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
252 changes: 252 additions & 0 deletions
252
.../how-to-guides/configuration/configure-sources/configure-mysql-source/_index.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,252 @@ | ||
| --- | ||
| type: "docs" | ||
| title: "Configure MySQL Source" | ||
| linkTitle: "MySQL" | ||
| weight: 55 | ||
| description: "Stream changes from MySQL using binlog replication" | ||
| related: | ||
| concepts: | ||
| - title: "Sources" | ||
| url: "/concepts/sources/" | ||
| howto: | ||
| - title: "Configure Bootstrap Providers" | ||
| url: "/drasi-server/how-to-guides/configuration/configure-bootstrap-providers/" | ||
| reference: | ||
| - title: "Configuration Reference" | ||
| url: "/drasi-server/reference/configuration/" | ||
| --- | ||
|
|
||
| The MySQL {{< term "Source" >}} streams row-level changes from a MySQL database using **binary log (binlog) replication**. | ||
|
|
||
| ## When to use the MySQL source | ||
|
|
||
| - Keep Drasi queries continuously updated from a system-of-record MySQL database. | ||
| - Drive reactions from database changes (alerts, notifications, downstream sync, cache/materialized-view updates). | ||
| - Build reactive services that need transactional ordering of changes. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - MySQL **8.0+**. | ||
| - Binary logging enabled with row-based format: | ||
| - `binlog_format = ROW` | ||
| - `binlog_row_image = FULL` | ||
| - `binlog_row_metadata = FULL` | ||
| - A database user with **REPLICATION SLAVE**, **REPLICATION CLIENT**, and **SELECT** permissions. | ||
|
|
||
| ## How it connects | ||
|
|
||
| This source **connects outbound** from Drasi Server to MySQL over the MySQL protocol; it does not open an inbound port. | ||
|
|
||
| ## Quick example (Drasi Server config) | ||
|
|
||
| Drasi Server source configuration uses **camelCase** keys. | ||
|
|
||
| ```yaml | ||
| sources: | ||
| - kind: mysql | ||
| id: orders-db | ||
| autoStart: true | ||
|
|
||
| host: ${MYSQL_HOST:-localhost} | ||
| port: ${MYSQL_PORT:-3306} | ||
| database: ${MYSQL_DATABASE:-mydb} | ||
| user: ${MYSQL_USER:-drasi_user} | ||
| password: ${MYSQL_PASSWORD} | ||
|
|
||
| # Tables to monitor | ||
| tables: | ||
| - orders | ||
| - customers | ||
|
|
||
| # Optional: override key columns for tables without primary keys | ||
| tableKeys: | ||
| - table: order_items | ||
| keyColumns: [order_id, product_id] | ||
|
|
||
| # Optional: where to start reading the binlog | ||
| startPosition: from_end | ||
|
|
||
| # Optional: preload initial state for newly-subscribed queries | ||
| bootstrapProvider: | ||
| kind: mysql | ||
| ``` | ||
|
|
||
| ## Configure MySQL | ||
|
|
||
| ### 1) Enable binary logging with row format | ||
|
|
||
| Add to your MySQL configuration (`my.cnf` or `my.ini`) and restart MySQL: | ||
|
|
||
| ```ini | ||
| [mysqld] | ||
| log-bin = mysql-bin | ||
| binlog_format = ROW | ||
| binlog_row_image = FULL | ||
| binlog_row_metadata = FULL | ||
| server-id = 1 | ||
| ``` | ||
|
|
||
| ### 2) Create a replication user | ||
|
|
||
| ```sql | ||
| CREATE USER 'drasi_user'@'%' IDENTIFIED BY 'your-password'; | ||
| GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'drasi_user'@'%'; | ||
| GRANT SELECT ON mydb.* TO 'drasi_user'@'%'; | ||
| FLUSH PRIVILEGES; | ||
| ``` | ||
|
|
||
| ### 3) (Recommended) Ensure tables have primary keys | ||
|
|
||
| For reliable change tracking, tables should have primary keys. | ||
| If replicating tables without primary keys, configure `tableKeys` (below). | ||
|
|
||
| ## Data mapping | ||
|
|
||
| - Each changed row becomes a Drasi graph {{< term "Node" >}}. | ||
| - **Label**: table name (for example `orders`). | ||
| - **Properties**: columns become node properties. | ||
| - **Element ID**: `table:key` (for example `orders:123`). | ||
|
|
||
| If no key columns can be resolved for a row, the source logs a warning and falls back to a generated UUID: `table:uuid`. | ||
|
|
||
| ## Configuration reference (Drasi Server) | ||
|
|
||
| | Field | Type | Default | Description | | ||
| |---|---:|---:|---| | ||
| | `kind` | string | required | Must be `mysql`. | | ||
|
Comment on lines
+114
to
+116
|
||
| | `id` | string | required | Unique source identifier. | | ||
| | `autoStart` | boolean | `true` | Whether Drasi Server starts the source on startup. | | ||
| | `bootstrapProvider` | object | none | Optional bootstrap provider for initial state. For MySQL bootstrap, use `{ kind: mysql }`. | | ||
| | `host` | string | `localhost` | MySQL host. | | ||
| | `port` | integer | `3306` | MySQL port. | | ||
| | `database` | string | required | Database name. | | ||
| | `user` | string | required | Database user (must have replication permission). | | ||
| | `password` | string | `""` | Password. | | ||
| | `tables` | string[] | `[]` | List of tables to monitor for changes. | | ||
| | `sslMode` | string | `disabled` | SSL mode: `disabled`, `if_available`, `require`, `require_verify_ca`, `require_verify_full`. | | ||
| | `tableKeys` | array | `[]` | Override key columns per table (see below). | | ||
| | `startPosition` | string | `from_end` | Where to start the binlog stream: `from_start`, `from_end`, `from_position`, `from_gtid`. | | ||
| | `serverId` | integer | auto-generated | MySQL server ID for the replication connection. Auto-generated from source instance ID if not specified. | | ||
| | `heartbeatIntervalSeconds` | integer | `30` | Heartbeat interval in seconds for the replication connection. | | ||
|
|
||
| Fields support Drasi Server config references like `${ENV_VAR}` / `${ENV_VAR:-default}`. | ||
|
|
||
| ### tableKeys | ||
|
|
||
| Use `tableKeys` to define key columns for element ID generation when primary keys are missing or not suitable. | ||
|
|
||
| ```yaml | ||
| tableKeys: | ||
| - table: order_items | ||
| keyColumns: [order_id, product_id] | ||
| ``` | ||
|
|
||
| Notes: | ||
| - Key columns are joined with `_` in the element ID (for example `order_items:1001_5`). | ||
|
|
||
| ### startPosition | ||
|
|
||
| Controls where binlog replication begins when the source starts for the first time. | ||
|
|
||
| | Value | Description | | ||
| |---|---| | ||
| | `from_end` | Start from the current end of the binlog (default). Only captures new changes. | | ||
|
danielgerlag marked this conversation as resolved.
|
||
| | `from_start` | Replay the binlog from the beginning. | | ||
| | `from_position` | Start from a specific binlog file and position. | | ||
| | `from_gtid` | Start from a specific GTID set. | | ||
|
|
||
| ## Performance tuning notes | ||
|
|
||
| - Only list the tables you need in `tables`; this reduces processing overhead. | ||
| - The `heartbeatIntervalSeconds` prevents idle connection timeouts and allows the source to detect disconnections faster. | ||
| - Monitor binlog retention to ensure the source can reconnect after brief outages. | ||
|
|
||
| ## Verifying it works | ||
|
|
||
| After starting Drasi Server with your MySQL source, verify the connection: | ||
|
|
||
| ### 1. Check source status | ||
|
|
||
| ```bash | ||
| curl http://localhost:8080/api/v1/sources/orders-db | ||
| ``` | ||
|
|
||
| Expected response includes `"status": "running"`. | ||
|
|
||
| ### 2. Make a test change in MySQL | ||
|
|
||
| ```sql | ||
| INSERT INTO orders (id, customer_id, total, status) | ||
| VALUES (999, 1, 100.00, 'pending'); | ||
| ``` | ||
|
|
||
| ### 3. Verify the change was captured | ||
|
|
||
| If you have a log reaction configured: | ||
|
|
||
| ``` | ||
| [console-output] Query 'my-query' (1 items): | ||
| [console-output] [ADD] {"id":"999","customer_id":"1","total":"100.00","status":"pending"} | ||
| ``` | ||
|
|
||
| Or query results via API: | ||
|
|
||
| ```bash | ||
| curl http://localhost:8080/api/v1/queries/my-query/results | ||
| ``` | ||
|
|
||
| ### 4. Check binlog status in MySQL | ||
|
|
||
| ```sql | ||
| SHOW MASTER STATUS; | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| | Error | Cause | Solution | | ||
| |-------|-------|----------| | ||
| | `binlog_format must be ROW` | Binlog format not set to ROW | Set `binlog_format = ROW` in `my.cnf` and restart | | ||
|
Comment on lines
+206
to
+208
|
||
| | `Access denied; you need the REPLICATION SLAVE privilege` | Missing permission | `GRANT REPLICATION SLAVE ON *.* TO 'drasi_user'@'%';` | | ||
| | `Can't connect to MySQL server` | Wrong host/port | Verify `host` and `port` values, check firewall | | ||
| | No changes captured | Table not in `tables` list | Add the table to the `tables` configuration | | ||
| | Unstable element IDs | No primary key | Add primary key or configure `tableKeys` | | ||
| | `binlog_row_image must be FULL` | Row image not complete | Set `binlog_row_image = FULL` in `my.cnf` and restart | | ||
|
|
||
| ### MySQL Permissions Checklist | ||
|
|
||
| Your database user needs: | ||
|
|
||
| ```sql | ||
| -- Required permissions | ||
| GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'drasi_user'@'%'; | ||
| GRANT SELECT ON mydb.* TO 'drasi_user'@'%'; | ||
| FLUSH PRIVILEGES; | ||
| ``` | ||
|
|
||
| ## Known limitations | ||
|
|
||
| - Packets larger than 16 MB are not supported. | ||
| - Schema changes (migrations, column additions/removals) during streaming may require a source restart. | ||
|
|
||
| ## Documentation resources | ||
|
|
||
| <div class="card-grid card-grid--2"> | ||
| <a href="https://github.com/drasi-project/drasi-core/blob/main/components/sources/mysql/README.md" target="_blank" rel="noopener"> | ||
| <div class="unified-card unified-card--tutorials"> | ||
| <div class="unified-card-icon"><i class="fab fa-github"></i></div> | ||
| <div class="unified-card-content"> | ||
| <h3 class="unified-card-title">MySQL Source README</h3> | ||
| <p class="unified-card-summary">Implementation notes, prerequisites, and behavior details</p> | ||
| </div> | ||
| </div> | ||
| </a> | ||
| <a href="https://crates.io/crates/drasi-source-mysql" target="_blank" rel="noopener"> | ||
| <div class="unified-card unified-card--howto"> | ||
| <div class="unified-card-icon"><i class="fas fa-box"></i></div> | ||
| <div class="unified-card-content"> | ||
| <h3 class="unified-card-title">drasi-source-mysql on crates.io</h3> | ||
| <p class="unified-card-summary">Package info and release history</p> | ||
| </div> | ||
| </div> | ||
| </a> | ||
| </div> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.