You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Map MySQL AUTO_INCREMENT columns to Spanner IDENTITY columns (#1227)
* Map auto increment source columns to Spanner IDENTITY columns
Note, I'm not using the AUTO_INCREMENT keyword or SERIAL type here because those don't allow setting of the skip range
or start counter with values. Additionally, they both alias to the IDENTITY clause I'm using here anyways, so it's
effectively the same, but this is more flexible.
* Remove sequence creation for AUTO_INCREMENT columns
The use of a sequence will no longer be needed. The next commit will update the AST printing to use the new
AUTO_INCREMENT keyword (Spanner SQL) and SERIAL type (PGSQL).
* Add support for migrating AUTO_INCREMENT columns from mysql dump files
The AST mapping will be added in a subsequent commit.
* Minor cleanup to align cassandra's GetColumnAutoGen impl with other DBs that don't support migration of auto incrementing columns
* Add note to set skip range or start counter with to avoid collisions for identity columns
* Add Identity as an option for Auto-Gen columns in the web UI
* Allow for setting of the skip range and start counter with values via the web UI
* Update docs
* Fix notation for identity column type (needed to use underscores instead of spaces...)
* Fix lint and integration test workflows
* Allow Go version 1.24 for lint tool (latest staticcheck requires at least 1.24)
---------
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
Copy file name to clipboardExpand all lines: docs/data-types/mysql.md
+51-3Lines changed: 51 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -170,9 +170,57 @@ maps `UNIQUE` constraint into `UNIQUE` secondary index. Note that due to limitat
170
170
mysqldump parser, we are not able to handle key column ordering (i.e. ASC/DESC) in
171
171
mysqldump files. All key columns in mysqldump files will be treated as ASC.
172
172
173
-
## Auto-Increment and Sequences
174
-
175
-
The tool creates a new sequence for auto-increment columns and maps the auto-generation of these columns to this sequence. The sequence type is of *bit reversed positive*. Users need to set skip range and/or start with counter to avoid duplicate key errors.
173
+
## Auto-Increment Columns
174
+
175
+
The tool maps auto-increment columns to [Spanner IDENTITY
Users need to set the SKIP RANGE and/or START COUNTER WITH values to avoid duplicate key errors.
178
+
179
+
The SKIP RANGE and START COUNTER WITH values can be set most via both the web UI (recommended) and the CLI.
180
+
181
+
The Column tab of the web UI exposes fields to set the SKIP RANGE and START COUNTER WITH values. For more details, see [here](../ui/schema-conv/spanner-draft.md).
182
+
183
+
To set the SKIP RANGE and/or START COUNTER WITH values via the CLI, the recommended steps are as follows:
184
+
- Do a dry-run schema-only migration to generate a session JSON file:
185
+
```sh
186
+
spanner-migration-tool schema -dry-run ...
187
+
```
188
+
- Open the resulting session file and find the relevant column definition(s) in the `ColDefs` collection of the table
189
+
it belongs to
190
+
- Set the appropriate fields in that column's `AutoGen.AutoIncrementOptions` node. All three values are expected to be
191
+
strings containing a numeric value. For example:
192
+
```json
193
+
{
194
+
"SpSchema": {
195
+
"table1": {
196
+
"Name": "SomeTable",
197
+
"ColDefs": {
198
+
"column1": {
199
+
"Name": "some_column",
200
+
"AutoGen": {
201
+
"Name": "Auto Increment",
202
+
"GenerationType": "Auto Increment",
203
+
"AutoIncrementOptions": {
204
+
"SkipRangeMin": "1000",
205
+
"SkipRangeMax": "10000",
206
+
"StartCounterWith": "500"
207
+
}
208
+
},
209
+
...
210
+
},
211
+
...
212
+
},
213
+
...
214
+
},
215
+
...
216
+
},
217
+
...
218
+
}
219
+
```
220
+
- Save the session file and run your desired migration using the updated session file:
221
+
```sh
222
+
spanner-migration-tool schema -session=<path to session file> ...
Copy file name to clipboardExpand all lines: docs/ui/schema-conv/spanner-draft.md
+12-3Lines changed: 12 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ Users can view detailed information for a table by selecting it from the **Spann
26
26
27
27
### Column
28
28
29
-
Column tab provides information on the columns that are a part of the selected table. It also provides the option to edit the column wherein a user can modify a column name, delete a column, change the data type of the column, add auto-generation to the column, modify the default value or modify the null property of the column. Once the user is done with required modifications, they can click on **SAVE & CONVERT **and the update would reflect in the session file and across all the components in the database.
29
+
Column tab provides information on the columns that are a part of the selected table. It also provides the option to edit the column wherein a user can modify a column name, delete a column, change the data type of the column, add and configure auto-generation for the column, modify the default value or modify the null property of the column. Once the user is done with required modifications, they can click on **SAVE & CONVERT **and the update would reflect in the session file and across all the components in the database.
@@ -120,7 +120,8 @@ Apart from the existing indexes for the source database, users can also add seco
120
120
121
121
Auto-Generated Columns populate Spanner columns automatically if no value is provided. Currently Spanner Migration Tool support the following techniques for auto-generation:
122
122
1. UUID function: Generate a UUID (v4) as part of a table’s primary key DEFAULT expression.
123
-
2. Bit reverse function: Map existing integer keys using the same logic as a bit-reversed sequence to avoid hotspotting.
123
+
2. Identity columns: automatically generate integer values without having to manually maintain an underlying sequence. Existing auto-incrementing columns are mapped to Identity columns.
124
+
3. Bit reverse function: Map existing integer keys using the same logic as a bit-reversed sequence to avoid hotspotting.
124
125
125
126
Refer to [documentation](https://cloud.google.com/spanner/docs/primary-key-default-value).
126
127
@@ -129,9 +130,17 @@ Refer to [documentation](https://cloud.google.com/spanner/docs/primary-key-defau
129
130
The default recommendation for primary keys in Spanner is to use a Universally Unique Identifier, or UUID. Users can convert existing columns to be filled by UUID by choosing the **Edit** option in a table and under the **Auto-Generated** column choosing UUID.
Identity columns automatically generate integer values without requiring manual maintenance or management of an underlying sequence by users.
136
+
137
+
By default, the Spanner Migration Tool maps auto-incrementing columns from the source database to Identity columns in Spanner, without setting any SKIP RANGE or START COUNTER WITH values. In order to avoid duplicates, it is recommended to set either a SKIP RANGE or a START COUNTER WITH value for all auto-incrementing columns that will be migrated.
138
+
139
+
See [documentation](https://cloud.google.com/spanner/docs/primary-key-default-value#identity-columns) for more details on Identity columns.
140
+
132
141
### Sequences
133
142
134
-
Spanner offers a SEQUENCE object that generates unique integers as part of a primary key DEFAULT expression. However, unlike a monotonic sequence, the values generated by a Spanner sequence are distributed uniformly and thus won’t hotspot at scale. Existing **Auto-Increment** columns will be mapped to a new Sequence.
143
+
Spanner offers a SEQUENCE object that generates unique integers as part of a primary key DEFAULT expression. However, unlike a monotonic sequence, the values generated by a Spanner sequence are distributed uniformly and thus won’t hotspot at scale.
135
144
136
145
#### Create a new Sequence
137
146
In order to add a sequence, the user needs to select the **Add Sequence** option and provide some details mandatory to create a sequence like **sequence name** and **sequence type**. An existing sequence can also be modified using the **Edit** button.
Description: fmt.Sprintf("Column '%s' is an autoincrement column in table '%s'. %s", spColName, conv.SpSchema[tableId].Name, IssueDB[i].Brief),
325
325
}
326
326
l=append(l, toAppend)
327
+
caseinternal.IdentitySkipRange:
328
+
toAppend:=Issue{
329
+
Category: IssueDB[i].Category,
330
+
Description: fmt.Sprintf("Column '%s' has been converted to an identity column in table '%s'. %s", spColName, conv.SpSchema[tableId].Name, IssueDB[i].Brief),
331
+
}
332
+
l=append(l, toAppend)
327
333
caseinternal.SequenceCreated:
328
334
toAppend:=Issue{
329
335
Category: IssueDB[i].Category,
@@ -635,6 +641,7 @@ var IssueDB = map[internal.SchemaIssue]struct {
635
641
internal.DecimalThatFits: {Brief: "Spanner does not support decimal, but this type mapping preserves the decimal's specified precision", Severity: suggestion, Category: "DECIMAL_THAT_FITS"},
636
642
internal.Serial: {Brief: "Spanner does not support autoincrementing types", Severity: warning, Category: "AUTOINCREMENTING_TYPE_USES"},
637
643
internal.AutoIncrement: {Brief: "Spanner does not support auto_increment attribute", Severity: warning, Category: "AUTO_INCREMENT_ATTRIBUTE_USES"},
644
+
internal.IdentitySkipRange: {Brief: "Set Skip Range or Start Counter With values to avoid duplicate value errors.", Severity: note, Category: "IDENTITY_SKIP_RANGE_SUGGESTION"},
638
645
internal.Timestamp: {Brief: "Spanner timestamp is closer to PostgreSQL timestamptz", Severity: suggestion, batch: true, Category: "TIMESTAMP_SUGGESTION"},
639
646
internal.Datetime: {Brief: "Spanner timestamp is closer to MySQL timestamp", Severity: warning, batch: true, Category: "TIMESTAMP_WARNING"},
640
647
internal.Time: {Brief: "Spanner does not support time/year types", Severity: warning, batch: true, Category: "TIME_YEAR_TYPE_USES"},
0 commit comments