Skip to content

MySQL JSON field type with string should decode in python3#273

Open
howmp wants to merge 2 commits into
julien-duponchelle:masterfrom
howmp:master
Open

MySQL JSON field type with string should decode in python3#273
howmp wants to merge 2 commits into
julien-duponchelle:masterfrom
howmp:master

Conversation

@howmp

@howmp howmp commented Jul 10, 2018

Copy link
Copy Markdown

Fix #272

Table:

CREATE TABLE test (id int, value json);

With data:

id value
1 {"string1":"string2"}

But event's values in python3 is :

{"id":1,"value":{b"string1":b"string2"}

And when use json.dumps will raise TypeError: keys must be a string

According to rfc7159 : https://tools.ietf.org/html/rfc7159#section-8.1

JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32. The default
encoding is UTF-8

So should decode as utf-8

@howmp

howmp commented Jul 10, 2018

Copy link
Copy Markdown
Author

Please restart build, seems network error.

https://travis-ci.org/noplay/python-mysql-replication/jobs/402200041

@baloo

@baloo baloo closed this Jul 11, 2018
@baloo baloo reopened this Jul 11, 2018
@baloo

baloo commented Jul 11, 2018

Copy link
Copy Markdown
Collaborator

I'm not the owner of the project so can't restart directly. have to open/close :/ sorry

@howmp

howmp commented Jul 12, 2018

Copy link
Copy Markdown
Author

Failed when install cryptography in python 3.3,But I didn't change CI

@baloo

baloo commented Sep 5, 2018

Copy link
Copy Markdown
Collaborator

We've been converting charset all over the place. I dislike that a lot (it's not our job as a lib). Forcing encoding to utf-8 would be a nightmare to maintain on the long run. I'm not sure we want that.

We should find a better way for user to provide a encoding (even if we default to utf-8). Consider a column with two json fields, one utf8, one utf16.

@howmp

howmp commented Sep 6, 2018

Copy link
Copy Markdown
Author

In the documentation: https://dev.mysql.com/doc/refman/5.7/en/json.html

MySQL handles strings used in JSON context using the utf8mb4 character set and utf8mb4_bin collation. Strings in other character sets are converted to utf8mb4 as necessary. (For strings in the ascii or utf8 character sets, no conversion is needed because ascii and utf8 are subsets of utf8mb4.)

Json field can't be utf16

@rudaporto

Copy link
Copy Markdown

It would be good to remove support for Python 3.3 since nobody should be using it anymore (I guess).

Makes sense?

@howmp

howmp commented Oct 11, 2018

Copy link
Copy Markdown
Author

It would be good to remove support for Python 3.3 since nobody should be using it anymore (I guess).

Makes sense?

Yeah. I agree

@telemmaite telemmaite left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not move forward with this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants