String enums in dicts/lists are json serialized differently in Python 3.15

[befeleme]

Bug report

Bug description:

Cross posting from cattrs: python-attrs/cattrs#744
There's a difference in how Python 3.14 and Pytthon 3.15 handle json serialization of string enum values stored in other types which is exposed by cattrs tests.
I have failed to find any track record for this behavior in the available docs.

See:

Python 3.15.0a7

>>> import json
>>> from enum import Enum
>>> class AStringEnum(str, Enum):
...     A = "a"
...     
>>> val = AStringEnum.A
>>> json.dumps(val)
'"a"'
>>> in_dict = json.dumps({"key": val})
>>> in_dict
'{"key": "AStringEnum.A"}'
>>> parsed = json.loads(in_dict)
>>> type(parsed['key'])
<class 'str'>
>>> repr(parsed['key'])
"'AStringEnum.A'"
>>> AStringEnum(parsed['key'])
Traceback (most recent call last):
  File "<python-input-15>", line 1, in <module>
    AStringEnum(parsed['key'])
    ~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.15/enum.py", line 732, in __call__
    return cls.__new__(cls, value)
           ~~~~~~~~~~~^^^^^^^^^^^^
  File "/usr/lib64/python3.15/enum.py", line 1221, in __new__
    raise ve_exc
ValueError: 'AStringEnum.A' is not a valid AStringEnum

Python 3.14.3

>>> import json
>>> from enum import Enum
>>> class AStringEnum(str, Enum):
...     A = "a"
...  
>>> val = AStringEnum.A
>>> json.dumps(val)
'"a"'
>>> json.dumps({"key": val})
'{"key": "a"}'
>>> in_dict = json.dumps({"key": val})
>>> parsed = json.loads(in_dict)
>>> type(parsed['key'])
<class 'str'>
>>> repr(parsed['key'])
"'a'"
>>> AStringEnum(parsed['key'])
<AStringEnum.A: 'a'>

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

• gh-148241: Fix json serialization for str subclasses #148249

[hugovk]

Reproducible with 3.15.0a1, and bisects to dec624e (PR #133239, issue #135336): "Add fast path to json string encoding".

cc @nineteendo @vstinner

[StanFromIreland]

This is because the __str__ changed, on main:

>>> str(parsed['key'])
'AStringEnum.A'

whereas previously:

>>> str(parsed['key'])
'a'

[vstinner]

Code to write the dictionary key:

• Python 3.14: encoder_listencode_obj() => encoder_encode_string() => py_encode_basestring_ascii() => ascii_escape_unicode(). This function copies the string character by character using PyUnicode_READ().
• Python 3.15: encoder_listencode_obj() => encoder_writer_string() => write_escaped_ascii(). This function calls PyUnicodeWriter_WriteStr() which calls PyObject_Str() since the dictionary key is a str subclass and not exactly the str type. PyObject_Str() returns AStringEnum.A string.

So the difference is in write_escaped_ascii() which now calls PyObject_Str() for str subclasses.

[vstinner]

Example:

import enum
import _testlimitedcapi

unicode_fromobject = _testlimitedcapi.unicode_fromobject

class AStringEnum(str, enum.Enum):
    A = "a"

obj = AStringEnum.A
print(f"str: {str(obj)!r}")
print(f"PyUnicode_FromObject: {unicode_fromobject(obj)!r}")

Output on Python 3.15:

str: 'AStringEnum.A'
PyUnicode_FromObject: 'a'

[vstinner]

I created PR gh-148249 to fix the json regression.

I also created issue gh-148250 to discuss PyUnicodeWriter_WriteStr() usage with str subclasses.