gh-148241: Fix json serialization for str subclasses

[vstinner]

Fix json serialization: no longer call str(obj) on str subclasses.

Replace PyUnicodeWriter_WriteStr() with PyUnicodeWriter_WriteASCII() and private _PyUnicodeWriter_WriteStr().

• Issue: String enums in dicts/lists are json serialized differently in Python 3.15 #148241

[vstinner]

An alternative to using the private API (which is bad!) is to call PyUnicode_FromObject(). But PyUnicode_FromObject() has to copy the string, it's less efficient.

[vstinner]

See also the issue gh-148250 that I just created for this issue.

[sunmy2019]

using the private API

It would raise a deprecated warning.

_Py_DEPRECATED_EXTERNALLY(3.14) PyAPI_FUNC(int) _PyUnicodeWriter_WriteStr(
    _PyUnicodeWriter *writer,
    PyObject *str); 

[serhiy-storchaka]

Is it safe to cast PyUnicodeWriter* to _PyUnicodeWriter*?

[vstinner]

Yes, it's safe. That's how PyUnicodeWriter_WriteStr() is implemented: by casting the first argument to _PyUnicodeWriter* and calling _PyUnicodeWriter_WriteStr().

Do you prefer calling PyUnicode_FromObject() to avoid the private API?

It would raise a deprecated warning.

_Py_DEPRECATED_EXTERNALLY() only emits a deprecation warning if the Py_BUILD_CORE macro is not defined. This macro is defined in Modules/_json.c, so no warning is emitted.

[sunmy2019]

Is it safe to cast PyUnicodeWriter* to _PyUnicodeWriter*?

Yes. I checked, PyUnicodeWriter is an opaque type that is always cast to _PyUnicodeWriter before use.

[sunmy2019]

I am in favor of public API.

Is PyUnicodeWriter_WriteSubstring applicable for this?

PyUnicodeWriter_WriteSubstring the only PyUnicodeWriter public API that treats str subclasses as str.

[serhiy-storchaka]

It is fine. We already use private API in this file.

[vstinner]

cc @nineteendo @methane @serhiy-storchaka (who worked on PR gh-133239 which introduced the regession)

[vstinner]

"Tests / CIFuzz / python3-libraries (address)" failure was already fixed by google/oss-fuzz#15317 but the fix was not deployed yet.

[StanFromIreland]

Re-running, it should be OK now.

[serhiy-storchaka]

There is test_enum specially for such cases. I think it is better to add a test here.

[vstinner]

There is test_enum specially for such cases. I think it is better to add a test here.

Ok, I also added a test to test_json.test_enum. I would prefer to keep also other tests since they test different code paths.

[serhiy-storchaka]

LGTM. 👍

I am not sure we need so many tests, but this is up to you.

[serhiy-storchaka]

It is fine. We already use private API in this file.