Deserialization of generic class with nullable values failes

[Janik7777]

Search before asking

• I searched in the issues and found nothing similar.
• I have confirmed that the same problem is not reproduced if I exclude the KotlinModule.
• I searched in the issues of databind and other modules used and found nothing similar.
• I have confirmed that the problem does not reproduce in Java and only occurs when using Kotlin and KotlinModule.

Describe the bug

Error occurs during deserialization of my generic class:

com.fasterxml.jackson.module.kotlin.MissingKotlinParameterException: Instantiation of [simple type, class SerializationTest$test_deserialization$Example<java.lang.String>] value failed for JSON property data due to missing (therefore NULL) value for creator parameter data which is a non-nullable type
 at [Source: (StringReader); line: 3, column: 1] (through reference chain: SerializationTest$test_deserialization$Example["data"])
	at com.fasterxml.jackson.module.kotlin.KotlinValueInstantiator.createFromObjectWith(KotlinValueInstantiator.kt:97)
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:214)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:541)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1497)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:348)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4917)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3860)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3843)
	at SerializationTest.test_deserialization(SerializationTest.kt:36)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

This is my data class

@JsonIgnoreProperties(ignoreUnknown = true)
data class Example<T> @JsonCreator constructor(
    @JsonProperty("data") var data: T,
    @JsonProperty("number") var number : Int,
)

I want use it for non null and nullable data types for T. One example usage would be:

val value = Example<String?>(null, 10)

We want our YAML file only to include non-null fields, so we use the option setSerializationInclusion(JsonInclude.Include.NON_NULL) for the mapper.

But now the deserialization fails.

To Reproduce

import com.fasterxml.jackson.annotation.JsonCreator
import com.fasterxml.jackson.annotation.JsonIgnoreProperties
import com.fasterxml.jackson.annotation.JsonInclude
import com.fasterxml.jackson.annotation.JsonProperty
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory
import com.fasterxml.jackson.module.kotlin.readValue
import com.fasterxml.jackson.module.kotlin.registerKotlinModule
import org.junit.jupiter.api.Test

class SerializationTest {
    @Test
    fun test_deserialization () {
        @JsonIgnoreProperties(ignoreUnknown = true)
        data class Example<T> @JsonCreator constructor(
            @JsonProperty("data") var data: T,
            @JsonProperty("number") var number : Int,
        )
        val yamlMapper = ObjectMapper(
            YAMLFactory()
        )
            .registerKotlinModule()
            .setSerializationInclusion(JsonInclude.Include.NON_NULL)

        val value = Example<String?>(null, 10)
        val yamlString = yamlMapper.writeValueAsString(value)
        val deserializedValue = yamlMapper.readValue<Example<String?>>(yamlString)
    }
}

Expected behavior

deserializedValue == Example<String?>(null, 10)

Versions

Kotlin:
Jackson-module-kotlin: 2.18.2
Jackson-databind: 2.18.2

Additional context

No response

[wakingrufus]

I think there is a flaw in the extension methods in regards to nullable types.
for example, with the current implementation:

inline fun <reified T> ObjectMapper.readValue(content: String): T = readValue(content, jacksonTypeRef<T>())

it is possible to write code like:

val person : Person = mapper.readValue("null")

and the value of person will be null. I think a runtime exception can be thrown for this case via the NewStrictNullChecks feature, but I think that is only a bandaid and still exposes issues such as the OP of this issue.

I think a better implementation might be:

inline fun <reified T> ObjectMapper.readValue(content: String): T? = readValue(content, jacksonTypeRef<T>())

which would make the above usage fail to compile and force the usage to look like thiss:

val person : Person? = mapper.readValue("null")
// which is short for:
val person : Person? = mapper.readValue<Person>("null")

This is a more accurate representation of the fact that the readValue can return nulls.

I would be happy to open a PR for this, if @cowtowncoder and @k163377 agree that this is better.

[wakingrufus]

I have done some more investigating, and found that while the above change would work for the nullability of the outer types, it would not work for generic type parameters. for example, it would still allow for something like this to be written:

val actual: List<Person>? = objectMapper.readValue<List<Person>>("""[{"id":1,"name":"Bob Dobalina"}, null]""")

which would still rely on runtime null enforcing

[k163377]

@Janik7777
Sorry for the late reply.
I checked and it was very difficult to fix this issue.

The direct cause of this problem is that the required flag set by kotlin-module is unexpectedly true (the same as specifying JsonProperty(required = true)).

First, it seems that kotlin-module cannot read what type is specified for the type argument T during deserialization.
This makes it impossible to achieve processing without edge cases.

Next, I thought about setting the required flag heuristically from the upper bound of T, but it seems that it is not possible to read the upper bound from KParameter and it is difficult to get it from other definitions.
The only thing that is possible is to read the nullability definition of the parameter itself.

Finally, I thought about forcing an override of the required flag, like @JsonProperty(required = false), but this was not possible due to conflicts with the basic behavior of the kotlin-module.

I believe the most practical thing to do would be to modify Jackson 3.0 to make the last idea feasible.
I have proposed this to the team.
FasterXML/jackson-future-ideas#81 (comment)

[k163377]

@wakingrufus
Sorry for the late reply.

it is possible to write code like:

That is a different issue, duplicate of #399.

I think a better implementation might be:

Such interface changes are unacceptable because they are destructive changes that cause compile errors.
The Kotlin problem also makes it difficult to define a distinction in the nullability of type argument, so a new function would need to be defined.
https://youtrack.jetbrains.com/issue/KT-56930

[k163377]

@Janik7777

I believe the most practical thing to do would be to modify Jackson 3.0 to make the last idea feasible.
I have proposed this to the team.
FasterXML/jackson-future-ideas#81 (comment)

Regarding the above comment, functionality has been added to 2.19 so that workarounds for manual specification will now work.
A sample is shown below.
https://github.com/FasterXML/jackson-module-kotlin/pull/930/files#diff-cc3a9b23c55bd9e5a9ce86af173f50717713da8058a9b89699ad632f3be2ff5eR29-R38

[DavidRigglemanININ]

I just ran into a similar problem which I'm guessing has the same root issue as this bug. Here's my code snippet that failed

data class ListWrapper<T>(val values: List<T>)
data class TestObject(val obj: ListWrapper<Int?>)

@Test
fun testNullableLists() {
    objectMapper.read(TestObject::class.java, """{ "obj": { "values": [null, 1] } }""".toByteArray())
}

I've worked around it for now by creating two generic variants (one with List and another with List<T?>) and that does solve the problem for now. But this caught us a bit off guard because it wasn't a problem until we turned on the newStrictNullChecks option.

[k163377]

memo:
If KParameter.type.classifier is a KTypeParameter, upperBounds contains nullability information.
Accessing this information would allow for more accurate nullability checks and potentially resolve this issue.

However, this would involve a change in behavior, so implementation in 3.3.x and later versions should be considered.