Skip to content

Commit d7ddd90

Browse files
orisgarnochaokunyang
orisgarno
and
chaokunyang
authored
fix(java): Compatible mode on de/serialize api failed to deserialize (#1996)
## What does this PR do? Read and write class data on COMPATIBLE mode for de/serializeJavaObject api. When COMPATIBLE mode is on and need to serialize and deserialize different POJO, users are required to register classes those are going to be serialized or deserialized. ## Related issues <!-- Is there any related issue? Please attach here. - #xxxx0 - #xxxx1 - #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. --> --------- Co-authored-by: Shawn Yang <chaokunyang@apache.org>
1 parent 1c250e1 commit d7ddd90

File tree

8 files changed

+331
-2
lines changed

8 files changed

+331
-2
lines changed

‎docs/guide/java_serialization_guide.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ public class Example {
102102
| `compressLong` | Enables or disables long compression for smaller size. | `true` |
103103
| `compressString` | Enables or disables string compression for smaller size. | `false` |
104104
| `classLoader` | The classloader should not be updated; Fury caches class metadata. Use `LoaderBinding` or `ThreadSafeFury` for classloader updates. | `Thread.currentThread().getContextClassLoader()` |
105-
| `compatibleMode` | Type forward/backward compatibility config. Also Related to `checkClassVersion` config. `SCHEMA_CONSISTENT`: Class schema must be consistent between serialization peer and deserialization peer. `COMPATIBLE`: Class schema can be different between serialization peer and deserialization peer. They can add/delete fields independently. | `CompatibleMode.SCHEMA_CONSISTENT` |
105+
| `compatibleMode` | Type forward/backward compatibility config. Also Related to `checkClassVersion` config. `SCHEMA_CONSISTENT`: Class schema must be consistent between serialization peer and deserialization peer. `COMPATIBLE`: Class schema can be different between serialization peer and deserialization peer. They can add/delete fields independently. [See more](#class-inconsistency-and-class-version-check). | `CompatibleMode.SCHEMA_CONSISTENT` |
106106
| `checkClassVersion` | Determines whether to check the consistency of the class schema. If enabled, Fury checks, writes, and checks consistency using the `classVersionHash`. It will be automatically disabled when `CompatibleMode#COMPATIBLE` is enabled. Disabling is not recommended unless you can ensure the class won't evolve. | `false` |
107107
| `checkJdkClassSerializable` | Enables or disables checking of `Serializable` interface for classes under `java.*`. If a class under `java.*` is not `Serializable`, Fury will throw an `UnsupportedOperationException`. | `true` |
108108
| `registerGuavaTypes` | Whether to pre-register Guava types such as `RegularImmutableMap`/`RegularImmutableList`. These types are not public API, but seem pretty stable. | `true` |
@@ -518,6 +518,13 @@ fury with
518518
`CompatibleMode.COMPATIBLE` has more performance and space cost, do not set it by default if your classes are always
519519
consistent between serialization and deserialization.
520520

521+
### Deserialize POJO into another type
522+
523+
Fury allows you to serialize one POJO and deserialize it into a different POJO. To achieve this, configure Fury with
524+
`CompatibleMode` set to `org.apache.fury.config.CompatibleMode.COMPATIBLE`. Additionally, you only need to register the
525+
specific classes you want to serialize or deserialize to setup type mapping relationship; there's no need to register any nested classes within them.
526+
[See example here](/java/fury-core/src/test/java/org/apache/fury/serializer/compatible/DifferentPOJOCompatibleSerializerTest.java)
527+
521528
### Use wrong API for deserialization
522529

523530
If you serialize an object by invoking `Fury#serialize`, you should invoke `Fury#deserialize` for deserialization

‎java/fury-core/src/main/java/org/apache/fury/Fury.java‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1070,6 +1070,7 @@ public void serializeJavaObject(MemoryBuffer buffer, Object obj) {
10701070
buffer.writeInt32(-1); // preserve 4-byte for meta start offsets.
10711071
if (!refResolver.writeRefOrNull(buffer, obj)) {
10721072
ClassInfo classInfo = classResolver.getOrUpdateClassInfo(obj.getClass());
1073+
classResolver.writeClass(buffer, classInfo);
10731074
writeData(buffer, classInfo, obj);
10741075
MetaContext metaContext = serializationContext.getMetaContext();
10751076
if (metaContext != null && !metaContext.writingClassDefs.isEmpty()) {
@@ -1119,7 +1120,13 @@ public <T> T deserializeJavaObject(MemoryBuffer buffer, Class<T> cls) {
11191120
T obj;
11201121
int nextReadRefId = refResolver.tryPreserveRefId(buffer);
11211122
if (nextReadRefId >= NOT_NULL_VALUE_FLAG) {
1122-
obj = (T) readDataInternal(buffer, classResolver.getClassInfo(cls));
1123+
ClassInfo classInfo;
1124+
if (shareMeta) {
1125+
classInfo = classResolver.readClassInfo(buffer);
1126+
} else {
1127+
classInfo = classResolver.getClassInfo(cls);
1128+
}
1129+
obj = (T) readDataInternal(buffer, classInfo);
11231130
return obj;
11241131
} else {
11251132
return null;

‎java/fury-core/src/test/java/org/apache/fury/serializer/EnumSerializerTest.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,10 @@ public void testEnumSerializationAsString() {
140140
.build();
141141

142142
// serialize enum "B"
143+
furySerialization.register(cls1);
143144
byte[] bytes = furySerialization.serializeJavaObject(cls1.getEnumConstants()[1]);
144145

146+
furyDeserialize.register(cls2);
145147
Object data = furyDeserialize.deserializeJavaObject(bytes, cls2);
146148
assertEquals(cls2.getEnumConstants()[0], data);
147149
}
@@ -175,8 +177,10 @@ public void testEnumSerializationAsString_differentClass() {
175177
.build();
176178

177179
// serialize enum "B"
180+
furySerialization.register(cls1);
178181
byte[] bytes = furySerialization.serializeJavaObject(cls1.getEnumConstants()[1]);
179182

183+
furyDeserialize.register(cls2);
180184
Object data = furyDeserialize.deserializeJavaObject(bytes, cls2);
181185
assertEquals(cls2.getEnumConstants()[0], data);
182186
}
@@ -209,9 +213,11 @@ public void testEnumSerializationAsString_invalidEnum() {
209213
.withAsyncCompilation(false)
210214
.build();
211215

216+
furySerialization.register(cls1);
212217
byte[] bytes = furySerialization.serializeJavaObject(cls1.getEnumConstants()[0]);
213218

214219
try {
220+
furyDeserialize.register(cls2);
215221
furyDeserialize.deserializeJavaObject(bytes, cls2);
216222
fail("expected to throw exception");
217223
} catch (Exception e) {

‎java/fury-core/src/test/java/org/apache/fury/serializer/compatible/DifferentPOJOCompatibleSerializerTest.java‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.fury.serializer.compatible;
21+
22+
import org.apache.fury.Fury;
23+
import org.apache.fury.config.CompatibleMode;
24+
import org.apache.fury.config.Language;
25+
import org.apache.fury.serializer.compatible.classes.ClassCompleteField;
26+
import org.apache.fury.serializer.compatible.classes.ClassMissingField;
27+
import org.testng.Assert;
28+
import org.testng.annotations.Test;
29+
30+
/**
31+
* Test COMPATIBILITY mode that supports - same field type and name can be deserialized to other
32+
* class with different name - scrambled field order to make sure it could handle different field
33+
* order - missing or extra field from source class to target class - generic class
34+
*/
35+
public class DifferentPOJOCompatibleSerializerTest extends Assert {
36+
37+
Fury getFury(Class<?>... classes) {
38+
Fury instance =
39+
Fury.builder()
40+
.withLanguage(Language.JAVA)
41+
.withRefTracking(true)
42+
.withCompatibleMode(CompatibleMode.COMPATIBLE)
43+
.withMetaShare(true)
44+
.withScopedMetaShare(true)
45+
.requireClassRegistration(false)
46+
.withAsyncCompilation(true)
47+
.serializeEnumByName(true)
48+
.build();
49+
if (classes != null) {
50+
for (Class<?> clazz : classes) {
51+
instance.register(clazz);
52+
}
53+
}
54+
;
55+
return instance;
56+
}
57+
58+
@Test
59+
void testTargetHasLessFieldComparedToSourceClass() throws InterruptedException {
60+
61+
ClassCompleteField<String> subclass = new ClassCompleteField<>("subclass", "subclass2");
62+
ClassCompleteField<ClassCompleteField<String>> classCompleteField =
63+
new ClassCompleteField<>(subclass, subclass);
64+
byte[] serialized = getFury(ClassCompleteField.class).serializeJavaObject(classCompleteField);
65+
ClassMissingField<ClassMissingField<String>> classMissingField =
66+
getFury(ClassMissingField.class).deserializeJavaObject(serialized, ClassMissingField.class);
67+
68+
assertEq(classCompleteField, classMissingField);
69+
}
70+
71+
@Test
72+
void testTargetHasMoreFieldComparedToSourceClass() throws InterruptedException {
73+
74+
ClassMissingField<String> subclass = new ClassMissingField<>("subclass");
75+
ClassMissingField classMissingField = new ClassMissingField(subclass);
76+
byte[] serialized = getFury(ClassMissingField.class).serializeJavaObject(classMissingField);
77+
78+
ClassCompleteField classCompleteField =
79+
getFury(ClassCompleteField.class)
80+
.deserializeJavaObject(serialized, ClassCompleteField.class);
81+
82+
assertEq(classCompleteField, classMissingField);
83+
}
84+
85+
void assertEq(ClassCompleteField classCompleteField, ClassMissingField classMissingField) {
86+
assertEqSubClass(
87+
(ClassCompleteField) classCompleteField.getPrivateFieldSubClass(),
88+
(ClassMissingField) classMissingField.getPrivateFieldSubClass());
89+
assertEquals(classCompleteField.getPrivateMap(), classMissingField.getPrivateMap());
90+
assertEquals(classCompleteField.getPrivateList(), classMissingField.getPrivateList());
91+
assertEquals(classCompleteField.getPrivateString(), classMissingField.getPrivateString());
92+
assertEquals(classCompleteField.getPrivateInt(), classMissingField.getPrivateInt());
93+
}
94+
95+
void assertEqSubClass(
96+
ClassCompleteField classCompleteField, ClassMissingField classMissingField) {
97+
assertEquals(
98+
classCompleteField.getPrivateFieldSubClass(), classMissingField.getPrivateFieldSubClass());
99+
assertEquals(classCompleteField.getPrivateMap(), classMissingField.getPrivateMap());
100+
assertEquals(classCompleteField.getPrivateList(), classMissingField.getPrivateList());
101+
assertEquals(classCompleteField.getPrivateString(), classMissingField.getPrivateString());
102+
assertEquals(classCompleteField.getPrivateInt(), classMissingField.getPrivateInt());
103+
}
104+
}

‎java/fury-core/src/test/java/org/apache/fury/serializer/compatible/classes/ClassCompleteField.java‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.fury.serializer.compatible.classes;
21+
22+
import java.util.ArrayList;
23+
import java.util.HashMap;
24+
import java.util.List;
25+
import java.util.Map;
26+
import lombok.EqualsAndHashCode;
27+
import lombok.Getter;
28+
29+
@EqualsAndHashCode
30+
@Getter
31+
public class ClassCompleteField<T> {
32+
private boolean privateBoolean = true;
33+
private int privateInt = 10;
34+
private String privateString = "notNull";
35+
private Map<String, String> privateMap;
36+
private List<String> privateList;
37+
private T privateFieldSubClass;
38+
39+
private boolean privateBoolean2 = true;
40+
private int privateInt2 = 10;
41+
private String privateString2 = "notNull";
42+
private Map<String, String> privateMap2;
43+
private List<String> privateList2;
44+
private T privateFieldSubClass2;
45+
46+
public ClassCompleteField(T privateFieldSubClass, T privateFieldSubClass2) {
47+
privateMap = new HashMap<>();
48+
privateMap.put("a", "b");
49+
privateList = new ArrayList<>();
50+
privateList.add("a");
51+
52+
privateMap2 = new HashMap<>();
53+
privateMap2.put("a", "b");
54+
privateList2 = new ArrayList<>();
55+
privateList2.add("a");
56+
57+
this.privateFieldSubClass = privateFieldSubClass;
58+
this.privateFieldSubClass2 = privateFieldSubClass2;
59+
}
60+
}

‎java/fury-core/src/test/java/org/apache/fury/serializer/compatible/classes/ClassMissingField.java‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.fury.serializer.compatible.classes;
21+
22+
import java.util.ArrayList;
23+
import java.util.HashMap;
24+
import java.util.List;
25+
import java.util.Map;
26+
import lombok.EqualsAndHashCode;
27+
import lombok.Getter;
28+
29+
@EqualsAndHashCode
30+
@Getter
31+
public class ClassMissingField<T> {
32+
private T privateFieldSubClass;
33+
private List<String> privateList;
34+
private Map<String, String> privateMap;
35+
private String privateString = "missing";
36+
private int privateInt = 999;
37+
private boolean privateBoolean = false;
38+
39+
public ClassMissingField(T privateFieldSubClass) {
40+
privateMap = new HashMap<>();
41+
privateMap.put("z", "x");
42+
privateList = new ArrayList<>();
43+
privateList.add("c");
44+
this.privateFieldSubClass = privateFieldSubClass;
45+
}
46+
}

‎java/fury-core/src/test/java/org/apache/fury/serializer/compatible/classes/SubclassCompleteField.java‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.fury.serializer.compatible.classes;
21+
22+
import java.util.ArrayList;
23+
import java.util.HashMap;
24+
import java.util.List;
25+
import java.util.Map;
26+
import lombok.EqualsAndHashCode;
27+
import lombok.Getter;
28+
29+
@Getter
30+
@EqualsAndHashCode
31+
public class SubclassCompleteField {
32+
private boolean privateBoolean = true;
33+
private int privateInt = 10;
34+
private String privateString = "notNull";
35+
private Map<String, String> privateMap;
36+
private List<String> privateList;
37+
38+
private boolean privateBoolean2 = true;
39+
private int privateInt2 = 10;
40+
private String privateString2 = "notNull";
41+
private Map<String, String> privateMap2;
42+
private List<String> privateList2;
43+
44+
public SubclassCompleteField() {
45+
privateMap = new HashMap<>();
46+
privateMap.put("a", "b");
47+
privateList = new ArrayList<>();
48+
privateList.add("a");
49+
50+
privateMap2 = new HashMap<>();
51+
privateMap2.put("a", "b");
52+
privateList2 = new ArrayList<>();
53+
privateList2.add("a");
54+
}
55+
}

‎java/fury-core/src/test/java/org/apache/fury/serializer/compatible/classes/SubclassMissingField.java‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.fury.serializer.compatible.classes;
21+
22+
import java.util.ArrayList;
23+
import java.util.HashMap;
24+
import java.util.List;
25+
import java.util.Map;
26+
import lombok.EqualsAndHashCode;
27+
import lombok.Getter;
28+
29+
@Getter
30+
@EqualsAndHashCode
31+
public class SubclassMissingField {
32+
private boolean privateBoolean = true;
33+
private int privateInt = 10;
34+
private String privateString = "notNull";
35+
private Map<String, String> privateMap;
36+
private List<String> privateList;
37+
38+
public SubclassMissingField() {
39+
privateMap = new HashMap<>();
40+
privateMap.put("a", "b");
41+
privateList = new ArrayList<>();
42+
privateList.add("a");
43+
}
44+
}

0 commit comments

Comments
 (0)