Transfer schema tree in batches & add memory control for schema tree#16035
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #16035 +/- ##
============================================
- Coverage 38.96% 38.93% -0.03%
Complexity 198 198
============================================
Files 4874 4876 +2
Lines 317061 317569 +508
Branches 39958 40034 +76
============================================
+ Hits 123530 123638 +108
- Misses 193531 193931 +400 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| } | ||
| } else if (type == 1) { | ||
| resultSchemaTree.mergeSchemaTree(ClusterSchemaTree.deserialize(inputStream)); | ||
| } else if (type == 1 || type == 2) { |
There was a problem hiding this comment.
better using type 2 and 3 to keep forward compatibility
There was a problem hiding this comment.
Fixed, but it should be noted that the current version and the previous version may still be incompatible, because the datanode of current version may provide incomplete schema tree tsblock to a datanode of old version
| } else if (type == 1 || type == 2) { | ||
| if (deserializer.isFirstBatch()) { | ||
| long memCost = ReadWriteIOUtils.readLong(inputStream); | ||
| context.reserveMemoryForSchemaTree(memCost); |
There was a problem hiding this comment.
add if (context != null)
| // to indicate this binary data is database info | ||
| ReadWriteIOUtils.write((byte) 1, baos); |
There was a problem hiding this comment.
why first byte is 1 means that this binary data is database info
| public static class SchemaNodeBatchDeserializer { | ||
| private byte nodeType; | ||
| private int childNum; | ||
| private Deque<SchemaNode> stack = new ArrayDeque<>(); |
There was a problem hiding this comment.
| private Deque<SchemaNode> stack = new ArrayDeque<>(); | |
| private final Deque<SchemaNode> stack = new ArrayDeque<>(); |
| nodeType = 0; | ||
| childNum = 0; | ||
| stack.clear(); | ||
| child = null; | ||
| hasLogicalView = false; | ||
| hasNormalTimeSeries = false; | ||
| templateMap = new HashMap<>(); | ||
| isFirstBatch = true; |
There was a problem hiding this comment.
put these codes into a private method named reset
There was a problem hiding this comment.
add some comments about why you need to construct a new HashMap instead of calling clear method for it
| } | ||
|
|
||
| private static class SchemaNodePostOrderIterator implements Iterator<SchemaNode> { | ||
| private final Deque<Pair<SchemaNode, Iterator<SchemaNode>>> stack = new ArrayDeque<>(); |
There was a problem hiding this comment.
Maybe Stack<Pair<SchemaNode, Boolean>> is enough
|
2 participants
Description
Transfer schema tree in batches