Skip to content

Commit aea1f6d

Browse files
zhangyue1818my-ship-it
zhangyue1818
authored and
my-ship-it
committed
Add a README.md file to document the architecture and incorporate feedback from the review.
1 parent 0508a3e commit aea1f6d

6 files changed

Lines changed: 267 additions & 37 deletions

File tree

‎contrib/udp2/README.md‎

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# UDP2 Interconnect Protocol Implementation
21+
22+
## Project Background
23+
24+
UDP2 is a next-generation interconnect protocol implementation based on the original UDP protocol, located in the `contrib/udp2` directory. In CloudBerry Database, the interconnect is responsible for data transmission and synchronization between nodes, serving as a core component for distributed query execution.
25+
26+
Currently, the database supports three interconnect protocol implementations:
27+
- **TCP** (`contrib/interconnect/tcp`) - Reliable transmission based on TCP protocol
28+
- **UDP** (`contrib/interconnect/udp`) - High-performance transmission based on UDP protocol
29+
- **Proxy** (`contrib/interconnect/proxy`) - Proxy-based transmission
30+
31+
UDP2 is an architectural refactoring based on the original UDP protocol implementation, aimed at achieving complete separation between interconnect and the database kernel.
32+
33+
## Project Goals
34+
35+
The core objectives of the UDP2 protocol implementation are:
36+
37+
1. **Architecture Decoupling**: Completely separate the interconnect protocol implementation from the database kernel, enabling independent development and evolution
38+
2. **Independent Testing**: Enable end-to-end functional and performance testing of interconnect without depending on the database kernel
39+
3. **Rapid Diagnosis**: Quickly identify whether issues are at the interconnect level or database kernel level
40+
4. **Modular Design**: Provide clear interface boundaries for easier extension and maintenance
41+
42+
## Current Project Implementation Architecture
43+
44+
### Overall Architecture Design
45+
46+
UDP2 adopts a layered architecture design, primarily divided into two layers:
47+
48+
```
49+
┌─────────────────────────────────────────────────────────┐
50+
│ Database Kernel Layer │
51+
│ ┌────────────────────────────────────────────────────┐ │
52+
│ │ contrib/udp2/ │ │
53+
│ │ ┌─────────────────┐ ┌────────────────────────┐ │ │
54+
│ │ │ ic_modules.c │ │ ic_udp2.c │ │ │
55+
│ │ │ ic_modules.h │ │ ic_udp2.h │ │ │
56+
│ │ └─────────────────┘ └────────────────────────┘ │ │
57+
│ │ Adapter Layer (Database Adapter) │ │
58+
│ └────────────────────────────────────────────────────┘ │
59+
└─────────────────────────────────────────────────────────┘
60+
│
61+
│ C/C++ Interface
62+
▼
63+
┌──────────────────────────────────────────────────────────┐
64+
│ Independent IC Communication Library │
65+
│ ┌─────────────────────────────────────────────────────┐ │
66+
│ │ contrib/udp2/ic_common/ │ │
67+
│ │ ┌─────────────────┐ ┌─────────────────────────┐ │ │
68+
│ │ │ ic_types.h │ │ ic_utility.hpp │ │ │
69+
│ │ │ ic_except.hpp │ │ ic_faultinjection.h │ │ │
70+
│ │ └─────────────────┘ └─────────────────────────┘ │ │
71+
│ │ ┌────────────────────────────────────────────────┐ │ │
72+
│ │ │ contrib/udp2/ic_common/udp2/ │ │ │
73+
│ │ │ ┌─────────────────┐ ┌─────────────────────┐ │ │ │
74+
│ │ │ │ ic_udp2.h │ │ ic_udp2.hpp │ │ │ │
75+
│ │ │ │ ic_udp2.cpp │ │ic_udp2_internal.hpp │ │ │ │
76+
│ │ │ └─────────────────┘ └─────────────────────┘ │ │ │
77+
│ │ └────────────────────────────────────────────────┘ │ │
78+
│ │ Core Communication Library │ │
79+
│ └─────────────────────────────────────────────────────┘ │
80+
└──────────────────────────────────────────────────────────┘
81+
```
82+
83+
### Core Component Description
84+
85+
#### 1. Adapter Layer (`contrib/udp2/`)
86+
- **ic_modules.c/h**: Module registration and initialization, implementing the `MotionIPCLayer` interface
87+
- **ic_udp2.c/h**: Adapter layer between database kernel and ic_common library
88+
- Responsible for converting database kernel data structures to ic_common library standard interfaces
89+
90+
#### 2. Core Communication Library (`contrib/udp2/ic_common/`)
91+
- **ic_types.h**: Defines core data types and interfaces, decoupled from database kernel
92+
- **ic_utility.hpp**: Common utility functions and logging system
93+
- **ic_except.hpp**: Exception handling mechanism
94+
- **ic_faultinjection.h**: Fault injection testing support
95+
96+
#### 3. UDP2 Protocol Implementation (`contrib/udp2/ic_common/udp2/`)
97+
- **ic_udp2.h**: C language interface definition
98+
- **ic_udp2.hpp**: C++ interface definition
99+
- **ic_udp2.cpp**: Core protocol implementation
100+
- **ic_udp2_internal.hpp**: Internal implementation details
101+
102+
### Build System
103+
104+
UDP2 uses CMake build system with support for independent compilation:
105+
106+
```
107+
contrib/udp2/
108+
├── CMakeLists.txt # Main build configuration
109+
├── Makefile # PostgreSQL-compatible Makefile
110+
└── ic_common/
111+
├── CMakeLists.txt # ic_common library build configuration
112+
└── build/ # Build output directory
113+
```
114+
115+
Build process:
116+
1. First build the `ic_common` dynamic library (`libic_common.so`)
117+
2. Then build the `udp2` module (`udp2.so`), linking against the `ic_common` library
118+
119+
## How to Switch Database to This Protocol Implementation
120+
121+
### Enable UDP2 Support at Compile Time
122+
123+
1. **Configure compilation options**:
124+
```bash
125+
./configure --enable-ic-udp2 [other options]
126+
make && make install
127+
```
128+
129+
2. **Verify compilation results**:
130+
```bash
131+
# Check if udp2.so is generated
132+
ls -la $GPHOME/lib/postgresql/udp2.so
133+
134+
# Check if ic_common library is installed
135+
ls -la $GPHOME/lib/libic_common.so
136+
```
137+
138+
### Runtime Configuration
139+
140+
```bash
141+
# Set cluster to use UDP2 by default
142+
gpconfig -c gp_interconnect_type -v udp2
143+
144+
# Reload configuration
145+
gpstop -air
146+
```
147+
148+
```sql
149+
-- Check current interconnect type
150+
SHOW gp_interconnect_type;
151+
```
152+
153+
## Technical Details
154+
155+
### Interface Design
156+
157+
UDP2 achieves decoupling between database kernel and communication library through standardized C interfaces:
158+
159+
```c
160+
// Core interface functions (ic_common/udp2/ic_udp2.h)
161+
extern ICChunkTransportState* UDP2_SetupUDP(ICSliceTable *sliceTable,
162+
SessionMotionLayerIPCParam *param);
163+
extern void UDP2_TeardownUDP(ICChunkTransportState *transportStates, bool hasErrors);
164+
165+
// Data send/receive interfaces
166+
extern bool UDP2_SendData(ICChunkTransportState *transportStates,
167+
int16 motNodeID, int16 targetRoute,
168+
DataBlock *pblocks, int num, bool broadcast);
169+
extern void UDP2_RecvAny(ICChunkTransportState *transportStates,
170+
int16 motNodeID, int16 *srcRoute,
171+
GetDataLenInPacket getLen, DataBlock *data);
172+
```
173+
174+
### Data Structure Mapping
175+
176+
UDP2 defines lightweight data structures to replace complex database kernel structures:
177+
178+
```c
179+
// Lightweight process information (replaces CdbProcess)
180+
typedef struct ICCdbProcess {
181+
bool valid;
182+
char *listenerAddr;
183+
int listenerPort;
184+
int pid;
185+
int contentid;
186+
int dbid;
187+
} ICCdbProcess;
188+
189+
// Lightweight slice information (replaces ExecSlice)
190+
typedef struct ICExecSlice {
191+
int sliceIndex;
192+
int parentIndex;
193+
int numChildren;
194+
int *children;
195+
int numSegments;
196+
int numPrimaryProcesses;
197+
ICCdbProcess *primaryProcesses;
198+
} ICExecSlice;
199+
```
200+
201+
### Error Handling Mechanism
202+
203+
UDP2 implements a unified error handling mechanism:
204+
205+
```c
206+
typedef enum ErrorLevel {
207+
LEVEL_OK,
208+
LEVEL_ERROR,
209+
LEVEL_FATAL,
210+
} ErrorLevel;
211+
212+
// Error handling interfaces
213+
extern void SetLastError(ErrorLevel level, const char *msg);
214+
extern ICError* GetLastError();
215+
extern void ResetLastError();
216+
```
217+
218+
## Development and Debugging
219+
220+
### Independent Compilation Testing
221+
222+
```bash
223+
# Enter ic_common directory
224+
cd contrib/udp2/ic_common
225+
226+
# Create build directory
227+
mkdir build && cd build
228+
229+
# Configure and compile
230+
cmake -DCMAKE_BUILD_TYPE=Debug ..
231+
make -j
232+
```
233+
234+
### Debug Configuration
235+
236+
Enable verbose logging in development environment:
237+
238+
```sql
239+
-- Enable interconnect debug logging
240+
SET gp_log_interconnect = 'debug';
241+
242+
-- Set log level
243+
SET log_min_messages = 'debug1';
244+
245+
-- Enable detailed error information
246+
SET gp_interconnect_log_stats = on;
247+
```

‎contrib/udp2/ic_common/ic_utility.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1344,7 +1344,7 @@ comp_crc32c_sb8(uint32 crc, const void *data, size_t len)
13441344
extern "C" {
13451345
#endif
13461346

1347-
static ICError ic_error = {LEVEL_ERROR};
1347+
static thread_local ICError ic_error = {LEVEL_ERROR};
13481348

13491349
ICError*
13501350
GetLastError()

‎contrib/udp2/ic_common/ic_utility.hpp‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -81,16 +81,6 @@
8181
/*
8282
* CONTAINER_OF
8383
*/
84-
#define CONTAINER_OF(ptr, type, member) \
85-
({ \
86-
const typeof( ((type *)0)->member ) *__member_ptr = (ptr); \
87-
(type *)( (char *)__member_ptr - offsetof(type,member) ); \
88-
})
89-
90-
#ifdef CONTAINER_OF
91-
#undef CONTAINER_OF
92-
#endif
93-
9484
#define CONTAINER_OF(ptr, type, member) \
9585
( \
9686
reinterpret_cast<type*>(reinterpret_cast<char*>(ptr) - offsetof(type, member)) \

0 commit comments

Comments
 (0)