Skip to content

Commit 82868de

Browse files
zhicwuzhicwu
committed
Fit and finish
1 parent b37e5b5 commit 82868de

6 files changed

Lines changed: 277 additions & 4 deletions

File tree

‎.github/workflows/benchmark.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ jobs:
6262
mvn --batch-mode --update-snapshots -q -DskipTests install
6363
cd clickhouse-benchmark
6464
java -DclickhouseVersion="21.8" -jar target/benchmarks.jar -rf text \
65-
-p client=clickhouse-http-jdbc -p client=clickhouse-grpc-jdbc -p type=default Basic
65+
-p client=clickhouse-http-jdbc1 -p client=clickhouse-grpc-jdbc -p type=object Basic
6666
echo "BENCHMARK_REPORT<<EOF" >> $GITHUB_ENV
6767
cat jmh-result.text >> $GITHUB_ENV
6868
echo "EOF" >> $GITHUB_ENV

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ Java 8 or higher is required in order to use Java client([clickhouse-client](htt
8181

8282
| Property | Default | Description |
8383
| -------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
84-
| continueBatchOnError | `true` | Whether to continue batch processing when error occurred |
84+
| continueBatchOnError | `false` | Whether to continue batch processing when error occurred |
8585
| custom_http_headers | | comma separated custom http headers, for example: `User-Agent=client1,X-Gateway-Id=123` |
8686
| custom_http_params | | comma separated custom http query parameters, for example: `extremes=0,max_result_rows=100` |
8787
| jdbcCompliance | `true` | Whether to support standard synchronous UPDATE/DELETE and fake transaction |

‎clickhouse-client/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
Async Java client for ClickHouse. `clickhouse-client` is an abstract module, so it does not work by itself until being used together with an implementation like `clickhouse-grpc-client` or `clickhouse-http-client`.
44

5+
56
## Quick Start
67

78
```xml

‎clickhouse-jdbc/README.md‎

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# ClickHouse JDBC driver
2+
3+
Build on top of `clickhouse-client`, `clickhouse-jdbc` follows JDBC standards and provides additional features like custom type mapping, fake transaction, and standard synchronous UPDATE and DELETE statement etc., so that it can be easily used together with legacy applications and tools.
4+
5+
Keep in mind that `clickhouse-jdbc` is synchronous, and in general it has more overheads(e.g. SQL parsing and type mapping/conversion etc.). You should consider `clickhouse-client` when performance is critical and/or you prefer more direct way to work with ClickHouse.
6+
7+
## Maven Dependency
8+
9+
```xml
10+
<dependency>
11+
<!-- will stop using ru.yandex.clickhouse starting from 0.4.0 -->
12+
<groupId>com.clickhouse</groupId>
13+
<artifactId>clickhouse-jdbc</artifactId>
14+
<version>0.3.2</version>
15+
</dependency>
16+
```
17+
18+
## Configuration
19+
20+
**Driver Class**: `com.clickhouse.jdbc.ClickHouseDriver`
21+
22+
Note: `ru.yandex.clickhouse.ClickHouseDriver` and everything under `ru.yandex.clickhouse` will be removed starting from 0.4.0.
23+
24+
**URL Syntax**: `jdbc:<prefix>[:<protocol>]://<host>:[<port>][/<database>[?param1=value1&param2=value2]]`, for examples:
25+
26+
- `jdbc:ch:grpc://localhost` is same as `jdbc:clickhouse:grpc://localhost:9100`
27+
- `jdbc:ch:grpc://localhost` is same as `jdbc:clickhouse:grpc://localhost:9100`)
28+
- `jdbc:ch://localhost/test?socket_timeout=120000`
29+
30+
**Connection Properties**:
31+
32+
| Property | Default | Description |
33+
| -------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
34+
| continueBatchOnError | `false` | Whether to continue batch processing when error occurred |
35+
| custom_http_headers | | comma separated custom http headers, for example: `User-Agent=client1,X-Gateway-Id=123` |
36+
| custom_http_params | | comma separated custom http query parameters, for example: `extremes=0,max_result_rows=100` |
37+
| jdbcCompliance | `true` | Whether to support standard synchronous UPDATE/DELETE and fake transaction |
38+
| typeMappings | | Customize mapping between ClickHouse data type and Java class, which will affect result of both [getColumnType()](https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSetMetaData.html#getColumnType-int-) and [getObject(Class<?>)](https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getObject-java.lang.String-java.lang.Class-). For example: `UInt128=java.lang.String,UInt256=java.lang.String` |
39+
| wrapperObject | `false` | Whether [getObject()](https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#getObject-int-) should return java.sql.Array / java.sql.Struct for Array / Tuple. |
40+
41+
Note: please refer to [JDBC specific configuration](https://github.com/ClickHouse/clickhouse-jdbc/blob/master/clickhouse-jdbc/src/main/java/com/clickhouse/jdbc/JdbcConfig.java) and client options([common](https://github.com/ClickHouse/clickhouse-jdbc/blob/master/clickhouse-client/src/main/java/com/clickhouse/client/config/ClickHouseClientOption.java), [http](https://github.com/ClickHouse/clickhouse-jdbc/blob/master/clickhouse-http-client/src/main/java/com/clickhouse/client/http/config/ClickHouseHttpOption.java) and [grpc](https://github.com/ClickHouse/clickhouse-jdbc/blob/master/clickhouse-grpc-client/src/main/java/com/clickhouse/client/grpc/config/ClickHouseGrpcOption.java)) for more.
42+
43+
## Examples
44+
45+
<details>
46+
<summary>Connect to ClickHouse and issue a query...</summary>
47+
48+
```java
49+
String url = "jdbc:ch://my-server/system"; // use http protocol and port 8123 by default
50+
ClickHouseDataSource dataSource = new ClickHouseDataSource(url, new Properties());
51+
try (Connection conn = dataSource.getConnection("default", "password");
52+
Statement stmt = conn.createStatement()) {
53+
ResultSet rs = stmt.executeQuery("select * from numbers(50000)");
54+
while(rs.next()) {
55+
// ...
56+
}
57+
}
58+
```
59+
60+
</details>
61+
62+
<details>
63+
<summary>Batch insert...</summary>
64+
65+
Tips:
66+
1. Use `PreparedStatement` instead of `Statement`
67+
2. Use [input function](https://clickhouse.com/docs/en/sql-reference/table-functions/input/) whenever possible
68+
69+
```java
70+
// create table mytable(id String, ts DateTime64(3), desc Nullable(String)) engine=Memory
71+
72+
// 1. recommended as it performs the best
73+
try (PreparedStatement ps = conn.prepareStatement(
74+
"insert into mytable select col1, col2 from input('col1 String, col2 DateTime64(3), col3 Int32')")) {
75+
// the column definition will be parsed so the driver knows there are 3 parameters: col1, col2 and col3
76+
ps.setString(1, "test"); // col1
77+
ps.setObject(2, LocalDateTime.now()); // col2, setTimestamp is slow and not recommended
78+
ps.setInt(3, 123); // col3
79+
ps.addBatch(); // parameters will be write into stream in binary format
80+
...
81+
ps.executeBatch(); // stream everything on-hand into ClickHouse
82+
}
83+
84+
// 2. easier to use but slower compare to input function
85+
try (PreparedStatement ps = conn.prepareStatement("insert into mytable(* except (desc))")) {
86+
// the driver will issue query "select * except (desc) from mytable where 0" for type inferring
87+
// since desc column is excluded, we know there are only two parameters: col1 and col2
88+
ps.setString(1, "test"); // col1
89+
ps.setObject(2, LocalDateTime.now()); // col2
90+
ps.addBatch(); // parameters will be write into stream in binary format
91+
...
92+
ps.executeBatch(); // stream everything on-hand into ClickHouse
93+
}
94+
95+
// 3. not recommended as it's based on a large SQL
96+
try (PreparedStatement ps = conn.prepareStatement("insert into mytable values(?,?,?)")) {
97+
ps.setString(1, "test"); // col1
98+
ps.setObject(2, LocalDateTime.now()); // col2
99+
ps.setString(3, null); // col3
100+
ps.addBatch(); // append parameters to the query
101+
...
102+
ps.executeBatch(); // issue the composed query: insert into mytable values(...)(...)...(...)
103+
}
104+
```
105+
106+
</details>
107+
108+
<details>
109+
<summary>Handling DateTime and time zone...</summary>
110+
111+
Please to use `java.time.LocalDateTime` or `java.time.OffsetDateTime` instead of `java.sql.Timestamp`, and `java.time.LocalDate` instead of `java.sql.Date`.
112+
113+
```java
114+
try (PreparedStatement ps = conn.prepareStatement("select date_time from mytable where date_time > ?")) {
115+
ps.setObject(2, LocalDateTime.now());
116+
ResultSet rs = ps.executeQuery();
117+
while(rs.next()) {
118+
LocalDateTime dateTime = (LocalDateTime) rs.getObject(1);
119+
}
120+
...
121+
}
122+
```
123+
124+
</details>
125+
126+
<details>
127+
<summary>Handling AggregateFunction...</summary>
128+
129+
As of now, only `groupBitmap` is supported.
130+
131+
```java
132+
// batch insert using input function
133+
try (ClickHouseConnection conn = newConnection(props);
134+
Statement s = conn.createStatement();
135+
PreparedStatement stmt = conn.prepareStatement(
136+
"insert into test_batch_input select id, name, value from input('id Int32, name Nullable(String), desc Nullable(String), value AggregateFunction(groupBitmap, UInt32)')")) {
137+
s.execute("drop table if exists test_batch_input;"
138+
+ "create table test_batch_input(id Int32, name Nullable(String), value AggregateFunction(groupBitmap, UInt32))engine=Memory");
139+
Object[][] objs = new Object[][] {
140+
new Object[] { 1, "a", "aaaaa", ClickHouseBitmap.wrap(1, 2, 3, 4, 5) },
141+
new Object[] { 2, "b", null, ClickHouseBitmap.wrap(6, 7, 8, 9, 10) },
142+
new Object[] { 3, null, "33333", ClickHouseBitmap.wrap(11, 12, 13) }
143+
};
144+
for (Object[] v : objs) {
145+
stmt.setInt(1, (int) v[0]);
146+
stmt.setString(2, (String) v[1]);
147+
stmt.setString(3, (String) v[2]);
148+
stmt.setObject(4, v[3]);
149+
stmt.addBatch();
150+
}
151+
int[] results = stmt.executeBatch();
152+
...
153+
}
154+
155+
// use bitmap as query parameter
156+
try (PreparedStatement stmt = conn.prepareStatement(
157+
"SELECT bitmapContains(my_bitmap, toUInt32(1)) as v1, bitmapContains(my_bitmap, toUInt32(2)) as v2 from {tt 'ext_table'}")) {
158+
stmt.setObject(1, ClickHouseExternalTable.builder().name("ext_table")
159+
.columns("my_bitmap AggregateFunction(groupBitmap,UInt32)").format(ClickHouseFormat.RowBinary)
160+
.content(new ByteArrayInputStream(ClickHouseBitmap.wrap(1, 3, 5).toBytes()))
161+
.asTempTable()
162+
.build());
163+
ResultSet rs = stmt.executeQuery();
164+
Assert.assertTrue(rs.next());
165+
Assert.assertEquals(rs.getInt(1), 1);
166+
Assert.assertEquals(rs.getInt(2), 0);
167+
Assert.assertFalse(rs.next());
168+
}
169+
```
170+
171+
</details>
172+
173+
174+
<details>
175+
<summary>Before 0.3.2...</summary>
176+
177+
#### **Basic**
178+
179+
```java
180+
String url = "jdbc:clickhouse://localhost:8123/test";
181+
ClickHouseProperties properties = new ClickHouseProperties();
182+
// set connection options - see more defined in ClickHouseConnectionSettings
183+
properties.setClientName("Agent #1");
184+
...
185+
// set default request options - more in ClickHouseQueryParam
186+
properties.setSessionId("default-session-id");
187+
...
188+
189+
ClickHouseDataSource dataSource = new ClickHouseDataSource(url, properties)
190+
String sql = "select * from mytable";
191+
Map<ClickHouseQueryParam, String> additionalDBParams = new HashMap<>();
192+
// set request options, which will override the default ones in ClickHouseProperties
193+
additionalDBParams.put(ClickHouseQueryParam.SESSION_ID, "new-session-id");
194+
...
195+
try (ClickHouseConnection conn = dataSource.getConnection();
196+
ClickHouseStatement stmt = conn.createStatement();
197+
ResultSet rs = stmt.executeQuery(sql, additionalDBParams)) {
198+
...
199+
}
200+
```
201+
202+
Additionally, if you have a few instances, you can use `BalancedClickhouseDataSource`.
203+
204+
#### **Extended API**
205+
206+
In order to provide non-JDBC complaint data manipulation functionality, proprietary API exists.
207+
Entry point for API is `ClickHouseStatement#write()` method.
208+
209+
1) Importing file into table
210+
211+
```java
212+
import ru.yandex.clickhouse.ClickHouseStatement;
213+
ClickHouseStatement sth = connection.createStatement();
214+
sth
215+
.write() // Write API entrypoint
216+
.table("default.my_table") // where to write data
217+
.option("format_csv_delimiter", ";") // specific param
218+
.data(new File("/path/to/file.csv.gz"), ClickHouseFormat.CSV, ClickHouseCompression.gzip) // specify input
219+
.send();
220+
```
221+
222+
2) Configurable send
223+
224+
```java
225+
import ru.yandex.clickhouse.ClickHouseStatement;
226+
ClickHouseStatement sth = connection.createStatement();
227+
sth
228+
.write()
229+
.sql("INSERT INTO default.my_table (a,b,c)")
230+
.data(new MyCustomInputStream(), ClickHouseFormat.JSONEachRow)
231+
.dataCompression(ClickHouseCompression.brotli)
232+
.addDbParam(ClickHouseQueryParam.MAX_PARALLEL_REPLICAS, 2)
233+
.send();
234+
```
235+
236+
3) Send data in binary formatted with custom user callback
237+
238+
```java
239+
import ru.yandex.clickhouse.ClickHouseStatement;
240+
ClickHouseStatement sth = connection.createStatement();
241+
sth.write().send("INSERT INTO test.writer", new ClickHouseStreamCallback() {
242+
@Override
243+
public void writeTo(ClickHouseRowBinaryStream stream) throws IOException {
244+
for (int i = 0; i < 10; i++) {
245+
stream.writeInt32(i);
246+
stream.writeString("Name " + i);
247+
}
248+
}
249+
},
250+
ClickHouseFormat.RowBinary); // RowBinary or Native are supported
251+
```
252+
</details>

‎clickhouse-jdbc/src/main/java/com/clickhouse/jdbc/JdbcConfig.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ public class JdbcConfig {
3232
private static final String BOOLEAN_TRUE = "true";
3333

3434
private static final String DEFAULT_AUTO_COMMIT = BOOLEAN_TRUE;
35-
private static final String DEFAULT_CONTINUE_BATCH = BOOLEAN_TRUE;
35+
private static final String DEFAULT_CONTINUE_BATCH = BOOLEAN_FALSE;
3636
private static final String DEFAULT_FETCH_SIZE = "0";
3737
private static final String DEFAULT_JDBC_COMPLIANT = BOOLEAN_TRUE;
3838
private static final String DEFAULT_NAMED_PARAM = BOOLEAN_FALSE;

‎clickhouse-jdbc/src/test/java/com/clickhouse/jdbc/ClickHousePreparedStatementTest.java‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -283,8 +283,28 @@ public void testBatchInsert() throws SQLException {
283283
}
284284

285285
@Test(groups = "integration")
286-
public void testBatchInput() throws SQLException {
286+
public void testQueryWithDateTime() throws SQLException {
287287
try (ClickHouseConnection conn = newConnection(new Properties());
288+
Statement s = conn.createStatement();
289+
PreparedStatement stmt = conn.prepareStatement(
290+
"select id, dt from test_query_datetime where dt > ? order by id")) {
291+
s.execute("drop table if exists test_query_datetime;"
292+
+ "create table test_query_datetime(id Int32, dt DateTime32)engine=Memory;"
293+
+ "insert into test_query_datetime values(1, '2021-03-25 12:34:56'), (2, '2021-03-26 12:34:56')");
294+
stmt.setObject(1, LocalDateTime.of(2021, 3, 25, 12, 34, 57));
295+
ResultSet rs = stmt.executeQuery();
296+
Assert.assertTrue(rs.next());
297+
Assert.assertEquals(rs.getInt(1), 2);
298+
Assert.assertEquals(rs.getObject(2), LocalDateTime.of(2021, 3, 26, 12, 34, 56));
299+
Assert.assertFalse(rs.next());
300+
}
301+
}
302+
303+
@Test(groups = "integration")
304+
public void testBatchInput() throws SQLException {
305+
Properties props = new Properties();
306+
props.setProperty("continueBatchOnError", "true");
307+
try (ClickHouseConnection conn = newConnection(props);
288308
Statement s = conn.createStatement();
289309
PreparedStatement stmt = conn.prepareStatement(
290310
"insert into test_batch_input select id, name, value from input('id Int32, name Nullable(String), desc Nullable(String), value AggregateFunction(groupBitmap, UInt32)')")) {

0 commit comments

Comments
 (0)