Skip to content

Commit 7f7d92a

Browse files
committed
feat(types): support CHAR/VARCHAR/BINARY/VARBINARY in data type json parser
Previously CHAR/VARCHAR/BINARY/VARBINARY were registered as keywords but had no handling branch in ParseTypeByKeyword, so deserializing a schema that contained them failed with "Unsupported type: VARCHAR". Map CHAR/VARCHAR to arrow::utf8() and BINARY/VARBINARY to arrow::binary() (consistent with STRING and BYTES), parsing and discarding the optional length parameter. Add parser test cases covering these types with and without a length argument, and update the data type mapping doc accordingly.
1 parent b631683 commit 7f7d92a

3 files changed

Lines changed: 28 additions & 4 deletions

File tree

docs/source/user_guide/data_types.rst

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,48 +38,58 @@ and `Arrow DataTypes <https://arrow.apache.org/docs/format/Columnar.html#data-ty
3838
* - ``CHAR``
3939

4040
``CHAR(n)``
41-
- Not Supported
41+
- Utf8
4242
- Data type of a fixed-length character string.
4343

4444
The type can be declared using ``CHAR(n)`` where n is the number of code
4545
points. n must have a value between 1 and 2,147,483,647 (both inclusive).
4646
If no length is specified, n is equal to 1.
4747

48+
**Note:** Apache Arrow has no fixed-length string type, so the declared
49+
length ``n`` is not enforced; values are mapped to a variable-length ``Utf8``.
50+
4851
* - ``VARCHAR``
4952

5053
``VARCHAR(n)``
5154

52-
- Not Supported
55+
- Utf8
5356
- Data type of a variable-length character string.
5457

5558
The type can be declared using ``VARCHAR(n)`` where n is the maximum
5659
number of code points. n must have a value between 1 and 2,147,483,647
5760
(both inclusive). If no length is specified, n is equal to 1.
5861

62+
**Note:** The declared maximum length ``n`` is not enforced.
63+
5964
* - ``STRING``
6065
- Utf8
6166
- Data type of a variable-length character string. ``STRING`` is a synonym for ``VARCHAR(2147483647)``.
6267

6368
* - ``BINARY``
6469

6570
``BINARY(n)``
66-
- Not Supported
71+
- Binary
6772
- Data type of a fixed-length binary string (=a sequence of bytes).
6873

6974
The type can be declared using ``BINARY(n)`` where n is the number of
7075
bytes. n must have a value between 1 and 2,147,483,647 (both inclusive).
7176
If no length is specified, n is equal to 1.
7277

78+
**Note:** Apache Arrow has no fixed-length binary type here, so the declared
79+
length ``n`` is not enforced; values are mapped to a variable-length ``Binary``.
80+
7381
* - ``VARBINARY``
7482

7583
``VARBINARY(n)``
76-
- Not Supported
84+
- Binary
7785
- Data type of a variable-length binary string (=a sequence of bytes).
7886

7987
The type can be declared using ``VARBINARY(n)`` where n is the maximum
8088
number of bytes. n must have a value between 1 and 2,147,483,647
8189
(both inclusive). If no length is specified, n is equal to 1.
8290

91+
**Note:** The declared maximum length ``n`` is not enforced.
92+
8393
* - ``BYTES``
8494
- Binary
8595
- ``BYTES`` is a synonym for ``VARBINARY(2147483647)``.

src/paimon/common/types/data_type_json_parser.cpp

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -465,6 +465,12 @@ Result<std::shared_ptr<arrow::DataType>> TokenParser::ParseTypeWithNullability(b
465465
Result<std::shared_ptr<arrow::DataType>> TokenParser::ParseTypeByKeyword(bool* is_blob) {
466466
PAIMON_RETURN_NOT_OK(NextToken(TokenType::KEYWORD));
467467
switch (TokenAsKeyword()) {
468+
case Keyword::CHAR:
469+
case Keyword::VARCHAR:
470+
return ParseStringType<arrow::StringType>();
471+
case Keyword::BINARY:
472+
case Keyword::VARBINARY:
473+
return ParseStringType<arrow::BinaryType>();
468474
case Keyword::BYTES:
469475
return arrow::binary();
470476
case Keyword::BLOB: {

src/paimon/common/types/data_type_json_parser_test.cpp

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,14 @@ TEST(DataTypeJsonParserTest, ParseTypeAtomicTypeSuccess) {
111111
{"TIMESTAMP_LTZ(9)", arrow::timestamp(arrow::TimeUnit::NANO, timezone)},
112112
{"BYTES", arrow::binary()},
113113
{"STRING", arrow::utf8()},
114+
{"CHAR", arrow::utf8()},
115+
{"CHAR(10)", arrow::utf8()},
116+
{"VARCHAR", arrow::utf8()},
117+
{"VARCHAR(10)", arrow::utf8()},
118+
{"BINARY", arrow::binary()},
119+
{"BINARY(10)", arrow::binary()},
120+
{"VARBINARY", arrow::binary()},
121+
{"VARBINARY(10)", arrow::binary()},
114122
};
115123

116124
for (const auto& test_case : test_cases) {

0 commit comments

Comments
 (0)