Skip to content

Commit e06b29d

Browse files
author
B Vadlamani
committed
address_review_comments
1 parent 245b63a commit e06b29d

2 files changed

Lines changed: 14 additions & 162 deletions

File tree

‎docs/source/user-guide/latest/compatibility.md‎

Lines changed: 13 additions & 122 deletions
Original file line numberDiff line numberDiff line change
@@ -73,134 +73,19 @@ should not be used in production. The feature will be enabled in a future releas
7373

7474
Cast operations in Comet fall into three levels of support:
7575

76-
- **Compatible**: The results match Apache Spark
77-
- **Incompatible**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
76+
- **C (Compatible)**: The results match Apache Spark
77+
- **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
7878
will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
7979
`spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
8080
recommended for production use.
81-
- **Unsupported**: Comet does not provide a native version of this cast expression and the query stage will fall back to
81+
- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
8282
Spark.
83+
- **N/A**: Spark does not support this cast.
8384

84-
### Compatible Casts
85-
86-
The following cast operations are generally compatible with Spark except for the differences noted here.
85+
### Legacy Mode
8786

8887
<!-- WARNING! DO NOT MANUALLY MODIFY CONTENT BETWEEN THE BEGIN AND END TAGS -->
8988

90-
<!--BEGIN:COMPAT_CAST_TABLE-->
91-
<!-- prettier-ignore-start -->
92-
| From Type | To Type | Notes |
93-
|-|-|-|
94-
| boolean | byte | |
95-
| boolean | short | |
96-
| boolean | integer | |
97-
| boolean | long | |
98-
| boolean | float | |
99-
| boolean | double | |
100-
| boolean | string | |
101-
| byte | boolean | |
102-
| byte | short | |
103-
| byte | integer | |
104-
| byte | long | |
105-
| byte | float | |
106-
| byte | double | |
107-
| byte | decimal | |
108-
| byte | string | |
109-
| short | boolean | |
110-
| short | byte | |
111-
| short | integer | |
112-
| short | long | |
113-
| short | float | |
114-
| short | double | |
115-
| short | decimal | |
116-
| short | string | |
117-
| integer | boolean | |
118-
| integer | byte | |
119-
| integer | short | |
120-
| integer | long | |
121-
| integer | float | |
122-
| integer | double | |
123-
| integer | decimal | |
124-
| integer | string | |
125-
| long | boolean | |
126-
| long | byte | |
127-
| long | short | |
128-
| long | integer | |
129-
| long | float | |
130-
| long | double | |
131-
| long | decimal | |
132-
| long | string | |
133-
| float | boolean | |
134-
| float | byte | |
135-
| float | short | |
136-
| float | integer | |
137-
| float | long | |
138-
| float | double | |
139-
| float | string | There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45 |
140-
| double | boolean | |
141-
| double | byte | |
142-
| double | short | |
143-
| double | integer | |
144-
| double | long | |
145-
| double | float | |
146-
| double | string | There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45 |
147-
| decimal | boolean | |
148-
| decimal | byte | |
149-
| decimal | short | |
150-
| decimal | integer | |
151-
| decimal | long | |
152-
| decimal | float | |
153-
| decimal | double | |
154-
| decimal | decimal | |
155-
| decimal | string | There can be formatting differences in some case due to Spark using scientific notation where Comet does not |
156-
| string | boolean | |
157-
| string | byte | |
158-
| string | short | |
159-
| string | integer | |
160-
| string | long | |
161-
| string | float | |
162-
| string | double | |
163-
| string | binary | |
164-
| string | date | Only supports years between 262143 BC and 262142 AD |
165-
| binary | string | |
166-
| date | string | |
167-
| timestamp | long | |
168-
| timestamp | string | |
169-
| timestamp | date | |
170-
<!-- prettier-ignore-end -->
171-
<!--END:COMPAT_CAST_TABLE-->
172-
173-
### Incompatible Casts
174-
175-
The following cast operations are not compatible with Spark for all inputs and are disabled by default.
176-
177-
<!-- WARNING! DO NOT MANUALLY MODIFY CONTENT BETWEEN THE BEGIN AND END TAGS -->
178-
179-
<!--BEGIN:INCOMPAT_CAST_TABLE-->
180-
<!-- prettier-ignore-start -->
181-
| From Type | To Type | Notes |
182-
|-|-|-|
183-
| float | decimal | There can be rounding differences |
184-
| double | decimal | There can be rounding differences |
185-
| string | decimal | Does not support fullwidth unicode digits (e.g \\uFF10)
186-
or strings containing null bytes (e.g \\u0000) |
187-
| string | timestamp | Not all valid formats are supported |
188-
<!-- prettier-ignore-end -->
189-
<!--END:INCOMPAT_CAST_TABLE-->
190-
191-
### Unsupported Casts
192-
193-
Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the
194-
[tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
195-
196-
### Cast Support by Eval Mode
197-
198-
The following tables show cast support levels for each evaluation mode.
199-
200-
**Legend:** C = Compatible, I = Incompatible, U = Unsupported, - = Same type, N = Spark does not support the cast
201-
202-
#### LEGACY Mode
203-
20489
<!--BEGIN:CAST_LEGACY_TABLE-->
20590
<!-- prettier-ignore-start -->
20691
| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp |
@@ -232,7 +117,9 @@ The following tables show cast support levels for each evaluation mode.
232117
- **string -> timestamp**: Not all valid formats are supported
233118
<!--END:CAST_LEGACY_TABLE-->
234119

235-
#### TRY Mode
120+
### Try Mode
121+
122+
<!-- WARNING! DO NOT MANUALLY MODIFY CONTENT BETWEEN THE BEGIN AND END TAGS -->
236123

237124
<!--BEGIN:CAST_TRY_TABLE-->
238125
<!-- prettier-ignore-start -->
@@ -265,7 +152,9 @@ The following tables show cast support levels for each evaluation mode.
265152
- **string -> timestamp**: Not all valid formats are supported
266153
<!--END:CAST_TRY_TABLE-->
267154

268-
#### ANSI Mode
155+
### ANSI Mode
156+
157+
<!-- WARNING! DO NOT MANUALLY MODIFY CONTENT BETWEEN THE BEGIN AND END TAGS -->
269158

270159
<!--BEGIN:CAST_ANSI_TABLE-->
271160
<!-- prettier-ignore-start -->
@@ -297,3 +186,5 @@ The following tables show cast support levels for each evaluation mode.
297186
or strings containing null bytes (e.g \\u0000)
298187
- **string -> timestamp**: ANSI mode not supported
299188
<!--END:CAST_ANSI_TABLE-->
189+
190+
See the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.

‎spark/src/main/scala/org/apache/comet/GenerateDocs.scala‎

Lines changed: 1 addition & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -110,46 +110,7 @@ object GenerateDocs {
110110
val w = new BufferedOutputStream(new FileOutputStream(filename))
111111
for (line <- lines) {
112112
w.write(s"${line.stripTrailing()}\n".getBytes)
113-
if (line.trim == "<!--BEGIN:COMPAT_CAST_TABLE-->") {
114-
w.write("<!-- prettier-ignore-start -->\n".getBytes)
115-
w.write("| From Type | To Type | Notes |\n".getBytes)
116-
w.write("|-|-|-|\n".getBytes)
117-
for (fromType <- CometCast.supportedTypes) {
118-
for (toType <- CometCast.supportedTypes) {
119-
if (Cast.canCast(fromType, toType) && (fromType != toType || fromType.typeName
120-
.contains("decimal"))) {
121-
val fromTypeName = fromType.typeName.replace("(10,2)", "")
122-
val toTypeName = toType.typeName.replace("(10,2)", "")
123-
CometCast.isSupported(fromType, toType, None, CometEvalMode.LEGACY) match {
124-
case Compatible(notes) =>
125-
val notesStr = notes.getOrElse("").trim
126-
w.write(s"| $fromTypeName | $toTypeName | $notesStr |\n".getBytes)
127-
case _ =>
128-
}
129-
}
130-
}
131-
}
132-
w.write("<!-- prettier-ignore-end -->\n".getBytes)
133-
} else if (line.trim == "<!--BEGIN:INCOMPAT_CAST_TABLE-->") {
134-
w.write("<!-- prettier-ignore-start -->\n".getBytes)
135-
w.write("| From Type | To Type | Notes |\n".getBytes)
136-
w.write("|-|-|-|\n".getBytes)
137-
for (fromType <- CometCast.supportedTypes) {
138-
for (toType <- CometCast.supportedTypes) {
139-
if (Cast.canCast(fromType, toType) && fromType != toType) {
140-
val fromTypeName = fromType.typeName.replace("(10,2)", "")
141-
val toTypeName = toType.typeName.replace("(10,2)", "")
142-
CometCast.isSupported(fromType, toType, None, CometEvalMode.LEGACY) match {
143-
case Incompatible(notes) =>
144-
val notesStr = notes.getOrElse("").trim
145-
w.write(s"| $fromTypeName | $toTypeName | $notesStr |\n".getBytes)
146-
case _ =>
147-
}
148-
}
149-
}
150-
}
151-
w.write("<!-- prettier-ignore-end -->\n".getBytes)
152-
} else if (line.trim == "<!--BEGIN:CAST_LEGACY_TABLE-->") {
113+
if (line.trim == "<!--BEGIN:CAST_LEGACY_TABLE-->") {
153114
writeCastMatrixForMode(w, CometEvalMode.LEGACY)
154115
} else if (line.trim == "<!--BEGIN:CAST_TRY_TABLE-->") {
155116
writeCastMatrixForMode(w, CometEvalMode.TRY)

0 commit comments

Comments
 (0)