Commit 61a79ca
committed
[SPARK-52622][PS] Avoid CAST_INVALID_INPUT of
### What changes were proposed in this pull request?
Avoid CAST_INVALID_INPUT of `DataFrame.melt` in ANSI mode
### Why are the changes needed?
Ensure pandas on Spark works well with ANSI mode on.
Part of https://issues.apache.org/jira/browse/SPARK-52556.
### Does this PR introduce _any_ user-facing change?
Yes, `DataFrame.melt` work well with ANSI.
```py
>>> import pandas as pd
>>> import numpy as np
>>>
>>> ps.set_option("compute.fail_on_ansi_mode", False)
>>> ps.set_option("compute.ansi_mode_support", True)
>>> df = ps.DataFrame({'A': {0: 'a', 1: 'b', 2: 'c'},
... 'B': {0: 1, 1: 3, 2: 5},
... 'C': {0: 2, 1: 4, 2: 6}},
... columns=['A', 'B', 'C'])
```
BEFORE
```py
>>> ps.melt(df)
...
pyspark.errors.exceptions.captured.NumberFormatException: [CAST_INVALID_INPUT] The value 'a' of the type "STRING" cannot be cast to "BIGINT" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. SQLSTATE: 22018
```
AFTER
```py
>>> ps.melt(df)
variable value
0 A a
1 B 1
2 C 2
3 A b
4 B 3
5 C 4
6 A c
7 B 5
8 C 6
```
### How was this patch tested?
Unit tests. Commands below pass
```
SPARK_ANSI_SQL_MODE=true ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.computation.test_melt FrameMeltTests.test_melt"
SPARK_ANSI_SQL_MODE=false ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.computation.test_melt FrameMeltTests.test_melt"
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #51326 from xinrong-meng/melt.
Authored-by: Xinrong Meng <xinrong@apache.org>
Signed-off-by: Xinrong Meng <xinrong@apache.org>DataFrame.melt in ANSI mode1 parent dc8fba6 commit 61a79ca
1 file changed
Lines changed: 19 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10617 | 10617 | | |
10618 | 10618 | | |
10619 | 10619 | | |
| 10620 | + | |
| 10621 | + | |
| 10622 | + | |
| 10623 | + | |
| 10624 | + | |
| 10625 | + | |
| 10626 | + | |
| 10627 | + | |
| 10628 | + | |
| 10629 | + | |
10620 | 10630 | | |
10621 | 10631 | | |
10622 | 10632 | | |
10623 | 10633 | | |
10624 | 10634 | | |
10625 | | - | |
| 10635 | + | |
| 10636 | + | |
| 10637 | + | |
| 10638 | + | |
| 10639 | + | |
| 10640 | + | |
| 10641 | + | |
| 10642 | + | |
| 10643 | + | |
10626 | 10644 | | |
10627 | 10645 | | |
10628 | 10646 | | |
| |||
0 commit comments