Commit dce5047
[SPARK-56122][PS] Use pandas-aware numeric dtype check in Series.cov
### What changes were proposed in this pull request?
This PR updates `pyspark.pandas.Series.cov` to use pandas' `is_numeric_dtype` instead of `np.issubdtype(..., np.number)` when validating the input Series dtypes.
The previous NumPy-based check works for plain NumPy dtypes, but it raises a `TypeError` for pandas extension dtypes such as `StringDtype`, which causes `Series.cov` to fail before it reaches the intended `unsupported dtype` error path.
This PR also updates the related `Series.cov` tests to:
- accept the dtype name shown by both pandas 2 (`object`) and pandas 3 (`str`) for non-numeric string inputs
- add extension dtype coverage for nullable `Float64`
- add a string extension dtype case using pandas `"string"`
### Why are the changes needed?
With pandas 3, `Series(["a", "b", "c"]).dtype` is a pandas extension dtype (`StringDtype`), and `np.issubdtype(self.dtype, np.number)` raises:
`TypeError: Cannot interpret '<StringDtype(na_value=nan)>' as a data type`
That means `Series.cov` fails in dtype validation instead of returning the expected `TypeError("unsupported dtype: ...")`.
Using `is_numeric_dtype` is more appropriate here because it is pandas-aware and works with both NumPy dtypes and pandas extension dtypes. This keeps the existing behavior for numeric inputs while making the validation path robust across pandas versions.
### Does this PR introduce _any_ user-facing change?
Yes.
For pandas extension dtypes such as pandas 3 string dtype, `pyspark.pandas.Series.cov` now raises the intended `TypeError("unsupported dtype: ...")` instead of failing earlier with a NumPy dtype interpretation error.
### How was this patch tested?
Added the related tests and the other existing tests should pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex GPT-5
Closes #54933 from ueshin/issues/SPARK-56122/cov.
Authored-by: Takuya Ueshin <ueshin@databricks.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>1 parent bda099a commit dce5047
2 files changed
Lines changed: 28 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| 57 | + | |
57 | 58 | | |
58 | 59 | | |
59 | 60 | | |
| |||
1092 | 1093 | | |
1093 | 1094 | | |
1094 | 1095 | | |
1095 | | - | |
| 1096 | + | |
1096 | 1097 | | |
1097 | | - | |
| 1098 | + | |
1098 | 1099 | | |
1099 | 1100 | | |
1100 | 1101 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
630 | 630 | | |
631 | 631 | | |
632 | 632 | | |
633 | | - | |
| 633 | + | |
634 | 634 | | |
635 | | - | |
| 635 | + | |
636 | 636 | | |
637 | 637 | | |
638 | 638 | | |
| |||
655 | 655 | | |
656 | 656 | | |
657 | 657 | | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
658 | 681 | | |
659 | 682 | | |
660 | 683 | | |
| |||
0 commit comments