Skip to content

Commit dce5047

Browse files
ueshindongjoon-hyun
authored andcommitted
[SPARK-56122][PS] Use pandas-aware numeric dtype check in Series.cov
### What changes were proposed in this pull request? This PR updates `pyspark.pandas.Series.cov` to use pandas' `is_numeric_dtype` instead of `np.issubdtype(..., np.number)` when validating the input Series dtypes. The previous NumPy-based check works for plain NumPy dtypes, but it raises a `TypeError` for pandas extension dtypes such as `StringDtype`, which causes `Series.cov` to fail before it reaches the intended `unsupported dtype` error path. This PR also updates the related `Series.cov` tests to: - accept the dtype name shown by both pandas 2 (`object`) and pandas 3 (`str`) for non-numeric string inputs - add extension dtype coverage for nullable `Float64` - add a string extension dtype case using pandas `"string"` ### Why are the changes needed? With pandas 3, `Series(["a", "b", "c"]).dtype` is a pandas extension dtype (`StringDtype`), and `np.issubdtype(self.dtype, np.number)` raises: `TypeError: Cannot interpret '<StringDtype(na_value=nan)>' as a data type` That means `Series.cov` fails in dtype validation instead of returning the expected `TypeError("unsupported dtype: ...")`. Using `is_numeric_dtype` is more appropriate here because it is pandas-aware and works with both NumPy dtypes and pandas extension dtypes. This keeps the existing behavior for numeric inputs while making the validation path robust across pandas versions. ### Does this PR introduce _any_ user-facing change? Yes. For pandas extension dtypes such as pandas 3 string dtype, `pyspark.pandas.Series.cov` now raises the intended `TypeError("unsupported dtype: ...")` instead of failing earlier with a NumPy dtype interpretation error. ### How was this patch tested? Added the related tests and the other existing tests should pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex GPT-5 Closes #54933 from ueshin/issues/SPARK-56122/cov. Authored-by: Takuya Ueshin <ueshin@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent bda099a commit dce5047

2 files changed

Lines changed: 28 additions & 4 deletions

File tree

python/pyspark/pandas/series.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454
from pandas.api.types import (
5555
is_list_like,
5656
is_hashable,
57+
is_numeric_dtype,
5758
CategoricalDtype,
5859
)
5960
from pandas.tseries.frequencies import DateOffset # type: ignore[attr-defined]
@@ -1092,9 +1093,9 @@ def cov(self, other: "Series", min_periods: Optional[int] = None, ddof: int = 1)
10921093
"""
10931094
if not isinstance(other, Series):
10941095
raise TypeError("unsupported type: %s" % type(other))
1095-
if not np.issubdtype(self.dtype, np.number): # type: ignore[arg-type]
1096+
if not is_numeric_dtype(self.dtype):
10961097
raise TypeError("unsupported dtype: %s" % self.dtype)
1097-
if not np.issubdtype(other.dtype, np.number): # type: ignore[arg-type]
1098+
if not is_numeric_dtype(other.dtype):
10981099
raise TypeError("unsupported dtype: %s" % other.dtype)
10991100
if not isinstance(ddof, int):
11001101
raise TypeError("ddof must be integer")

python/pyspark/pandas/tests/series/test_stat.py

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -630,9 +630,9 @@ def test_cov(self):
630630
index=[0, 1, 2],
631631
)
632632
psdf = ps.from_pandas(pdf)
633-
with self.assertRaisesRegex(TypeError, "unsupported dtype: object"):
633+
with self.assertRaisesRegex(TypeError, "unsupported dtype: (object|str)"):
634634
psdf["s1"].cov(psdf["s2"])
635-
with self.assertRaisesRegex(TypeError, "unsupported dtype: object"):
635+
with self.assertRaisesRegex(TypeError, "unsupported dtype: (object|str)"):
636636
psdf["s2"].cov(psdf["s1"])
637637
with self.assertRaisesRegex(TypeError, "ddof must be integer"):
638638
psdf["s2"].cov(psdf["s2"], ddof="ddof")
@@ -655,6 +655,29 @@ def test_cov(self):
655655
)
656656
self._test_cov(pdf)
657657

658+
# Extension Dtypes
659+
pdf = pd.DataFrame(
660+
{
661+
"s1": pd.Series([0.90010907, None, 0.13484424, 0.62036035], dtype="Float64"),
662+
"s2": pd.Series([0.12528585, 0.81131178, 0.26962463, 0.51111198], dtype="Float64"),
663+
},
664+
index=[0, 1, 2, 3],
665+
)
666+
self._test_cov(pdf)
667+
668+
pdf = pd.DataFrame(
669+
{
670+
"s1": pd.Series(["a", "b", "c"], dtype="string"),
671+
"s2": pd.Series([0.12528585, 0.26962463, 0.51111198], dtype="Float64"),
672+
},
673+
index=[0, 1, 2],
674+
)
675+
psdf = ps.from_pandas(pdf)
676+
with self.assertRaisesRegex(TypeError, "unsupported dtype: (object|str)"):
677+
psdf["s1"].cov(psdf["s2"])
678+
with self.assertRaisesRegex(TypeError, "unsupported dtype: (object|str)"):
679+
psdf["s2"].cov(psdf["s1"])
680+
658681
def _test_cov(self, pdf):
659682
psdf = ps.from_pandas(pdf)
660683

0 commit comments

Comments
 (0)