Skip to content

Commit 5967cc2

Browse files
vyasrvyasr
authored
Add debug-cudf-pandas skill for fixing pandas test compatibility failures (rapidsai#22625)
Adds a skill for debugging cudf.pandas test failures. This skill aims to encode a standardized workflow for failures in a systematic way. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Matthew Roeschke (https://github.com/mroeschke) URL: rapidsai#22625
1 parent ad987ea commit 5967cc2

1 file changed

Lines changed: 321 additions & 0 deletions

File tree

  • .agents/skills/debug-cudf-pandas

‎.agents/skills/debug-cudf-pandas/SKILL.md‎

Lines changed: 321 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,321 @@
1+
---
2+
name: debug-cudf-pandas
3+
description: Debug and fix pandas test suite failures under the cudf.pandas compatibility layer. Use when given pytest node IDs of failing pandas tests that need to be fixed for cudf.pandas compatibility.
4+
---
5+
6+
# Debug cudf.pandas Compatibility Failures
7+
8+
## Overview
9+
10+
`cudf.pandas` is a zero-code-change GPU accelerator for pandas. It intercepts `import pandas` via a Python `MetaPathFinder` (`python/cudf/cudf/pandas/module_accelerator.py`) and wraps all pandas objects and functions in proxy types that try the cudf (GPU) path first, falling back to pandas (CPU) if cudf raises an exception.
11+
12+
When the pandas test suite is run with `-p cudf.pandas`, test failures indicate one of:
13+
- A **cudf implementation bug** — cudf's DataFrame/Series/Column/Index behavior differs from pandas for an edge case
14+
- A **proxy/dispatch bug** — the wrapping/unwrapping mechanism doesn't correctly handle a type or operation
15+
- A **missing proxy registration** — a pandas type or return value has no registered cudf equivalent
16+
- A **to/from_pandas conversion bug** — data is corrupted or lost when converting between cudf and pandas objects
17+
- A **test setup bug** — the testing scripts or conftest-patch introduce an issue
18+
- A **pandas bug** — rarely, the expected behavior in the pandas test itself is wrong
19+
20+
Your job is to find the root cause and implement the fix.
21+
22+
## Unacceptable Fix Patterns
23+
24+
The following patterns are prohibited regardless of whether they make a test pass. Stop immediately and ask the user if the only apparent solution falls into one of these categories.
25+
26+
- **Esoteric/test-specific special cases**: Do not implement narrow, brittle logic targeting only the specific failing node IDs (e.g. special-casing a single dtype/shape combination). If a test is failing, find the general API contract mismatch and fix it at that level.
27+
- **Pandas CPU fallback as a fix**: Typically you should not make tests pass by simply raising an exception inside cudf to force CPU fallback. Falling back to pandas/CPU is only an acceptable fix if attempting to fix the bug leads to discovering that large features must be implemented in cudf to support the change (e.g. a completely new class).
28+
- **Private pandas APIs**: Do not import or call any symbol from `pandas.core`, `pandas.compat`, or any underscored pandas module (e.g. `pandas._libs.tslibs.parsing`). These are explicitly unstable per the pandas API policy. Use public pandas APIs or write equivalent local logic instead.
29+
- **PyArrow as a CPU execution backend**: Do not route GPU operations through `pyarrow.compute` on CPU as a substitute for cudf/libcudf semantics. Arrow is an interchange format; it is not an acceptable execution backend for cudf operations.
30+
- **Returning pandas objects from cudf APIs**: cudf public methods (`Series`, `Index`, `DataFrame` operations and accessors) must return cudf-native objects, not `pd.Series`, `pd.Index`, or `pd.DataFrame`. Use `_return_or_inplace` and the existing cudf container reconstruction helpers.
31+
32+
---
33+
34+
## Prerequisites
35+
36+
Before starting, verify you are at the repository root. All commands in this skill assume the working directory is the cudf repository root.
37+
38+
**Clean up any previous test run state.** The test runner appends `conftest-patch.py` to the pandas conftest on every invocation. If `pandas-testing/pandas-tests/` already exists from a prior run, the conftest will have duplicate hook registrations and cause spurious errors. Always delete it before running:
39+
40+
```bash
41+
rm -rf pandas-testing/pandas-tests/
42+
```
43+
44+
The cudf Python package is almost entirely pure Python. For **inplace installs** (e.g. `pip install -e .`), changes to `.py` files take effect immediately — no rebuild is needed. For non-inplace installs (e.g. `./build.sh`), you must either reinstall or copy changed files to site-packages.
45+
46+
---
47+
48+
## Input Format
49+
50+
Input is one or more pytest node IDs, relative to the `pandas-tests/` directory inside the test harness:
51+
52+
```
53+
tests/groupby/test_reductions.py::test_first_last_skipna[Float64-False-False-first]
54+
tests/frame/methods/test_fillna.py::test_fillna_inplace
55+
```
56+
57+
Node IDs with parameters like `[Float64-False-False-first]` target a specific parametrized case. When multiple node IDs are provided, check whether they share a root cause before attempting separate fixes.
58+
59+
---
60+
61+
## Step 0 — Update conftest-patch.py
62+
63+
The file `python/cudf/cudf/pandas/scripts/conftest-patch.py` contains three dictionaries that gate how tests are handled:
64+
65+
- **`NODEIDS_THAT_FAIL`** — tests marked `xfail` (expected to fail). Keys are alphabetically sorted.
66+
- **`NODEIDS_TO_SKIP`** — tests marked `skip` (not run at all). Keys are alphabetically sorted.
67+
- **`NODEIDS_PATHS_TO_SKIP`** — prefix-based path skips covering entire modules.
68+
69+
Because the test runner sets `xfail_strict = true`, a test listed in `NODEIDS_THAT_FAIL` that unexpectedly *passes* is reported as `XPASS` — which is also a failure. You must remove the entry before testing your fix, or you will never see a genuine pass.
70+
71+
Search for the node ID:
72+
73+
```bash
74+
grep -n "tests/groupby/test_reductions.py::test_first_last_skipna" \
75+
python/cudf/cudf/pandas/scripts/conftest-patch.py
76+
```
77+
78+
If found, remove the line. Keys must remain in alphabetical order after the edit. Then validate the file still parses:
79+
80+
```bash
81+
python -c "exec(open('python/cudf/cudf/pandas/scripts/conftest-patch.py').read())"
82+
```
83+
84+
If the node ID is *not* found in any dictionary, you are dealing with a new regression — proceed directly to Step 1.
85+
86+
---
87+
88+
## Step 1 — Reproduce the Failure
89+
90+
Run from the repo root:
91+
92+
```bash
93+
rm -rf pandas-testing/pandas-tests/
94+
bash python/cudf/cudf/pandas/scripts/run-pandas-tests.sh \
95+
"tests/groupby/test_reductions.py::test_first_last_skipna[Float64-False-False-first]" \
96+
-xvs
97+
```
98+
99+
The script clones the matching pandas version, copies tests, appends the conftest patch, and runs pytest with `-p cudf.pandas`. Substitute your actual node ID.
100+
101+
**If the test passes**: the xfail entry was stale. Commit only the `conftest-patch.py` change and stop.
102+
103+
**If the test fails**: read the failure output carefully — the assertion message tells you the exact behavioral difference. Proceed to Step 2.
104+
105+
---
106+
107+
## Step 2 — Understand the Test
108+
109+
After the first run, the test file exists at:
110+
111+
```
112+
pandas-testing/pandas-tests/tests/<module>/<test_file>.py
113+
```
114+
115+
Read the test. Identify:
116+
- Which pandas API is being exercised (method name, class, module)
117+
- What the test asserts — this is the exact behavior cudf must match
118+
- Whether parameters narrow the scope (e.g. only `Float64` dtype, only `skipna=False`)
119+
120+
The assertion error output from Step 1 tells you what cudf produced vs. what was expected. Use both together to understand the gap.
121+
122+
---
123+
124+
## Step 3 — Diagnose Root Cause
125+
126+
**Always check cudf's direct behavior first.** Do not jump to proxy investigation.
127+
128+
### 3a. Direct cudf test (most important step)
129+
130+
Write a minimal script that exercises the API using cudf directly, without going through `cudf.pandas`. This is strictly an example — adapt the structure to match what the failing test actually does (comparisons, exception checks, dtype validation, etc.):
131+
132+
```python
133+
import cudf
134+
import pandas as pd
135+
136+
# Mirror the test setup
137+
pdf = pd.DataFrame({"value": [1.0, None, 2.0, 3.0]}, dtype="Float64")
138+
cdf = cudf.from_pandas(pdf)
139+
140+
pd_result = pdf.groupby(level=0).first(skipna=False)
141+
cudf_result = cdf.groupby(level=0).first(skipna=False)
142+
143+
print("pandas:", pd_result)
144+
print("cudf: ", cudf_result)
145+
print("match:", pd_result.equals(cudf_result.to_pandas()))
146+
```
147+
148+
For tests that assert exceptions are raised, structure your script to verify both cudf and pandas raise the same exception type and message.
149+
150+
Save to a temporary file (e.g. `test_debug.py`) and run:
151+
152+
```bash
153+
python test_debug.py
154+
```
155+
156+
Results:
157+
- **cudf result differs from pandas** → cudf implementation bug → go to Step 4a
158+
- **cudf raises an exception** → missing feature or bug → evaluate scope; may need user input if the feature is large. Note: this may be OK if the test is verifying that an exception *should* be raised.
159+
- **cudf result matches pandas** → proxy/dispatch bug → go to Step 4b
160+
161+
**Classify the root cause before writing any fix.** Ask yourself: Is this a specific method/keyword handling bug? A broad dtype casting mismatch affecting many operations? A proxy/wrapping issue? A missing cudf capability? For broad issues, the fix should be applied at the shared/base layer, not patched per individual method. If the only apparent fix is test-shaped (i.e. it looks like it exists to make exactly these node IDs pass), step back and re-examine the general API contract.
162+
163+
### 3b. Environment variable diagnostics (run through cudf.pandas)
164+
165+
Use these env vars to trace what is happening at the proxy layer:
166+
167+
| Variable | Effect |
168+
|----------|--------|
169+
| `CUDF_PANDAS_FAIL_ON_FALLBACK=1` | Raises instead of silently falling back — shows exactly which operation triggers fallback |
170+
| `LOG_FAST_FALLBACK=1` | Logs every fallback with function name and exception |
171+
| `CUDF_PANDAS_DEBUGGING=1` | Runs both cudf and pandas paths in parallel, warns on result divergence |
172+
173+
```bash
174+
CUDF_PANDAS_FAIL_ON_FALLBACK=1 bash python/cudf/cudf/pandas/scripts/run-pandas-tests.sh \
175+
"<node_id>" -xvs
176+
```
177+
178+
```bash
179+
LOG_FAST_FALLBACK=1 bash python/cudf/cudf/pandas/scripts/run-pandas-tests.sh \
180+
"<node_id>" -xvs 2>&1 | grep -i fallback
181+
```
182+
183+
**Fallback is a diagnostic signal, not a fix.** If `CUDF_PANDAS_FAIL_ON_FALLBACK=1` causes the test to fail, routing the operation to pandas CPU is not acceptable as a final solution unless adding GPU support would require implementing large, entirely unsupported features. Bug fixes for cudf's behavior (e.g. to add support for a particular dtype to a function) are the expected path.
184+
185+
### 3c. Standalone instrumented script
186+
187+
For deeper investigation, copy the relevant test body into a temporary script (e.g. `test_debug.py`), add print statements or assertions at intermediate steps, then run through the proxy layer or try to write matching pandas code to see the differences in behaviors and fix them:
188+
189+
```bash
190+
python -m cudf.pandas test_debug.py
191+
```
192+
193+
This gives you full control to narrow down exactly where the divergence begins.
194+
195+
---
196+
197+
## Step 4a — Fix a cudf Implementation Bug
198+
199+
The fix must be **tightly scoped** and try to keep it minimal. Fixing one edge case can break another test. Do not refactor surrounding code while fixing, and do not add `mode.pandas_compatible` guards (ask the user first).
200+
201+
**Classify the failure mode** before writing a fix: is it a specific issue (e.g. one method handles a keyword incorrectly) or a broad failure mode (e.g. dtype casting inconsistency that affects many operations)? For broad issues, consider whether the fix should be applied at a shared/base level rather than patching individual methods.
202+
203+
Common fix locations:
204+
205+
| What fails | Where to look |
206+
|---|---|
207+
| DataFrame method | `python/cudf/cudf/core/dataframe.py` |
208+
| Series method | `python/cudf/cudf/core/series.py` |
209+
| Index operation | `python/cudf/cudf/core/index.py` |
210+
| Shared DataFrame/Series method | `python/cudf/cudf/core/indexed_frame.py` |
211+
| Column-level operation | `python/cudf/cudf/core/column/*.py` |
212+
| GroupBy operation | `python/cudf/cudf/core/groupby/` |
213+
| IO operation | `python/cudf/cudf/io/` |
214+
215+
Note: `mode.pandas_compatible` is automatically set to `True` when cudf.pandas is active. Account for this in any conditional logic, but do not add new guards for it without explicit user approval.
216+
217+
If the bug is not in cudf core, move to cudf.pandas-specific fixes.
218+
219+
**Prohibited in cudf implementation fixes:**
220+
- Do not call `pandas._libs.*` or any other private/underscored pandas module. Use public pandas APIs (`pd.Timestamp`, `pd.to_datetime`, `pd.DateOffset`, etc.) and write equivalent local logic if needed.
221+
- Do not convert to pyarrow and use `pyarrow.compute` as a substitute for libcudf behavior. If libcudf doesn't support the exact semantics, consult the user before adding a pyarrow fallback.
222+
- Do not construct and return `pd.Series(...)`, `pd.Index(...)`, or `pd.DataFrame(...)` from cudf public methods. Use `_return_or_inplace`, `_from_column`, or other cudf container reconstruction helpers so that return types remain cudf-native.
223+
224+
---
225+
226+
## Step 4b — Fix a Proxy/Dispatch Bug
227+
228+
Only reach this step after Step 3a has confirmed that cudf itself is correct. Some of the common cases you should consider at this stage:
229+
230+
**Most common cause**: a pandas object in a particular state is not round-tripping correctly through cudf's conversion APIs. `cudf.from_pandas()` and `<object>.to_pandas()` are culprits here. To debug this case, set up the standalone instrumented script as described in step 3c. Then instrument the code to perform direct modifications and testing of the proxy state, for instance by accessing the `_fsproxy_fast` and `_fsproxy_slow` attributes of the relevant objects and calling `cudf.from_pandas` and `<object>.to_pandas` to see if information is being lost or corrupted in one of the conversions.
231+
232+
**Next most common cause**: a pandas return type has no registered cudf proxy. Check `python/cudf/cudf/pandas/_wrappers/pandas.py` — this file registers which pandas types map to which cudf types using `make_final_proxy_type()` and related functions. If a new pandas type needs wrapping, add the registration here.
233+
234+
**`fast_slow_proxy.py` and `module_accelerator.py`** are core infrastructure files. Fix them only if you believe the bug is in one of them.
235+
236+
---
237+
238+
## Step 5 — Verify the Fix
239+
240+
Three checks are required. Run them in order.
241+
242+
**a. Target test passes:**
243+
244+
```bash
245+
bash python/cudf/cudf/pandas/scripts/run-pandas-tests.sh "<node_id>" -xvs
246+
```
247+
248+
Expected: exit code 0, test shows `PASSED`.
249+
250+
**b. Fix runs on GPU (no silent fallback):**
251+
252+
```bash
253+
CUDF_PANDAS_FAIL_ON_FALLBACK=1 bash python/cudf/cudf/pandas/scripts/run-pandas-tests.sh \
254+
"<node_id>" -xvs
255+
```
256+
257+
Expected: test still passes. If this fails, the fix works by falling back to pandas rather than actually fixing cudf — that is not acceptable.
258+
259+
Exception: some tests intentionally validate fallback behavior. If `FAIL_ON_FALLBACK` causes this test to fail but the test logic requires fallback, skip this check for that specific test and note the justification.
260+
261+
**c. No regressions in the module:**
262+
263+
```bash
264+
bash python/cudf/cudf/pandas/scripts/run-pandas-tests.sh \
265+
"tests/<module_directory>/" --tb=line -q
266+
```
267+
268+
Replace `<module_directory>` with the directory containing your test (e.g. `tests/groupby/`). Any new failures that are not already listed in `conftest-patch.py` must be investigated before committing.
269+
270+
**d. Add unit tests (where appropriate):**
271+
272+
If your fix changes cudf behavior, add a unit test in `cudf` classic or `cudf.pandas` depending on where the fix lives. When adding tests:
273+
- Prefer adding to existing test parameterizations if a related test already exists
274+
- If no related test exists, write a test that compares cudf output to pandas (using `assert_eq` or `assert_exceptions_equal`)
275+
- Place cudf classic tests alongside existing tests in `python/cudf/cudf/tests/`
276+
- Place cudf.pandas-specific tests in `python/cudf/cudf_pandas_tests/`
277+
278+
---
279+
280+
## Step 6 — Commit
281+
282+
Stage only the intended files — never anything from `pandas-testing/`:
283+
284+
```bash
285+
git add python/cudf/cudf/ # source fix (if applicable)
286+
git add python/cudf/cudf/pandas/scripts/conftest-patch.py # xfail removal
287+
git status # verify nothing from pandas-testing/ is staged
288+
git commit -m "fix(cudf.pandas): <short description>
289+
290+
Fixes the following failing pandas tests:
291+
- tests/groupby/test_reductions.py::test_first_last_skipna[Float64-False-False-first]"
292+
```
293+
294+
If many node IDs share the same root cause, summarize them in the commit message rather than listing every parametrized variant. Commit message should convey what was wrong and what was fixed.
295+
296+
---
297+
298+
## STOP Conditions
299+
300+
Stop immediately and report findings to the user when any of the following apply:
301+
302+
- The fix requires modifying `.pyx`, `.cu`, `.cuh`, or `CMakeLists.txt` files — this requires compilation, which is outside the scope of this skill
303+
- Investigation reveals the behavioral divergence is **intentional** — cudf was deliberately designed to differ from pandas
304+
- You have been through 3 or more investigation cycles without converging on a fix
305+
- The fix would add a new `mode.pandas_compatible` conditional
306+
307+
For intentional divergence: stop and ask the user. In most cases, the goal is to make cudf match pandas. Only when the divergence has significant performance implications should `mode.pandas_compatible` be used — and even then, sparingly. cudf already overuses this flag; the default behavior should agree with pandas.
308+
309+
---
310+
311+
## Important Notes
312+
313+
- `mode.pandas_compatible` is automatically set to `True` when `cudf.pandas` is active. This is done at the end of `python/cudf/cudf/pandas/_wrappers/pandas.py`.
314+
- cudf Python is almost entirely pure Python — for inplace installs, changes to `.py` files take effect immediately without rebuilding.
315+
- `pandas-testing/pandas-tests/` must be deleted before each test run to avoid duplicate conftest hook registrations (the script appends the patch file on every run).
316+
- Keys in all three `conftest-patch.py` dictionaries must remain in alphabetical order.
317+
- Never write comments that explain what old code was replaced — write comments about what the code does and why.
318+
- Never modify the pandas test files themselves — fix cudf, not pandas.
319+
- Never fix the testing APIs (like `assert_frame_equal`, `assert_series_equal`) — fix the actual APIs that produce wrong results.
320+
- First see if the problem is in cudf classic and fix it there; if not, then move over to cudf.pandas.
321+
- Tests run with `xfail_strict = true` — a test listed in `NODEIDS_THAT_FAIL` that unexpectedly passes is reported as `XPASS` (also a failure). Remove from the list before testing.

0 commit comments

Comments
 (0)