diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..14d846d --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,30 @@ +# Contributing to Percentify + +Thank you for your interest in contributing to Percentify! We welcome contributions, but ask that you follow the project's guiding principles to keep the library focused and simple. + +## Guiding Principles + +Percentify is built on the idea of simplicity and directness. + +> Keep each method as direct-to-output as possible. A percentify function should return the single most common answer in one line, and point users to the underlying library (pandas, scipy, statsmodels, scikit-learn) for the full, configurable version when the simplest output isn't what they're after. + +- **No unnecessary knobs and options**: Anything that adds parameters for their own sake, or duplicates what the parent libraries already do well, is out of scope. For those cases, users should be pointed to the source library instead. +- **Keep it compact**: We try to keep the API surface compact on purpose. Please discuss big new features in an issue before writing code. + +## Technical Requirements + +**It must support Polars.** +Every function in Percentify accepts both pandas and Polars objects (via the `@_backend_aware` decorator) and returns the same kind. Any new contribution must maintain this parity. You should not require the user to manually convert between backends or pass specific flags. + +## Contribution Workflow + +If your idea keeps things simple, direct, and adheres to the principles above, here is how you can contribute: + +1. **Discuss first**: Open an issue to discuss your proposed change or feature. This saves time and ensures your contribution aligns with the project's goals. +2. **Fork and Branch**: Fork the repository and create a new branch for your feature or bugfix. +3. **Develop**: Write your code, ensuring it supports both pandas and Polars. +4. **Test**: (Include any testing guidelines if applicable, e.g., using `pytest`). +5. **Commit and Push**: Commit your changes with clear, descriptive commit messages. +6. **Pull Request**: Open a pull request against the main branch. Reference the issue you opened in step 1. + +We look forward to reviewing your contributions! diff --git a/README.md b/README.md index 6c78146..ee45714 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,5 @@ -

- Percentify logo + Percentify logo

[![PyPI version](https://img.shields.io/pypi/v/percentify.svg?style=flat&color=blue)](https://pypi.org/project/percentify/) @@ -19,7 +18,7 @@ Exploratory stats and data-quality diagnostics for pandas and **Polars** DataFra Where a function wraps an existing library (pandas, scipy, statsmodels, scikit-learn), it names it, so you always know where to dig deeper. -## ⭐ The flagship: `profiler` +## ⭐ `profiler` **pandas `.describe()` tells you what your data _is_. `profiler()` tells you what to _do_ about it:** every issue ranked worst-first, each with its fix. @@ -34,13 +33,13 @@ report.health # a 0 to 100 data-health score assert not report.errors # drop it straight into a CI data-quality gate ``` -Point it at any messy DataFrame, pandas or Polars, and see what it flags before you model. [Try it on your own data β†’](https://data-centt.github.io/percentify/documentation/#profiler) +Point it at any messy DataFrame, pandas or Polars, and see what it flags before you model. [Try it β†’](https://data-centt.github.io/percentify/documentation/#profiler) -## πŸ“– Documentation +# πŸ“– Documentation -**Full guide, every function, and live examples β†’ [data-centt.github.io/percentify](https://data-centt.github.io/percentify/)** +**Full guides β†’ [data-centt.github.io/percentify](https://data-centt.github.io/percentify/)** -## πŸ“¦ Installation +# πŸ“¦ Installation ```bash pip install percentify @@ -48,6 +47,17 @@ pip install percentify Requires Python 3.10+, `numpy`, and `pandas` 2.0+. +## How to use percentify + +Import the function that matches the question you want to answer, pass in a pandas or Polars object, and use the returned DataFrame or scalar directly in your notebook, report, or pipeline. + +```python +from percentify import missing, profiler + +missing(df) # quick column-level check +profiler(df, target="churn") # ranked data-quality issues and fixes +``` + ## Quick example ```python @@ -69,7 +79,130 @@ missing(df) One import, one line. A clean, sorted DataFrame you can read or feed into the next step. -## What's inside +# 🀝 Contributing + +Contributions are welcome, provided they align with the repository’s guiding principles. Please review the [contributing](https://github.com/data-centt/percentify/blob/main/CONTRIBUTING.md) guidelines before submitting. + + +# More Examples + +These are short, recipe-style examples that go beyond the one-liner above and are intentionally not covered in the [documentation](https://data-centt.github.io/percentify/documentation/). The docs show each function in isolation; these show how to chain them into a real workflow. + +### A 30-second data-quality gate + +Drop this into CI to block training on a dataset that isn't ready: + +```python +import pandas as pd +from percentify import profiler + +df = pd.read_parquet("train.parquet") +report = profiler(df, target="label") + +assert report.errors.empty, report.to_frame() +assert report.health >= 80, f"health too low: {report.health}" +``` + +#### Rank correlations by significance + +Pull the pairs that are both strong *and* unlikely to be noise: + +```python +import pandas as pd +from percentify import correlate + +df = pd.DataFrame({ + "x": range(50), + "y": [v * 0.9 + (v % 3) for v in range(50)], + "z": [v * 0.05 for v in range(50)], + "w": [v % 7 for v in range(50)], +}) + +print(correlate(df).sort_values("p_value").head(5)) +``` + +#### Build a transform pipeline from `skew_report` + +Let `skew_report` tell you what to apply, then apply it: + +```python +import numpy as np +import pandas as pd +from percentify import skew_report + +df = pd.DataFrame({ + "income": [30_000, 35_000, 1_200_000, 40_000, 28_000], + "visits": [1, 1, 1, 50, 2], +}) + +plan = skew_report(df) +print(plan[["feature", "skew", "suggested_transform"]]) +# feature skew suggested_transform +# 0 income 2.27 log1p +# 1 visits 2.19 log1p + +df["income_log"] = np.log1p(df["income"]) # numpy / pandas, not percentify +df["visits_log"] = np.log1p(df["visits"]) +``` + +#### Interpret PCA with both calls + +Variance tells you *how much* of the signal each axis carries; loadings tell you *what it means*: + +```python +import pandas as pd +from percentify import pca_variance, pca_loadings + +df = pd.DataFrame({ + "height_cm": [160, 170, 180, 175, 165], + "weight_kg": [55, 68, 82, 74, 60], + "age": [25, 35, 45, 30, 28], +}) + +print(pca_variance(df)) # PC1 carries most of the variance +print(pca_loadings(df)) # PC1 = (height, weight) with similar signs +``` + +#### Drop collinear columns before modelling + +Use `vif` with a threshold to get a drop-list you can feed straight into `df.drop`: + +```python +import pandas as pd +from percentify import vif + +df = pd.DataFrame({ + "price": [10, 12, 11, 13, 9, 14, 8, 12], + "cost": [ 6, 7, 7, 8, 5, 8, 4, 7], # tracks price + "margin": [ 4, 5, 4, 5, 4, 6, 4, 5], # = price - cost + "stock": [100, 80, 90, 70, 110, 60, 120, 85], +}) + +to_drop = vif(df, flag=5.0)["feature"].tolist() +print(to_drop) # e.g. ['cost', 'margin'] +clean = df.drop(columns=to_drop) +``` + +#### Month-over-month KPI table + +`change` over a DataFrame applies period-over-period growth to every numeric column at once: + +```python +import pandas as pd +from percentify import change + +kpis = pd.DataFrame({ + "revenue": [100, 120, 150, 135, 180], + "signups": [400, 420, 470, 460, 510], +}, index=["Jan", "Feb", "Mar", "Apr", "May"]) + +print(change(kpis)) +``` + +- [Worked examples for every function](https://data-centt.github.io/percentify/documentation/) +- [Project documentation](https://data-centt.github.io/percentify/) + +# What's inside | Function | What it answers | |---|---| @@ -92,28 +225,3 @@ One import, one line. A clean, sorted DataFrame you can read or feed into the ne | `display` | Format numbers or a column as clean "%" strings | β†’ See the **[documentation](https://data-centt.github.io/percentify/)** for a worked, real-output example of every function. - -## πŸ›Ÿ Friendly by design - -- **No cryptic tracebacks**; Hand a function a text column where numbers are needed and you get a clear PercentifyWarning, not an Arrow/NumPy stack trace. -- **Sensible defaults**; Results come back sorted worst-first, and PCA is standardized out of the box. -- **DataFrames everywhere**; so the output drops straight into your notebook, your next filter, or your model. -- **Pandas or polars**; pass either a pandas or polars object and you get the same kind back, no flag needed. - - -## 🀝 Contributing - -Contributions are welcome but they must follow the repo's guiding principle: -> Keep each method as direct-to-output as possible. A percentify function should return the single most common answer in one line, and point users to the underlying library (pandas, scipy, statsmodels, scikit-learn) for the full, configurable version when the simplest output isn't what they're after. - -**It must support polars.** Every function accepts both pandas and polars objects (via the `@_backend_aware` decorator) and returns the same kind, so any new contribution must keep that parity. - -If your idea keeps things that simple and direct: -- Open an issue first to discuss it -- Fork the repo -- Create a branch -- Commit your changes -- Open a pull request - -> Anything that adds knobs and options for their own sake, or duplicates what the parent libraries already do well, is out of scope; those cases should point to the source library instead. -I try to keep it compact on purpose, to discuss big new features first. diff --git a/docs/documentation.md b/docs/documentation.md index 6c9cb2c..67d19f4 100644 --- a/docs/documentation.md +++ b/docs/documentation.md @@ -528,7 +528,7 @@ df = pd.DataFrame({ "score": np.random.randn(200), }) -correlate(df["age"], df["income"]) # (0.99, 0.0) +correlate(df["age"], df["income"]) # (0.99, 1.7e-166) ``` **A DataFrame returns a ranked table** @@ -538,14 +538,26 @@ correlate(df) ``` ```text -feature_1 feature_2 r p - age income 0.99 0.00 - age score 0.05 0.49 - income score 0.05 0.49 +feature_1 feature_2 r p + age income 0.99 1.700000e-166 + age score 0.05 4.900000e-01 + income score 0.05 4.900000e-01 ``` Pass `method="spearman"` for rank (monotonic) correlation. +!!! note "Reading the p-value" + A p-value is never exactly zero, so `correlate` never reports one. When `p` + is smaller than the `decimals` resolution it keeps two significant figures + (`1.7e-166`) instead of collapsing to a misleading `0.00`. One very small + value puts the whole column into scientific notation, so `4.900000e-01` is + just `0.49`. + + Read `r` for the strength and `p` only for "is this distinguishable from + zero". On a large sample almost any correlation is significant, so a tiny + `p` is not evidence of a strong relationship: `r = 0.05` with + `p = 1e-20` is still a negligible relationship. + --- ## `skew_report` @@ -643,6 +655,12 @@ permutation_test(a, b, random_state=0) # 0.001 The default statistic is the difference in means; pass your own `statistic(a, b)` for anything else. It returns the number, not a pass or fail verdict, so the judgement stays with you. +!!! note "The smallest p this test can report" + A permutation test can only resolve down to `1 / (n_permutations + 1)`, so + with the default 1000 shuffles the floor is `0.001`. That is a limit of the + shuffling, not proof the effect is that rare. Raise `n_permutations` to + resolve further. The result is never rounded down to `0.0`. + --- ## `effect_size` diff --git a/percentify/stats.py b/percentify/stats.py index 158f951..004e1b1 100644 --- a/percentify/stats.py +++ b/percentify/stats.py @@ -566,10 +566,16 @@ def correlate(a, b=None, method: str = "pearson", decimals: Optional[int] = 2): a: A Series (with b) or a DataFrame (matrix mode). b: The second Series for a pairwise correlation. method: "pearson" (linear) or "spearman" (rank / monotonic). - decimals: Number of decimal places to round to. + decimals: Number of decimal places to round to. A p-value smaller than + this resolution keeps two significant figures instead of rounding + to 0.0, so a tiny p reads as 1.7e-31 rather than a false zero. Returns: A (r, p) tuple for two Series, or a DataFrame for a DataFrame. + + Note: + On a large sample almost any r is "significant", so p mostly tells you + the correlation is not exactly zero. Read r for the strength. """ from scipy import stats @@ -586,7 +592,7 @@ def correlate(a, b=None, method: str = "pearson", decimals: Optional[int] = 2): _warn("correlate needs at least 3 complete numeric pairs. Returning NaN.") return (float("nan"), float("nan")) r, p = corr_fn(pair["a"].to_numpy(), pair["b"].to_numpy()) - return (_round(float(r), decimals), _round(float(p), decimals)) + return (_round(float(r), decimals), _round_p(float(p), decimals)) if not isinstance(a, pd.DataFrame): raise TypeError(f"correlate expects a Series or DataFrame, got {type(a).__name__}.") @@ -606,7 +612,7 @@ def correlate(a, b=None, method: str = "pearson", decimals: Optional[int] = 2): if len(pair) < 3 or np.std(x) == 0 or np.std(y) == 0: continue r, p = corr_fn(x, y) - rows.append((c1, c2, _round(float(r), decimals), _round(float(p), decimals))) + rows.append((c1, c2, _round(float(r), decimals), _round_p(float(p), decimals))) result = pd.DataFrame(rows, columns=["feature_1", "feature_2", "r", "p"]) if result.empty: @@ -710,7 +716,10 @@ def permutation_test(a, b, statistic=None, n_permutations: int = 1000, statistic: Function of (group_a, group_b) measuring the effect (default: difference in means). n_permutations: Number of label shuffles. - decimals: Number of decimal places to round to. + decimals: Number of decimal places to round to. A p-value smaller than + this resolution keeps two significant figures instead of rounding + to 0.0. The smallest p this test can report is + 1 / (n_permutations + 1); raise n_permutations to resolve further. random_state: Seed for reproducibility. Returns: @@ -735,7 +744,7 @@ def permutation_test(a, b, statistic=None, n_permutations: int = 1000, rng.shuffle(combined) if abs(statistic(combined[:n_a], combined[n_a:])) >= observed: count += 1 - return _round(float((count + 1) / (n_permutations + 1)), decimals) + return _round_p(float((count + 1) / (n_permutations + 1)), decimals) @_backend_aware @@ -892,6 +901,9 @@ def split(total, weights, decimals: Optional[int] = 2): shares = w.astype(float) / weight_sum * float(total) if decimals is not None: shares = shares.round(decimals) + remainder = round(float(total), decimals) - float(shares.sum()) + if remainder: + shares.iloc[-1] = round(shares.iloc[-1] + remainder, decimals) return shares if is_series else shares.tolist() diff --git a/tests/test_inferential.py b/tests/test_inferential.py index e689255..48ebcbf 100644 --- a/tests/test_inferential.py +++ b/tests/test_inferential.py @@ -29,6 +29,45 @@ def test_correlate_dataframe_tidy_and_sorted(): assert result["r"].abs().tolist() == sorted(result["r"].abs().tolist(), reverse=True) +def test_correlate_small_p_is_not_rounded_to_zero(): + # A p-value below the decimals resolution must keep significant figures + # instead of collapsing to a misleading 0.0. + np.random.seed(0) + n = 200 + a = np.random.randn(n) + b = 0.44 * a + np.sqrt(1 - 0.44 ** 2) * np.random.randn(n) + r, p = correlate(pd.Series(a), pd.Series(b)) + assert 0 < p < 0.01 # genuinely tiny, and never exactly zero + assert r == round(r, 2) # r still honours decimals + + +def test_correlate_underflowing_p_is_never_exactly_zero(): + # With a large sample the true p is ~1e-2000, so scipy underflows to a + # literal 0.0. We must report a vanishingly small number, not "impossible". + np.random.seed(0) + n = 40000 + a = np.random.randn(n) + b = 0.44 * a + np.sqrt(1 - 0.44 ** 2) * np.random.randn(n) + _, p = correlate(pd.Series(a), pd.Series(b)) + assert p > 0.0 + assert p < 1e-300 + + +def test_correlate_dataframe_p_column_never_zero(): + np.random.seed(0) + base = np.random.randn(5000) + df = pd.DataFrame({"a": base, "b": base * 2 + np.random.randn(5000) * 0.01}) + result = correlate(df) + assert (result["p"] > 0).all() + + +def test_correlate_ordinary_p_still_rounds_normally(): + # Values above the resolution keep the familiar 2-decimal formatting. + np.random.seed(3) + r, p = correlate(pd.Series(np.random.randn(50)), pd.Series(np.random.randn(50))) + assert p == round(p, 2) + + def test_correlate_spearman(): r, p = correlate(pd.Series([1, 2, 3, 4, 5]), pd.Series([1, 4, 9, 16, 25]), method="spearman") assert r == 1.0 # perfectly monotonic @@ -46,6 +85,18 @@ def test_correlate_single_numeric_column_warns(): assert result.empty +def test_correlate_small_p_value_not_rounded_to_zero(): + np.random.seed(0) + x = pd.Series(np.random.randn(200)) + y = pd.Series(x * 0.6 + np.random.randn(200) * 0.5) + + r, p = correlate(x, y) + assert p > 0 + + result = correlate(pd.DataFrame({"a": x, "b": y})) + assert (result["p"] > 0).all() + + # ===== skew_report ===== def test_skew_report_columns(): @@ -125,6 +176,17 @@ def test_permutation_test_flags_real_difference(): assert permutation_test(a, b, random_state=0) < 0.05 +def test_permutation_test_small_p_survives_rounding(): + # The floor is 1/(n_permutations+1); with 100k shuffles that is ~1e-5, + # which plain 4-decimal rounding would have flattened to 0.0. + np.random.seed(0) + a = np.random.randn(200) + b = np.random.randn(200) + 3 + p = permutation_test(a, b, n_permutations=100000, random_state=0) + assert p > 0.0 + assert p < 0.001 + + def test_permutation_test_reproducible(): a, b = [1, 2, 3, 4, 5], [2, 3, 4, 5, 6] assert permutation_test(a, b, random_state=1) == permutation_test(a, b, random_state=1) diff --git a/tests/test_stats.py b/tests/test_stats.py index 7ac2e01..2b81393 100644 --- a/tests/test_stats.py +++ b/tests/test_stats.py @@ -609,7 +609,9 @@ def test_split_list(): def test_split_equal(): - assert split(100, [1, 1, 1]) == [pytest.approx(33.33)] * 3 + result = split(100, [1, 1, 1]) + assert result == [33.33, 33.33, 33.34] + assert sum(result) == 100 def test_split_series_returns_series():