Skip to content

Commit 15d47b7

Browse files
raffaelemancusoraffaelemancuso
committed
Cleanup
1 parent cbc5eaa commit 15d47b7

39 files changed

Lines changed: 2445 additions & 547 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
__pycache__
2+
old/
23
/docs/build
34
/docs/source/*
45
!/docs/source/*.rst

‎README.md‎

Lines changed: 160 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,160 @@
1-
# raffalib-python
2-
3-
## Introduction
4-
5-
A library with helper functions for pandas, polars, selenium, and others
6-
7-
See the [docs](https://raffalib-python.readthedocs.io).
1+
# raffalib-python
2+
3+
A Python library of helper functions for data wrangling. Its main purpose is to
4+
enrich [pandas](https://pandas.pydata.org/) and [polars](https://pola.rs/) with
5+
[STATA](https://www.stata.com/)-like logging and `.docx` export capabilities, but
6+
it also bundles assorted utilities for logging, pickling, progress bars, Selenium,
7+
SQLAlchemy, and bibliometrics.
8+
9+
📖 **Documentation:** <https://raffalib-python.readthedocs.io> —
10+
[Quickstart](https://raffalib-python.readthedocs.io/en/latest/quickstart.html) ·
11+
[Examples](https://raffalib-python.readthedocs.io/en/latest/examples.html) ·
12+
API Reference
13+
14+
> ⚠️ This project is under active development.
15+
16+
## Highlights
17+
18+
- **STATA-like logging** — wrap a pandas/polars pipeline in `startlog()` / `endlog()`
19+
and get a log line describing how many rows, columns, or cell values changed (and
20+
optionally how long it took).
21+
- **`.docx` export** — write any DataFrame straight to a Word table with `to_docx()`.
22+
- **Frequency & cross tables** — `freq()` and `crosstab()` accessors for quick
23+
tabulations.
24+
- **Smarter joins** — a `join()` wrapper (pandas *and* polars) that logs where the
25+
output rows came from (left only / right only / both, dropped rows, duplicate keys),
26+
and auto-detects filtering (`semi`/`anti`) joins.
27+
28+
## Installation
29+
30+
Requires **Python ≥ 3.13**. The package is distributed as a local/editable install.
31+
32+
```console
33+
# with uv
34+
uv add --editable /path/to/raffalib-python
35+
36+
# or with pip
37+
python3 -m pip install --editable /path/to/raffalib-python
38+
```
39+
40+
### Optional dependencies
41+
42+
Core install is intentionally light. Heavier integrations live behind extras, so you
43+
only pull in what you use:
44+
45+
| Extra | Enables |
46+
| ---------------- | ---------------------------------------------------- |
47+
| `pandas` | `raffalib.pandas` accessors |
48+
| `polars` | `raffalib.polars` accessors and join logging |
49+
| `bibliometrics` | OpenAlex / Scopus helpers |
50+
| `crypto` | KeePassXC / GnuPG helpers |
51+
| `db` | SQLAlchemy view helpers |
52+
| `web` | Selenium helpers |
53+
| `docs` | Build the Sphinx documentation |
54+
| `dev` | Ruff + pytest for development |
55+
56+
```console
57+
# example: install with the pandas and polars extras
58+
uv add --editable "/path/to/raffalib-python[pandas,polars]"
59+
```
60+
61+
## Quick start
62+
63+
### Logging changes in a pandas pipeline
64+
65+
```python
66+
import pandas as pd
67+
import raffalib
68+
import raffalib.pandas # registers the `.raffa` accessor
69+
70+
logger = raffalib.create_logger(rich=False, fmt="{message}")
71+
72+
df = pd.read_csv("penguins.csv")
73+
74+
# Shape changes are logged automatically
75+
_ = df.raffa.startlog().dropna(subset=["bill_depth_mm"]).raffa.endlog(timeit=False)
76+
# -> Removed 2/344 (0.58%) rows.
77+
78+
# Pass clone=True to also detect value-level changes when the shape is unchanged
79+
_ = df.raffa.startlog(clone=True).fillna(0).raffa.endlog(timeit=False)
80+
# -> Changed 19/2,752 (0.69%) values.
81+
```
82+
83+
### The same with polars
84+
85+
```python
86+
import polars as pl
87+
import raffalib
88+
import raffalib.polars # registers the `.raffa` namespace
89+
90+
logger = raffalib.create_logger(rich=False, fmt="{message}")
91+
92+
df = pl.read_csv("penguins.csv")
93+
94+
_ = df.raffa.startlog().filter(pl.col("species") == "Adelie").raffa.endlog(timeit=False)
95+
# -> Removed 192/344 (55.81%) rows.
96+
```
97+
98+
Both backends share the same `startlog(clone=False)` / `endlog(custom_msg=None, timeit=True)`
99+
signature. Set `timeit=True` (the default) to append an elapsed-time line.
100+
101+
### Logging joins (pandas & polars)
102+
103+
Both accessors expose the same `join()` wrapper — over `pd.DataFrame.merge` for
104+
pandas and `pl.DataFrame.join` for polars — with identical logging output:
105+
106+
```python
107+
out = df1.raffa.join(df2, on="A", how="left")
108+
# Total rows in output table: 4
109+
# From left only: 1/4 (25.00%)
110+
# From right only: 0/4 (0.00%)
111+
# From both: 3/4 (75.00%) (left dups 0, right dups 0)
112+
# Dropped rows from left: 0/4 (0.00%)
113+
# Dropped rows from right: 2/5 (40.00%)
114+
```
115+
116+
Filtering joins (`how="semi"` / `how="anti"`) are detected automatically, and
117+
`keep_row_index=True` keeps the source row-index columns in the output.
118+
119+
### Exporting to Word
120+
121+
```python
122+
df.raffa.to_docx("table.docx")
123+
124+
# Heading and table options are routed to the right place automatically
125+
df.raffa.to_docx("table.docx", heading_text="Table 1", table_style="Light Grid")
126+
```
127+
128+
See the [Examples](https://raffalib-python.readthedocs.io) page for the full walkthrough.
129+
130+
## Modules
131+
132+
| Module | What it provides |
133+
| ----------------------- | ---------------------------------------------------------------------- |
134+
| `raffalib.pandas` | `.raffa` accessor: `startlog`/`endlog`/`midlog`, `join`, `freq`, `to_docx`, `add_prefix_if_not_exists`, `get_duplicates`, `sort_columns` |
135+
| `raffalib.polars` | `.raffa` namespace: logging, `freq`, `crosstab`, `join`, `replace_string_with_null`, `to_docx` |
136+
| `raffalib.logging` | `create_logger` — opinionated logging setup (plain or `rich`) |
137+
| `raffalib.export_docx` | `DocxFile` — low-level Word document/table builder |
138+
| `raffalib.tqdm` | `tqdm_batch` — batched progress bars |
139+
| `raffalib.itertools` | `batch_boundaries` — compute batch start/end indices |
140+
| `raffalib.list_replace` | `list_replace` — replace occurrences in a list |
141+
| `raffalib.mypickle` | `read_pickle` / `write_pickle` helpers |
142+
| `raffalib.selenium` | Scrolling and explicit-wait helpers for Selenium WebDriver |
143+
| `raffalib.sqlalchemy` | SQLAlchemy `CREATE VIEW` / `DROP VIEW` constructs and a `view()` helper |
144+
| `raffalib.ScopusUtils` | Scopus API helpers |
145+
| `raffalib.check_openalex_api_key` | Validate an OpenAlex API key |
146+
147+
## Development
148+
149+
```console
150+
uv sync --extra dev
151+
uv run pytest # run the test suite
152+
uv run ruff check # lint
153+
uv run ruff format # format
154+
```
155+
156+
## License
157+
158+
Released under the [GNU General Public License v3.0 or later](LICENSE.txt).
159+
160+
Copyright © 2026 Raffaele Mancuso.

‎docs/source/conf.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131

3232
autoapi_dirs = ['../../src']
3333

34+
# Document the class docstring and the __init__ docstring together.
35+
autoapi_python_class_content = "both"
36+
37+
# Keep the default options but drop "imported-members": the package re-exports
38+
# (e.g. ``raffalib.list_replace``) would otherwise be documented both on the
39+
# package page and on their own module page, producing duplicate-object warnings.
40+
autoapi_options = [
41+
"members",
42+
"undoc-members",
43+
"show-inheritance",
44+
"show-module-summary",
45+
"special-members",
46+
]
47+
3448
exclude_patterns = ['_build', '_templates']
3549

3650
# Add any paths that contain templates here, relative to this directory.

0 commit comments

Comments
 (0)