Skip to content

Commit c29c1aa

Browse files
JosiahParryJosiahParry
authored
init (#83)
1 parent c6fe360 commit c29c1aa

2 files changed

Lines changed: 175 additions & 0 deletions

File tree

‎blog/posts/2026-03-25-arrow-extendr-polars/index.qmd‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
---
2+
title: "arrow-extendr: 🏹 Polars support 🐻‍❄️"
3+
description: |
4+
The latest release of arrow-extendr adds support for Polars,
5+
making it straightforward to move data between R and Polars
6+
DataFrames via the Arrow C Stream interface.
7+
author: Josiah Parry
8+
date: "2026/03/25"
9+
categories: [arrow-extendr, arrow, release, polars]
10+
freeze: true
11+
---
12+
13+
## TL;DR
14+
15+
- [Arrow is the future of cross-language data science](https://josiah.rs/posts/cross-language-ds/)
16+
- New ✨ Polars support for arrow-extendr
17+
- Example: [round-trip from `{polars}`](https://pola-rs.github.io/r-polars/)
18+
19+
---
20+
21+
The latest release of
22+
[arrow-extendr](https://github.com/extendr/arrow-extendr) adds
23+
support for [Polars](https://pola.rs/), bridging the gap between
24+
Polars DataFrames and R's Arrow ecosystem.
25+
26+
## What is arrow-extendr?
27+
28+
[arrow-extendr](https://github.com/extendr/arrow-extendr) is a Rust
29+
crate that makes it straightforward to pass Apache Arrow memory
30+
between R and Rust. Rather than serializing and deserializing at
31+
every boundary, Arrow lets R and Rust share the same in-memory
32+
representation. This means your extendr package can talk directly to
33+
[`{nanoarrow}`](https://arrow.apache.org/nanoarrow/latest/r/index.html),
34+
the [`{arrow}`](https://arrow.apache.org/docs/r/) R package,
35+
[DuckDB](https://github.com/duckdb/duckdb-r),
36+
[DataFusion](https://datafusion.apache.org/), and now Polars, without
37+
any copying or conversion.
38+
39+
Arrow is, in our view, [the standard for cross-language data
40+
science](https://josiah.rs/posts/cross-language-ds/). Supporting it
41+
natively from extendr, rather than routing through a single R package,
42+
is a deliberate choice.
43+
44+
## Use Polars from extendr {#polars-feature}
45+
46+
Polars ships its own Arrow implementation, `polars-arrow`, which is
47+
separate from [`arrow-rs`](https://docs.rs/arrow). That difference
48+
has historically made it awkward to pass data between Polars and the
49+
rest of the Arrow ecosystem. The new `polars` feature flag handles
50+
the translation.
51+
52+
Add it to your `Cargo.toml`:
53+
54+
```toml
55+
[dependencies]
56+
arrow_extendr = {
57+
version = "58",
58+
features = ["polars"],
59+
default-features = false
60+
}
61+
polars-core = "0.53.0"
62+
anyhow = "1"
63+
```
64+
65+
You can also find the latest version on
66+
[crates.io](https://crates.io/crates/arrow-extendr).
67+
68+
This gives you the following conversions via the Arrow C Stream
69+
interface:
70+
71+
| Type | Direction | R object |
72+
| ---- | --------- | -------- |
73+
| `polars_core::frame::DataFrame` | `IntoArrowRobj` | `nanoarrow_array_stream` |
74+
| `polars_core::frame::DataFrame` | `FromArrowRobj` | `nanoarrow_array_stream` |
75+
| `polars_arrow::ffi::ArrowArrayStream` | `IntoArrowRobj` | `nanoarrow_array_stream` |
76+
| `polars_arrow::ffi::ArrowArrayStreamReader` | `FromArrowRobj` | `nanoarrow_array_stream` |
77+
78+
## Round-trip a Polars DataFrame through R {#round-trip}
79+
80+
Accept a `nanoarrow_array_stream` from R, load it into a Polars
81+
`DataFrame`, and return it back to R as a stream.
82+
83+
```rust
84+
use extendr_api::prelude::*;
85+
use anyhow::anyhow;
86+
use arrow_extendr::{FromArrowRobj, IntoArrowRobj};
87+
use polars_core::frame::DataFrame;
88+
89+
#[extendr]
90+
/// @export
91+
fn polars_round_trip(x: Robj) -> anyhow::Result<Robj> {
92+
let df = DataFrame::from_arrow_robj(&x)?;
93+
rprintln!("{df:?}");
94+
df.into_arrow_robj().map_err(|e| anyhow!("{e:?}"))
95+
}
96+
```
97+
98+
On the R side, pass any Arrow-compatible object. Here we use
99+
[`{polars}`](https://pola-rs.github.io/r-polars/) directly:
100+
101+
```r
102+
library(polars)
103+
library(nanoarrow)
104+
105+
df <- pl$DataFrame(a = 1:5, b = letters[1:5])
106+
stream <- as_nanoarrow_array_stream(df)
107+
result <- polars_round_trip(stream)
108+
109+
# convert back to a polars DataFrame
110+
pl$DataFrame(as_arrow_table(result))
111+
```
112+
113+
The return value is a `nanoarrow_array_stream`, so callers can
114+
convert to whatever they need: `{arrow}`, `{polars}`, DuckDB, or
115+
anything else that speaks Arrow.
116+
117+
## What changed
118+
119+
**Breaking changes**
120+
121+
- `FromArrowRobj`, `ToArrowRobj`, and `IntoArrowRobj` moved to the
122+
crate root (`arrow_extendr`). Update imports accordingly.
123+
- `FromArrowRobj::from_arrow_robj` now returns
124+
`std::result::Result<Self, anyhow::Error>` instead of
125+
`Result<Self, ArrowError>`, providing a uniform error type across
126+
both feature implementations.
127+
- `arrow-rs` is now an optional dependency behind the `arrow` feature
128+
flag, which is on by default. Add `features = ["arrow"]` explicitly
129+
if you depend on `arrow-rs` types.
130+
- `ErrArrowRobj` type alias removed. Use `anyhow::Error` directly.
131+
132+
**New features**
133+
134+
- `polars` feature flag enabling interop with `polars-core`.
135+
- `FromArrowRobj` for `polars_arrow::ffi::ArrowArrayStreamReader`.
136+
- `IntoArrowRobj` for `polars_arrow::ffi::ArrowArrayStream`.
137+
- `IntoArrowRobj` for `polars_core::frame::DataFrame`, preserving
138+
chunking.
139+
- `FromArrowRobj` for `polars_core::frame::DataFrame`.
140+
141+
## Community-driven development {#community}
142+
143+
This feature came directly out of a conversation with the
144+
[Bioconductor](https://www.bioconductor.org/) community last August.
145+
The topic was [SingleRust](https://github.com/SingleRust/SingleRust),
146+
a single-cell analysis library in Rust, and how to expose its results
147+
to R. SingleRust's core data structure is a Polars `DataFrame`, which
148+
meant crossing the polars-arrow/R boundary. That thread, between
149+
myself, [Mossa (@cgmossa)](https://github.com/cgmossa), and
150+
[Artür (@Artur-man)](https://github.com/Artur-man), is what motivated
151+
Mossa to build this out.
152+
153+
The only way we know what matters to **you** is for you to tell us.
154+
We really do listen 👂🏼. We're a small team, but when you need
155+
something, we'll provide, or do our best at least!
156+
157+
**Get involved:**
158+
159+
- Join the [Discord](https://discord.gg/7hmApuc).
160+
- Open an issue on
161+
[arrow-extendr](https://github.com/extendr/arrow-extendr/issues),
162+
[extendr](https://github.com/extendr/extendr/issues), or
163+
[rextendr](https://github.com/extendr/rextendr/issues).
164+
- Contribute, PRs are always welcome.
165+
166+
The full changelog is on
167+
[GitHub](https://github.com/extendr/arrow-extendr).

‎justfile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
default:
2+
just --list
3+
4+
serve:
5+
quarto preview
6+
7+
render:
8+
quarto render

0 commit comments

Comments
 (0)